m/n/kubernetes: fix storage provisioner rate limiting
Forget() resets the rate limiter, so it should only be called when
processing has suceeded.
For example, provisioning can fail for a block volume if there is not
enough disk space for the requested size. Previously, this caused the
log to be quickly spammed with "Failed processing item" messages, all
with "numrequeues: 0". With the fix, the retries are properly backed
off, with the requeue counter incrementing.
Change-Id: I8a31fa03fadb202205967e045d4e30f04567d9d1
Reviewed-on: https://review.monogon.dev/c/monogon/+/4007
Tested-by: Jenkins CI
Reviewed-by: Lorenz Brun <lorenz@monogon.tech>
diff --git a/metropolis/node/kubernetes/provisioner.go b/metropolis/node/kubernetes/provisioner.go
index 1ec6f5d..92caa59 100644
--- a/metropolis/node/kubernetes/provisioner.go
+++ b/metropolis/node/kubernetes/provisioner.go
@@ -186,9 +186,9 @@
if err := process(obj); err != nil {
p.logger.Warningf("Failed processing item %q, requeueing (numrequeues: %d): %v", obj, queue.NumRequeues(obj), err)
queue.AddRateLimited(obj)
+ } else {
+ queue.Forget(obj)
}
-
- queue.Forget(obj)
}(obj)
}
}