m/n/kubernetes: fix storage provisioner rate limiting

Forget() resets the rate limiter, so it should only be called when
processing has suceeded.

For example, provisioning can fail for a block volume if there is not
enough disk space for the requested size. Previously, this caused the
log to be quickly  spammed with "Failed processing item" messages, all
with "numrequeues: 0". With the fix, the retries are properly backed
off, with the requeue counter incrementing.

Change-Id: I8a31fa03fadb202205967e045d4e30f04567d9d1
Reviewed-on: https://review.monogon.dev/c/monogon/+/4007
Tested-by: Jenkins CI
Reviewed-by: Lorenz Brun <lorenz@monogon.tech>
diff --git a/metropolis/node/kubernetes/provisioner.go b/metropolis/node/kubernetes/provisioner.go
index 1ec6f5d..92caa59 100644
--- a/metropolis/node/kubernetes/provisioner.go
+++ b/metropolis/node/kubernetes/provisioner.go
@@ -186,9 +186,9 @@
 			if err := process(obj); err != nil {
 				p.logger.Warningf("Failed processing item %q, requeueing (numrequeues: %d): %v", obj, queue.NumRequeues(obj), err)
 				queue.AddRateLimited(obj)
+			} else {
+				queue.Forget(obj)
 			}
-
-			queue.Forget(obj)
 		}(obj)
 	}
 }