osbase/supervisor: only enter DONE state after runnable returns
Previously, a node was marked DONE as soon as it signaled DONE. If a GC
run happens between the time when the runnable signals DONE, and when
the runnable exit is processed, this causes problems. The test which I
added panics without the other changes:
panic: could not find [inner] (root.inner) in root (NODE_STATE_NEW)
If the delay is long enough that the inner node has already restarted,
then this could even end up with multiple instances of the same runnable
running simultaneously.
I fixed this problem by only entering the DONE state after the runnable
has returned.
Change-Id: If73b73f104c4cc204bce4374f4ba5f7e163e4a0b
Reviewed-on: https://review.monogon.dev/c/monogon/+/3715
Tested-by: Jenkins CI
Reviewed-by: Lorenz Brun <lorenz@monogon.tech>
diff --git a/osbase/supervisor/supervisor_processor.go b/osbase/supervisor/supervisor_processor.go
index 595e2be..667b2ab 100644
--- a/osbase/supervisor/supervisor_processor.go
+++ b/osbase/supervisor/supervisor_processor.go
@@ -271,10 +271,11 @@
n := s.nodeByDN(r.dn)
ctx := n.ctx
- // Simple case: it was marked as Done and quit with no error.
- if n.state == NodeStateDone && r.err == nil {
+ // Simple case: it has signaled Done and quit with no error.
+ if n.signaledDone && r.err == nil {
+ // Mark the node as DONE.
+ n.state = NodeStateDone
s.metrics.NotifyNodeState(r.dn, n.state)
- // Do nothing. This was supposed to happen. Keep the process as DONE.
return
}