tree 1d4a2a79f2f63cca28614df7f37fd044fd0844da
parent b354453656f82d0a38b3f2ed0d1ebf843c14d922
author Serge Bazanski <serge@monogon.tech> 1650886171 +0200
committer Sergiusz Bazanski <serge@monogon.tech> 1653406287 +0000

curator: remove dispatch system

This vastly simplifies the curator by removing the dispatch and per-RPC
switch logic, instead replacing it with the gRPC server stopping
whenever the leadership status of a curator switches.

The downside is technically the possibility of some 'stale' RPC
handling, where a leader/follower accepts some new RPC even though it's
in the process of switching its leadership.

The rationale for this change is:

   1. Leadership-exclusive actions are guarded by the etcd leadership
      lock being held, so there's no chance a long pending RPC to a
      leader that just stepped down will cause split brain scenarios.

   2. We're moving away from follower proxying, and followers will
      instead just serve a 'who's the leader' RPC. These are okay to
      serve stale data (ie. when the contacted follower should've
      switched to be a leader, or a follower of another leader), as
      during leadership failover we expect clients to perform retry
      loops until a new leadership connection is established.

Another downside (or perhaps upside) is that we don't start the listener
until we're ready to serve data, either the full API as a leader or a
reduced API as a follower. The downside is that clients will have to
retry connections until the leader is running, and that it might be
difficult to tell apart a node which isn't yet running the curator from
a broken node, or one that will not run the curator at all. On the other
hand, succesfully establishing a connections means that we are sure to
get a gRPC response instead of a hang because the curator isn't yet ready
to serve.

Change-Id: I2ec35f00bce72f0f337e8e25e8c71f5265a7d8bb
Reviewed-on: https://review.monogon.dev/c/monogon/+/685
Reviewed-by: Lorenz Brun <lorenz@monogon.tech>