tree fd718931af36798650ddbf1a8978a94220994e82
parent 1dcede9600e4c1584da4fbe89128970ea9532860
author Jan Schär <jan@monogon.tech> 1727263009 +0200
committer Jan Schär <jan@monogon.tech> 1727351049 +0000

m/n/c/consensus: fix startup after removing a cluster node

The consensus service was waiting for all initial peers to be DNS
resolvable before starting etcd. However, the list of initial peers is
never updated. If an etcd member is removed from the cluster, it is no
longer resolvable, but may still be contained in initial peer lists. The
consensus service then fails to start, as it is blocked forever waiting
for the removed peer to become resolvable.

The wait for resolvability was added in c1cb37ce9c43 with this
explanation:

> It also makes the consensus service wait for DNS resolvability before
> attempting to join an existing cluster, which makes etcd startup much
> cleaner (as etcd will itself crash if it cannot immediately resolve
> its ExistingPeers in startup).

This does not appear to be needed anymore. I did not observe etcd
crashes after removing the wait for resolvability.

I extended the e2e test to test this scenario. After removing the
consensus role, it also deletes the node and reboots the remaining
nodes. I moved these tests to the ha_cold suite, because with encryption
enabled, we currently cannot reboot a node in a 2-node cluster.

Change-Id: If811c79ea127550fa9ca750014272fa885767c77
Reviewed-on: https://review.monogon.dev/c/monogon/+/3454
Tested-by: Jenkins CI
Reviewed-by: Serge Bazanski <serge@monogon.tech>