core/internal/cluster: implement multi-node clusters with 'golden ticket'.
As we have fully ripped out all traces of the node management service or
integrity checks, we implement a stopgap system that allows us to
continue developing multi-node clusters. This mechanism is enrolment
using 'golden tickets', which are protobuf messages that can be
generated via the debug service on an existing cluster, and set on a new
node's EnrolmentConfig to bring that enrol that node into the cluster.
As this is a stopgap measure (waiting for better cluster lifecycle
design), this is somewhat poorly implemented, with known issues:
- odd enrolment flow that creates all certificates off-node and results
in some code duplication in the cluster manager and node debug
service
- (more) assumptions that every node is both a kubernetes and etcd
member.
- absolutely no protection against consensus loss due to even quorum
membership, repeated issuance of certificates
- dependence on knowing the IP address of the new node ahead of time,
which is not something that our test harness supports well (or that
we want to rely on at all)
Test Plan: part of existing multi-node tests
X-Origin-Diff: phab/D591
GitOrigin-RevId: 8f099e6ef37f8d47fb2272a3a14b25ed480e377a
diff --git a/core/proto/api/enrolment.proto b/core/proto/api/enrolment.proto
index cf109ad..d4176cc 100644
--- a/core/proto/api/enrolment.proto
+++ b/core/proto/api/enrolment.proto
@@ -18,8 +18,46 @@
package smalltown.core.proto.api;
option go_package = "git.monogon.dev/source/nexantic.git/core/proto/api";
-// The EnrolmentConfig is one of the inputs for the integrity mechanism.
+// EnrolmentConfig is the single boot configuration file contained in the Smalltown ESP. It configures
+// the way the node will start up (what cluster it will join/enroll into/create).
message EnrolmentConfig {
+ // Debug/temporary cluster enrolment method. If set, the node will attempt to enroll into the
+ // cluster that this ticket was generated for. Otherwise, a new cluster will be created.
+ GoldenTicket golden_ticket = 1;
+
// Filled in by node after it is enrolled
- string node_id = 1;
+ string node_id = 2;
+}
+
+// GoldenTicket is a ticket that allows any node to enroll into a cluster, bypassing any integrity
+// checks.
+//
+// Currently, enrolling into a cluster does not use a TPM-based workflow, and instead
+// bases on a simplified workflow of joining consensus by being started with a
+// TLS client certificate. This is a short-circuit fix to allow multi-node
+// clusters for testing before we design the final cluster node lifecycle system.
+message GoldenTicket {
+ // Etcd peer CA certificate.
+ bytes etcd_ca_cert = 1;
+ // Etcd peer client certificate.
+ bytes etcd_client_cert = 2;
+ // Etcd peer client key.
+ bytes etcd_client_key = 3;
+ // Initial etcd peer CRL.
+ bytes etcd_crl = 4;
+
+ message EtcdPeer {
+ string name = 1;
+ string address = 2;
+ }
+ // All other current etcd peers in the cluster.
+ repeated EtcdPeer peers = 5;
+ // The peer that this node should start running.
+ EtcdPeer this = 6;
+
+ // Node configuration. Currently unused (in the future, this will be used to run a node
+ // management service separate from etcd clustering).
+ string node_id = 7;
+ bytes node_cert = 8;
+ bytes node_key = 9;
}