Improve documentation, remove dead code plus some minor refactorings
This improves our code-to-comments ratio by a lot.
On the refactorings:
- Simplify the cluster join mode to just a single protobuf message -
a node can either join an existing cluster or bootstrap a new one.
All of the node-level setup like hostname and trust backend is done
using the setup call, since those are identical for both cases.
- We don't need a node name separate from the hostname. Ideally, we would
get rid of IP addresses for etcd as well.
- Google API design guidelines suggest the `List` term (vs. `Get`).
- Add username to comments for consistency. I think the names provide
useful context, but git blame is a thing. What do you think?
- Fixed or silenced some ignored error checks in preparation of using
an errcheck linter. Especially during early boot, many errors are
obviously not recoverable, but logging them can provide useful debugging info.
- Split up the common package into smaller subpackages.
- Remove the audit package (this will be a separate service that probably
uses it own database, rather than etcd).
- Move storage constants to storage package.
- Remove the unused KV type.
I also added a bunch of TODO comments with discussion points.
Added both of you as blocking reviewers - please comment if I
misunderstood any of your code.
Test Plan: Everything compiles and scripts:launch works (for whatever that's worth).
X-Origin-Diff: phab/D235
GitOrigin-RevId: 922fec5076e8d683e1138f26d2cb490de64a9777
diff --git a/core/api/api/schema.proto b/core/api/api/schema.proto
index 238cfce..bf25a74 100644
--- a/core/api/api/schema.proto
+++ b/core/api/api/schema.proto
@@ -21,71 +21,87 @@
option go_package = "git.monogon.dev/source/nexantic.git/core/generated/api";
+// TODO(leo): A "cluster" in terms of this API is an etcd cluster. We have
+// since realized that we will need multiple kinds of nodes in a Smalltown cluster
+// (like worker nodes), which aren't etcd members. This API is pretty strongly
+// coupled to etcd at this point. How do we handle cluster membership for workers?
+
+// The ClusterManagement service is used by an authenticated administrative user
+// to manage node membership in an existing Smalltown cluster.
service ClusterManagement {
- // Add a node to the Smalltown cluster
+ // Add a node to the cluster, subject to successful remote attestation.
rpc AddNode (AddNodeRequest) returns (AddNodeResponse) {
}
- // Remove a node from the Smalltown cluster
+ // Remove a node from the cluster.
rpc RemoveNode (RemoveNodeRequest) returns (RemoveNodeRequest) {
}
- // Get all cluster nodes
- rpc GetNodes (GetNodesRequest) returns (GetNodesResponse) {
+ // List all cluster nodes
+ rpc ListNodes (ListNodesRequest) returns (ListNodesResponse) {
}
}
+// SetupService manages a single node's lifecycle, and it called either by an administrative
+// user while bootstrapping the cluster, or by existing nodes in a cluster.
service SetupService {
- // SetupNewCluster configures this node to either start a new Smalltown cluster or join an existing one
+ // Setup bootstraps an unprovisioned node and selects its bootstrapping mode
+ // (either joining an existing cluster, or creating a new one).
rpc Setup (SetupRequest) returns (SetupResponse) {
}
- // JoinCluster can be called by another Smalltown node when the node has been put in to JOIN_CLUSTER mode using Setup.
- // This request sets up all necessary config variables, joins the consensus and puts the node in production state.
- rpc ProvisionCluster (ProvisionClusterRequest) returns (ProvisionClusterResponse) {
+ // BootstrapCluster is called by an administrative user to bootstrap the first node
+ // of a Smalltown cluster.
+ rpc BootstrapNewCluster (BootstrapNewClusterRequest) returns (BootstrapNewClusterResponse) {
}
+ // JoinCluster can be called by another Smalltown node when the node has been put into
+ // JOIN_CLUSTER mode using Setup. This request sets up all necessary config variables,
+ // joins the consensus and puts the node into production state.
+ rpc JoinCluster (JoinClusterRequest) returns (JoinClusterResponse) {
+
+ }
+
+ // Attest is called by an existing cluster node to verify a node's remote
+ // attestation identity.
+ //
+ // This is not yet implemented, but at least the following values will be signed:
+ //
+ // - the node's trust backend and other configuration
+ // (such that the new node can verify whether it is in an acceptable state)
+ //
+ // - the set of PCRs we use for sealing, which includes the firmware and secure boot state
+ // (see pkg/tpm/tpm.go)
rpc Attest (AttestRequest) returns (AttestResponse) {
}
}
message SetupRequest {
- oneof request {
- NewClusterRequest newCluster = 1;
- JoinClusterRequest joinCluster = 2;
- }
-
-}
-
-message NewClusterRequest {
+ // Hostname for the new node
+ // TODO(leo): how will we handle hostnames? do we let the customer choose them? etc.
string nodeName = 1;
- string externalHost = 2;
- smalltown.common.TrustBackend trustBackend = 3;
-}
-
-message JoinClusterRequest {
+ // Trust backend to be used. Right now, we support just one kind of trust
+ // backend (our internal one), but at some point, we might support external
+ // key management hardware. It has to be configured this early since it would
+ // also store cluster secrets used during provisioning and setup.
+ smalltown.common.TrustBackend trustBackend = 2;
}
message SetupResponse {
- oneof response {
- NewClusterResponse newCluster = 1;
- JoinClusterResponse joinCluster = 2;
- }
-}
-
-message NewClusterResponse {
-}
-
-message JoinClusterResponse {
+ // provisioningToken is a secret key that establishes a mutual trust-on-first-use
+ // relationship between the cluster and the new node (after passing attestation checks).
string provisioningToken = 1;
}
+// ConsensusCertificates is a node's individual etcd certificates.
+// When provisioning a new node, the existing node sends the new node
+// its certificates after authenticating it.
message ConsensusCertificates {
bytes ca = 1;
bytes crl = 2;
@@ -93,18 +109,25 @@
bytes key = 4;
}
-message ProvisionClusterRequest {
+message JoinClusterRequest {
+ // The callee's provisioningToken. Knowledge of this token authenticates the caller.
string provisioningToken = 1;
-
+ // Cluster bootstrap URI for etcd. The caller will set this to the
+ // list of existing nodes in the cluster. This value is only used during bootstrap.
string initialCluster = 2;
- string nodeName = 3;
- string externalHost = 4;
- smalltown.common.TrustBackend trustBackend = 5;
- bytes storeKey = 6;
- ConsensusCertificates certs = 7;
+ // New node's etcd client certificates
+ ConsensusCertificates certs = 3;
}
-message ProvisionClusterResponse {
+message JoinClusterResponse {
+
+}
+
+message BootstrapNewClusterRequest {
+
+}
+
+message BootstrapNewClusterResponse {
}
@@ -116,26 +139,12 @@
string response = 1;
}
-message Key {
- string label = 1;
- string type = 2;
-}
-
-message CreateKeyRequest {
- Key key = 1;
-}
-
-message CreateKeyResponse {
-
-}
-
message AddNodeRequest {
- string host = 1;
- uint32 apiPort = 2;
- uint32 consensusPort = 3;
- string token = 4;
- string name = 5;
- smalltown.common.TrustBackend trustBackend = 6;
+ // New node's address to connect to.
+ // TODO(leo): Is this always an IP address?
+ string addr = 1;
+ // New node's provisioning token.
+ string provisioningToken = 4;
}
message AddNodeResponse {
@@ -150,17 +159,22 @@
}
-message GetNodesRequest {
+message ListNodesRequest {
}
-message GetNodesResponse {
+message ListNodesResponse {
repeated Node nodes = 1;
}
+// Node describes a single node's etcd membership state
message Node {
+ // etcd member ID
uint64 id = 1;
+ // etcd member name
string name = 2;
+ // etcd peer URL
string address = 3;
+ // Whether the etcd member is synced with the cluster.
bool synced = 4;
}
diff --git a/core/api/common/main.proto b/core/api/common/main.proto
index 8ce8f99..dbc7b8e 100644
--- a/core/api/common/main.proto
+++ b/core/api/common/main.proto
@@ -19,11 +19,6 @@
option go_package = "git.monogon.dev/source/nexantic.git/core/generated/common";
package smalltown.common;
-message KV {
- string key = 1;
- bytes value = 2;
-}
-
enum TrustBackend {
DUMMY = 0;
TPM = 1;
diff --git a/core/cmd/init/main.go b/core/cmd/init/main.go
index a69de2c..1d068d4 100644
--- a/core/cmd/init/main.go
+++ b/core/cmd/init/main.go
@@ -36,8 +36,11 @@
debug.PrintStack()
}
unix.Sync()
- // TODO: Switch this to Reboot when init panics are less likely
- unix.Reboot(unix.LINUX_REBOOT_CMD_POWER_OFF)
+ // TODO(lorenz): Switch this to Reboot when init panics are less likely
+ // Best effort, nothing we can do if this fails except printing the error to the console.
+ if err := unix.Reboot(unix.LINUX_REBOOT_CMD_POWER_OFF); err != nil {
+ panic(fmt.Sprintf("failed to halt node: %v\n", err))
+ }
}()
logger, err := zap.NewDevelopment()
if err != nil {
@@ -71,7 +74,10 @@
if err != nil {
panic(err)
}
- networkSvc.Start()
+
+ if err := networkSvc.Start(); err != nil {
+ logger.Panic("Failed to start network service", zap.Error(err))
+ }
nodeInstance, err := node.NewSmalltownNode(logger, 7833, 7834)
if err != nil {
diff --git a/core/internal/api/BUILD.bazel b/core/internal/api/BUILD.bazel
index 6e3cb2b..b7aa48d 100644
--- a/core/internal/api/BUILD.bazel
+++ b/core/internal/api/BUILD.bazel
@@ -4,7 +4,7 @@
name = "go_default_library",
srcs = [
"cluster.go",
- "main.go",
+ "server.go",
"setup.go",
],
importpath = "git.monogon.dev/source/nexantic.git/core/internal/api",
@@ -12,8 +12,9 @@
deps = [
"//core/api/api:go_default_library",
"//core/internal/common:go_default_library",
+ "//core/internal/common/grpc:go_default_library",
+ "//core/internal/common/service:go_default_library",
"//core/internal/consensus:go_default_library",
- "@com_github_casbin_casbin//:go_default_library",
"@org_golang_google_grpc//:go_default_library",
"@org_uber_go_zap//:go_default_library",
],
diff --git a/core/internal/api/cluster.go b/core/internal/api/cluster.go
index 64a757f..3b45c40 100644
--- a/core/internal/api/cluster.go
+++ b/core/internal/api/cluster.go
@@ -22,7 +22,7 @@
"encoding/hex"
"fmt"
schema "git.monogon.dev/source/nexantic.git/core/generated/api"
- "git.monogon.dev/source/nexantic.git/core/internal/common"
+ "git.monogon.dev/source/nexantic.git/core/internal/common/grpc"
"errors"
@@ -35,7 +35,7 @@
func (s *Server) AddNode(ctx context.Context, req *schema.AddNodeRequest) (*schema.AddNodeResponse, error) {
// Setup API client
- c, err := common.NewSmalltownAPIClient(fmt.Sprintf("%s:%d", req.Host, req.ApiPort))
+ c, err := grpc.NewSmalltownAPIClient(fmt.Sprintf("%s:%d", req.Addr, s.config.Port))
if err != nil {
return nil, err
}
@@ -60,45 +60,44 @@
return nil, ErrAttestationFailed
}
- consensusCerts, err := s.consensusService.IssueCertificate(req.Host)
+ consensusCerts, err := s.consensusService.IssueCertificate(req.Addr)
if err != nil {
return nil, err
}
- // Provision cluster info locally
- memberID, err := s.consensusService.AddMember(ctx, req.Name, fmt.Sprintf("https://%s:%d", req.Host, req.ConsensusPort))
+ // TODO(leo): fetch remote hostname rather than using the addr
+ name := req.Addr
+
+ // Add new node to local etcd cluster.
+ memberID, err := s.consensusService.AddMember(ctx, name, fmt.Sprintf("https://%s:%d", req.Addr, s.config.Port))
if err != nil {
return nil, err
}
- s.Logger.Info("Added new node to consensus cluster; provisioning external node now",
- zap.String("host", req.Host), zap.Uint32("port", req.ApiPort),
- zap.Uint32("consensus_port", req.ConsensusPort), zap.String("name", req.Name))
+ s.Logger.Info("Added new node to consensus cluster; sending cluster join request to node",
+ zap.String("addr", req.Addr), zap.Uint16("port", s.config.Port))
- // Provision cluster info externally
- _, err = c.Setup.ProvisionCluster(ctx, &schema.ProvisionClusterRequest{
+ // Send JoinCluster request to new node to make it join.
+ _, err = c.Setup.JoinCluster(ctx, &schema.JoinClusterRequest{
InitialCluster: s.consensusService.GetInitialClusterString(),
- ProvisioningToken: req.Token,
- ExternalHost: req.Host,
- NodeName: req.Name,
- TrustBackend: req.TrustBackend,
+ ProvisioningToken: req.ProvisioningToken,
Certs: consensusCerts,
})
if err != nil {
- err3 := s.consensusService.RevokeCertificate(req.Host)
- if err3 != nil {
- s.Logger.Error("Failed to revoke a certificate after rollback, potential security risk", zap.Error(err3))
+ errRevoke := s.consensusService.RevokeCertificate(req.Addr)
+ if errRevoke != nil {
+ s.Logger.Error("Failed to revoke a certificate after rollback - potential security risk", zap.Error(errRevoke))
}
- // Revert Consensus add member - might fail if consensus cannot be established
- err2 := s.consensusService.RemoveMember(ctx, memberID)
- if err2 != nil || err3 != nil {
- return nil, fmt.Errorf("Rollback failed after failed provisioning; err=%v; err_rb=%v; err_revoke=%v", err, err2, err3)
+ // Revert etcd add member - might fail if consensus cannot be established.
+ errRemove := s.consensusService.RemoveMember(ctx, memberID)
+ if errRemove != nil || errRevoke != nil {
+ return nil, fmt.Errorf("rollback failed after failed provisioning; err=%v; err_rb=%v; err_revoke=%v", err, errRemove, errRevoke)
}
return nil, err
}
s.Logger.Info("Fully provisioned new node",
- zap.String("host", req.Host), zap.Uint32("port", req.ApiPort),
- zap.Uint32("consensus_port", req.ConsensusPort), zap.String("name", req.Name),
+ zap.String("host", req.Addr),
+ zap.Uint16("apiPort", s.config.Port),
zap.Uint64("member_id", memberID))
return &schema.AddNodeResponse{}, nil
@@ -108,7 +107,7 @@
panic("implement me")
}
-func (s *Server) GetNodes(context.Context, *schema.GetNodesRequest) (*schema.GetNodesResponse, error) {
+func (s *Server) ListNodes(context.Context, *schema.ListNodesRequest) (*schema.ListNodesResponse, error) {
nodes := s.consensusService.GetNodes()
resNodes := make([]*schema.Node, len(nodes))
@@ -121,7 +120,7 @@
}
}
- return &schema.GetNodesResponse{
+ return &schema.ListNodesResponse{
Nodes: resNodes,
}, nil
}
diff --git a/core/internal/api/main.go b/core/internal/api/server.go
similarity index 89%
rename from core/internal/api/main.go
rename to core/internal/api/server.go
index 20c3a3a..715e99e 100644
--- a/core/internal/api/main.go
+++ b/core/internal/api/server.go
@@ -20,8 +20,8 @@
"fmt"
schema "git.monogon.dev/source/nexantic.git/core/generated/api"
"git.monogon.dev/source/nexantic.git/core/internal/common"
+ "git.monogon.dev/source/nexantic.git/core/internal/common/service"
"git.monogon.dev/source/nexantic.git/core/internal/consensus"
- "github.com/casbin/casbin"
"go.uber.org/zap"
"google.golang.org/grpc"
"net"
@@ -29,9 +29,8 @@
type (
Server struct {
- *common.BaseService
+ *service.BaseService
- ruleEnforcer *casbin.Enforcer
setupService common.SetupService
grpcServer *grpc.Server
@@ -52,7 +51,7 @@
consensusService: consensusService,
}
- s.BaseService = common.NewBaseService("api", logger, s)
+ s.BaseService = service.NewBaseService("api", logger, s)
grpcServer := grpc.NewServer()
schema.RegisterClusterManagementServer(grpcServer, s)
@@ -75,7 +74,7 @@
s.Logger.Error("API server failed", zap.Error(err))
}()
- s.Logger.Info("GRPC listening", zap.String("host", listenHost))
+ s.Logger.Info("gRPC listening", zap.String("host", listenHost))
return nil
}
diff --git a/core/internal/api/setup.go b/core/internal/api/setup.go
index 6aeda40..f317534 100644
--- a/core/internal/api/setup.go
+++ b/core/internal/api/setup.go
@@ -34,66 +34,27 @@
)
func (s *Server) Setup(c context.Context, r *schema.SetupRequest) (*schema.SetupResponse, error) {
-
- switch r.Request.(type) {
- case *schema.SetupRequest_JoinCluster:
- token, err := s.enterJoinCluster(r.GetJoinCluster())
- if err != nil {
- return nil, err
- }
-
- return &schema.SetupResponse{
- Response: &schema.SetupResponse_JoinCluster{
- JoinCluster: &schema.JoinClusterResponse{
- ProvisioningToken: token,
- },
- },
- }, nil
-
- case *schema.SetupRequest_NewCluster:
- return &schema.SetupResponse{
- Response: &schema.SetupResponse_NewCluster{
- NewCluster: &schema.NewClusterResponse{},
- },
- }, s.setupNewCluster(r.GetNewCluster())
- }
-
return &schema.SetupResponse{}, nil
}
-func (s *Server) enterJoinCluster(r *schema.JoinClusterRequest) (string, error) {
- err := s.setupService.EnterJoinClusterMode()
- if err != nil {
- return "", err
- }
-
- return s.setupService.GetJoinClusterToken(), nil
+func (s *Server) BootstrapNewCluster(context.Context, *schema.BootstrapNewClusterRequest) (*schema.BootstrapNewClusterResponse, error) {
+ err := s.setupService.SetupNewCluster()
+ return &schema.BootstrapNewClusterResponse{}, err
}
-func (s *Server) setupNewCluster(r *schema.NewClusterRequest) error {
- if len(r.NodeName) < MinNameLength {
- return ErrInvalidNameLength
- }
- return s.setupService.SetupNewCluster(r.NodeName, r.ExternalHost)
-}
-
-func (s *Server) ProvisionCluster(ctx context.Context, req *schema.ProvisionClusterRequest) (*schema.ProvisionClusterResponse, error) {
- if len(req.NodeName) < MinNameLength {
- return nil, ErrInvalidNameLength
- }
-
+func (s *Server) JoinCluster(ctx context.Context, req *schema.JoinClusterRequest) (*schema.JoinClusterResponse, error) {
// Verify provisioning token
if s.setupService.GetJoinClusterToken() != req.ProvisioningToken {
return nil, ErrInvalidProvisioningToken
}
// Join cluster
- err := s.setupService.JoinCluster(req.NodeName, req.InitialCluster, req.ExternalHost, req.Certs)
+ err := s.setupService.JoinCluster(req.InitialCluster, req.Certs)
if err != nil {
return nil, err
}
- return &schema.ProvisionClusterResponse{}, nil
+ return &schema.JoinClusterResponse{}, nil
}
func (s *Server) Attest(c context.Context, r *schema.AttestRequest) (*schema.AttestResponse, error) {
diff --git a/core/internal/audit/main.go b/core/internal/audit/main.go
deleted file mode 100644
index 2d43dd0..0000000
--- a/core/internal/audit/main.go
+++ /dev/null
@@ -1,35 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package audit
-
-import "go.etcd.io/etcd/clientv3"
-
-type (
- Logger struct {
- kv *clientv3.KV
- }
-)
-
-func NewAuditLogger(kv *clientv3.KV) (*Logger, error) {
- return &Logger{
- kv: kv,
- }, nil
-}
-
-func Log(user, action, params string) error {
- return nil
-}
diff --git a/core/internal/common/BUILD.bazel b/core/internal/common/BUILD.bazel
index 4312b88..1adbfd8 100644
--- a/core/internal/common/BUILD.bazel
+++ b/core/internal/common/BUILD.bazel
@@ -2,19 +2,8 @@
go_library(
name = "go_default_library",
- srcs = [
- "grpc.go",
- "service.go",
- "setup.go",
- "storage.go",
- "util.go",
- ],
+ srcs = ["setup.go"],
importpath = "git.monogon.dev/source/nexantic.git/core/internal/common",
- visibility = ["//core:__subpackages__"],
- deps = [
- "//core/api/api:go_default_library",
- "//core/api/common:go_default_library",
- "@org_golang_google_grpc//:go_default_library",
- "@org_uber_go_zap//:go_default_library",
- ],
+ visibility = ["//:__subpackages__"],
+ deps = ["//core/api/api:go_default_library"],
)
diff --git a/core/internal/common/grpc/BUILD.bazel b/core/internal/common/grpc/BUILD.bazel
new file mode 100644
index 0000000..85661c6
--- /dev/null
+++ b/core/internal/common/grpc/BUILD.bazel
@@ -0,0 +1,12 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library")
+
+go_library(
+ name = "go_default_library",
+ srcs = ["grpc.go"],
+ importpath = "git.monogon.dev/source/nexantic.git/core/internal/common/grpc",
+ visibility = ["//:__subpackages__"],
+ deps = [
+ "//core/api/api:go_default_library",
+ "@org_golang_google_grpc//:go_default_library",
+ ],
+)
diff --git a/core/internal/common/grpc.go b/core/internal/common/grpc/grpc.go
similarity index 98%
rename from core/internal/common/grpc.go
rename to core/internal/common/grpc/grpc.go
index 5512a5c..e9cfed0 100644
--- a/core/internal/common/grpc.go
+++ b/core/internal/common/grpc/grpc.go
@@ -14,7 +14,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-package common
+package grpc
import (
"git.monogon.dev/source/nexantic.git/core/generated/api"
diff --git a/core/internal/audit/BUILD.bazel b/core/internal/common/service/BUILD.bazel
similarity index 66%
rename from core/internal/audit/BUILD.bazel
rename to core/internal/common/service/BUILD.bazel
index 9684d55..a06ca83 100644
--- a/core/internal/audit/BUILD.bazel
+++ b/core/internal/common/service/BUILD.bazel
@@ -2,8 +2,8 @@
go_library(
name = "go_default_library",
- srcs = ["main.go"],
- importpath = "git.monogon.dev/source/nexantic.git/core/internal/audit",
+ srcs = ["service.go"],
+ importpath = "git.monogon.dev/source/nexantic.git/core/internal/common/service",
visibility = ["//:__subpackages__"],
- deps = ["@io_etcd_go_etcd//clientv3:go_default_library"],
+ deps = ["@org_uber_go_zap//:go_default_library"],
)
diff --git a/core/internal/common/service.go b/core/internal/common/service/service.go
similarity index 96%
rename from core/internal/common/service.go
rename to core/internal/common/service/service.go
index 3bdc1f9..e093ff6 100644
--- a/core/internal/common/service.go
+++ b/core/internal/common/service/service.go
@@ -14,7 +14,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-package common
+package service
import (
"errors"
@@ -74,7 +74,7 @@
return nil
}
-// Stop stops the service. THis is an atomic operation and should only be called on a running service.
+// Stop stops the service. This is an atomic operation and should only be called on a running service.
func (b *BaseService) Stop() error {
b.mutex.Lock()
defer b.mutex.Unlock()
diff --git a/core/internal/common/setup.go b/core/internal/common/setup.go
index 331d29a..1124d27 100644
--- a/core/internal/common/setup.go
+++ b/core/internal/common/setup.go
@@ -18,20 +18,25 @@
import "git.monogon.dev/source/nexantic.git/core/generated/api"
+// TODO(leo): merge api and node packages and get rid of this extra layer of indirection?
+
type (
SetupService interface {
CurrentState() SmalltownState
GetJoinClusterToken() string
- SetupNewCluster(name string, externalHost string) error
+ SetupNewCluster() error
EnterJoinClusterMode() error
- JoinCluster(name string, clusterString string, externalHost string, certs *api.ConsensusCertificates) error
+ JoinCluster(initialCluster string, certs *api.ConsensusCertificates) error
}
SmalltownState string
)
const (
- StateSetupMode SmalltownState = "setup"
+ // Node is unprovisioned and waits for Setup to be called.
+ StateSetupMode SmalltownState = "setup"
+ // Setup() has been called, node waits for a JoinCluster or BootstrapCluster call.
StateClusterJoinMode SmalltownState = "join"
- StateConfigured SmalltownState = "configured"
+ // Node is fully provisioned.
+ StateConfigured SmalltownState = "configured"
)
diff --git a/core/internal/common/util.go b/core/internal/common/util.go
deleted file mode 100644
index fc8a72b..0000000
--- a/core/internal/common/util.go
+++ /dev/null
@@ -1,33 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package common
-
-import "git.monogon.dev/source/nexantic.git/core/generated/common"
-
-func MapToKVs(input map[string]string) []*common.KV {
- kvs := make([]*common.KV, len(input))
-
- i := 0
- for key, item := range input {
- kvs[i] = &common.KV{
- Key: key,
- Value: []byte(item),
- }
- }
-
- return kvs
-}
diff --git a/core/internal/consensus/BUILD.bazel b/core/internal/consensus/BUILD.bazel
index c1c6989..5fa33ac 100644
--- a/core/internal/consensus/BUILD.bazel
+++ b/core/internal/consensus/BUILD.bazel
@@ -7,7 +7,7 @@
visibility = ["//:__subpackages__"],
deps = [
"//core/api/api:go_default_library",
- "//core/internal/common:go_default_library",
+ "//core/internal/common/service:go_default_library",
"//core/internal/consensus/ca:go_default_library",
"@com_github_pkg_errors//:go_default_library",
"@io_etcd_go_etcd//clientv3:go_default_library",
diff --git a/core/internal/consensus/ca/ca.go b/core/internal/consensus/ca/ca.go
index 925f030..a8cfbd9 100644
--- a/core/internal/consensus/ca/ca.go
+++ b/core/internal/consensus/ca/ca.go
@@ -14,8 +14,16 @@
// See the License for the specific language governing permissions and
// limitations under the License.
+// package ca implements a simple standards-compliant certificate authority.
+// It only supports ed25519 keys, and does not maintain any persistent state.
+//
+// CA and certificates successfully pass https://github.com/zmap/zlint
+// (minus the CA/B rules that a public CA would adhere to, which requires
+// things like OCSP servers, Certificate Policies and ECDSA/RSA-only keys).
package ca
+// TODO(leo): add zlint test
+
import (
"crypto"
"crypto/ed25519"
@@ -48,7 +56,7 @@
// violates Section 4.2.1.2 of RFC 5280 without this. Should eventually be redundant.
//
// Taken from https://github.com/FiloSottile/mkcert/blob/master/cert.go#L295 written by one of Go's
-// crypto engineers
+// crypto engineers (BSD 3-clause).
func calculateSKID(pubKey crypto.PublicKey) ([]byte, error) {
spkiASN1, err := x509.MarshalPKIXPublicKey(pubKey)
if err != nil {
@@ -67,6 +75,7 @@
return skid[:], nil
}
+// New creates a new certificate authority with the given common name.
func New(name string) (*CA, error) {
pubKey, privKey, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
@@ -76,7 +85,7 @@
serialNumberLimit := new(big.Int).Lsh(big.NewInt(1), 127)
serialNumber, err := rand.Int(rand.Reader, serialNumberLimit)
if err != nil {
- return nil, fmt.Errorf("Failed to generate serial number: %w", err)
+ return nil, fmt.Errorf("failed to generate serial number: %w", err)
}
skid, err := calculateSKID(pubKey)
@@ -101,7 +110,7 @@
caCertRaw, err := x509.CreateCertificate(rand.Reader, caCert, caCert, pubKey, privKey)
if err != nil {
- return nil, fmt.Errorf("Failed to create root certificate: %w", err)
+ return nil, fmt.Errorf("failed to create root certificate: %w", err)
}
ca := &CA{
@@ -109,16 +118,17 @@
CACertRaw: caCertRaw,
CACert: caCert,
}
- if ca.reissueCRL() != nil {
+ if ca.ReissueCRL() != nil {
return nil, fmt.Errorf("failed to create initial CRL: %w", err)
}
return ca, nil
}
+// FromCertificates restores CA state.
func FromCertificates(caCert []byte, caKey []byte, crl []byte) (*CA, error) {
if len(caKey) != ed25519.PrivateKeySize {
- return nil, errors.New("Invalid CA private key size")
+ return nil, errors.New("invalid CA private key size")
}
privateKey := ed25519.PrivateKey(caKey)
@@ -135,11 +145,11 @@
CACertRaw: caCert,
CACert: caCertVal,
Revoked: crlVal.TBSCertList.RevokedCertificates,
- CRLRaw: crl,
}, nil
}
-func (ca *CA) IssueCertificate(hostname string) (cert []byte, privkey []byte, err error) {
+// IssueCertificate issues a certificate
+func (ca *CA) IssueCertificate(commonName string) (cert []byte, privkey []byte, err error) {
serialNumberLimit := new(big.Int).Lsh(big.NewInt(1), 127)
serialNumber, err := rand.Int(rand.Reader, serialNumberLimit)
if err != nil {
@@ -159,7 +169,7 @@
etcdCert := &x509.Certificate{
SerialNumber: serialNumber,
Subject: pkix.Name{
- CommonName: hostname,
+ CommonName: commonName,
OrganizationalUnit: []string{"etcd"},
},
IsCA: false,
@@ -167,13 +177,13 @@
NotBefore: time.Now(),
NotAfter: unknownNotAfter,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
- DNSNames: []string{hostname},
+ DNSNames: []string{commonName},
}
cert, err = x509.CreateCertificate(rand.Reader, etcdCert, ca.CACert, pubKey, ca.PrivateKey)
return
}
-func (ca *CA) reissueCRL() error {
+func (ca *CA) ReissueCRL() error {
compatCert := CompatCertificate(*ca.CACert)
newCRL, err := compatCert.CreateCRL(rand.Reader, ca.PrivateKey, ca.Revoked, time.Now(), unknownNotAfter)
if err != nil {
@@ -193,5 +203,5 @@
SerialNumber: serial,
RevocationTime: time.Now(),
})
- return ca.reissueCRL()
+ return ca.ReissueCRL()
}
diff --git a/core/internal/consensus/consensus.go b/core/internal/consensus/consensus.go
index d87a506..f5ee949 100644
--- a/core/internal/consensus/consensus.go
+++ b/core/internal/consensus/consensus.go
@@ -14,6 +14,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
+// package consensus manages the embedded etcd cluster.
package consensus
import (
@@ -23,6 +24,7 @@
"encoding/hex"
"encoding/pem"
"fmt"
+ "git.monogon.dev/source/nexantic.git/core/internal/common/service"
"io/ioutil"
"math/rand"
"net/url"
@@ -33,7 +35,6 @@
"time"
"git.monogon.dev/source/nexantic.git/core/generated/api"
- "git.monogon.dev/source/nexantic.git/core/internal/common"
"git.monogon.dev/source/nexantic.git/core/internal/consensus/ca"
"github.com/pkg/errors"
"go.etcd.io/etcd/clientv3"
@@ -61,13 +62,16 @@
type (
Service struct {
- *common.BaseService
+ *service.BaseService
- etcd *embed.Etcd
- kv clientv3.KV
- ready bool
- bootstrapCA *ca.CA
- bootstrapCert []byte
+ etcd *embed.Etcd
+ kv clientv3.KV
+ ready bool
+
+ // bootstrapCA and bootstrapCert cache the etcd cluster CA data during bootstrap.
+ bootstrapCA *ca.CA
+ bootstrapCert []byte
+
watchCRLTicker *time.Ticker
lastCRL []byte
@@ -79,10 +83,9 @@
DataDir string
InitialCluster string
NewCluster bool
-
- ExternalHost string
- ListenHost string
- ListenPort uint16
+ ExternalHost string
+ ListenHost string
+ ListenPort uint16
}
Member struct {
@@ -97,12 +100,14 @@
consensusServer := &Service{
config: &config,
}
- consensusServer.BaseService = common.NewBaseService("consensus", logger, consensusServer)
+ consensusServer.BaseService = service.NewBaseService("consensus", logger, consensusServer)
return consensusServer, nil
}
func (s *Service) OnStart() error {
+ // See: https://godoc.org/github.com/coreos/etcd/embed#Config
+
if s.config == nil {
return errors.New("config for consensus is nil")
}
@@ -121,17 +126,19 @@
}
s.lastCRL = lastCRL
- // Reset LCUrls because we don't want to expose any client
+ // Reset Listen Client URLs because we don't want to expose any client
cfg.LCUrls = nil
+ // Advertise Peer URLs
apURL, err := url.Parse(fmt.Sprintf("https://%s:%d", s.config.ExternalHost, s.config.ListenPort))
if err != nil {
- return errors.Wrap(err, "invalid external_host or listen_port")
+ return fmt.Errorf("invalid external_host or listen_port: %w", err)
}
+ // Listen Peer URLs
lpURL, err := url.Parse(fmt.Sprintf("https://%s:%d", s.config.ListenHost, s.config.ListenPort))
if err != nil {
- return errors.Wrap(err, "invalid listen_host or listen_port")
+ return fmt.Errorf("invalid listen_host or listen_port: %w", err)
}
cfg.APUrls = []url.URL{*apURL}
cfg.LPUrls = []url.URL{*lpURL}
@@ -160,11 +167,11 @@
// Override the logger
//*server.GetLogger() = *s.Logger.With(zap.String("component", "etcd"))
+ // TODO(leo): can we uncomment this?
go func() {
s.Logger.Info("waiting for etcd to become ready")
<-s.etcd.Server.ReadyNotify()
- s.ready = true
s.Logger.Info("etcd is now ready")
}()
@@ -177,7 +184,10 @@
return nil
}
-func (s *Service) SetupCertificates(certs *api.ConsensusCertificates) error {
+// WriteCertificateFiles writes the given node certificate data to local storage
+// such that it can be used by the embedded etcd server.
+// Unfortunately, we cannot pass the certificates directly to etcd.
+func (s *Service) WriteCertificateFiles(certs *api.ConsensusCertificates) error {
if err := ioutil.WriteFile(filepath.Join(s.config.DataDir, CRLPath), certs.Crl, 0600); err != nil {
return err
}
@@ -196,6 +206,7 @@
return nil
}
+// PrecreateCA generates the etcd cluster certificate authority and writes it to local storage.
func (s *Service) PrecreateCA() error {
// Provision an etcd CA
etcdRootCA, err := ca.New("Smalltown etcd Root CA")
@@ -211,7 +222,7 @@
}
// Preserve certificate for later injection
s.bootstrapCert = cert
- if err := s.SetupCertificates(&api.ConsensusCertificates{
+ if err := s.WriteCertificateFiles(&api.ConsensusCertificates{
Ca: etcdRootCA.CACertRaw,
Crl: etcdRootCA.CRLRaw,
Cert: cert,
@@ -227,14 +238,21 @@
caPathEtcd = "/etcd-ca/ca.der"
caKeyPathEtcd = "/etcd-ca/ca-key.der"
crlPathEtcd = "/etcd-ca/crl.der"
+
+ // This prefix stores the individual certs the etcd CA has issued.
certPrefixEtcd = "/etcd-ca/certs"
)
+// InjectCA copies the CA from data cached during PrecreateCA to etcd.
+// Requires a previous call to PrecreateCA.
func (s *Service) InjectCA() error {
+ if s.bootstrapCA == nil || s.bootstrapCert == nil {
+ panic("bootstrapCA or bootstrapCert are nil - missing PrecreateCA call?")
+ }
if _, err := s.kv.Put(context.Background(), caPathEtcd, string(s.bootstrapCA.CACertRaw)); err != nil {
return err
}
- // TODO: Should be wrapped by the master key
+ // TODO(lorenz): Should be wrapped by the master key
if _, err := s.kv.Put(context.Background(), caKeyPathEtcd, string([]byte(*s.bootstrapCA.PrivateKey))); err != nil {
return err
}
@@ -261,12 +279,12 @@
return nil, -1, fmt.Errorf("failed to get key from etcd: %w", err)
}
if len(res.Kvs) != 1 {
- return nil, -1, errors.New("key not available")
+ return nil, -1, errors.New("key not available or multiple keys returned")
}
return res.Kvs[0].Value, res.Kvs[0].ModRevision, nil
}
-func (s *Service) takeCAOnline() (*ca.CA, int64, error) {
+func (s *Service) getCAFromEtcd() (*ca.CA, int64, error) {
// TODO: Technically this could be done in a single request, but it's more logic
caCert, _, err := s.etcdGetSingle(caPathEtcd)
if err != nil {
@@ -289,7 +307,7 @@
}
func (s *Service) IssueCertificate(hostname string) (*api.ConsensusCertificates, error) {
- idCA, _, err := s.takeCAOnline()
+ idCA, _, err := s.getCAFromEtcd()
if err != nil {
return nil, err
}
@@ -303,6 +321,7 @@
}
serial := hex.EncodeToString(certVal.SerialNumber.Bytes())
if _, err := s.kv.Put(context.Background(), path.Join(certPrefixEtcd, serial), string(cert)); err != nil {
+ // We issued a certificate, but failed to persist it. Return an error and forget it ever happened.
return nil, fmt.Errorf("failed to persist certificate: %w", err)
}
return &api.ConsensusCertificates{
@@ -316,7 +335,7 @@
func (s *Service) RevokeCertificate(hostname string) error {
rand.Seed(time.Now().UnixNano())
for {
- idCA, crlRevision, err := s.takeCAOnline()
+ idCA, crlRevision, err := s.getCAFromEtcd()
if err != nil {
return err
}
@@ -338,6 +357,7 @@
}
}
}
+ // TODO(leo): this needs a test
cmp := clientv3.Compare(clientv3.ModRevision(crlPathEtcd), "=", crlRevision)
op := clientv3.OpPut(crlPathEtcd, string(idCA.CRLRaw))
res, err := s.kv.Txn(context.Background()).If(cmp).Then(op).Commit()
@@ -354,7 +374,7 @@
}
func (s *Service) watchCRL() {
- // TODO: Change etcd client to WatchableKV and make this an actual watch
+ // TODO(lorenz): Change etcd client to WatchableKV and make this an actual watch
// This needs changes in more places, so leaving it now
s.watchCRLTicker = time.NewTicker(30 * time.Second)
for range s.watchCRLTicker.C {
diff --git a/core/internal/iam/BUILD.bazel b/core/internal/iam/BUILD.bazel
deleted file mode 100644
index f737d5c..0000000
--- a/core/internal/iam/BUILD.bazel
+++ /dev/null
@@ -1,22 +0,0 @@
-load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
-
-go_library(
- name = "go_default_library",
- srcs = [
- "capabilities.go",
- "policies.go",
- ],
- importpath = "git.monogon.dev/source/nexantic.git/core/internal/iam",
- visibility = ["//visibility:private"],
- deps = [
- "@com_github_open_policy_agent_opa//ast:go_default_library",
- "@com_github_open_policy_agent_opa//rego:go_default_library",
- "@com_github_open_policy_agent_opa//util:go_default_library",
- ],
-)
-
-go_binary(
- name = "iam",
- embed = [":go_default_library"],
- visibility = ["//:__subpackages__"],
-)
diff --git a/core/internal/iam/README.md b/core/internal/iam/README.md
deleted file mode 100644
index 07635ad..0000000
--- a/core/internal/iam/README.md
+++ /dev/null
@@ -1,102 +0,0 @@
-## Smalltown IAM
-
-There are 4 kinds of elements in Smalltown's Authorization system
-* Identities
- * User
- * Key
- * Module
-* Objects
- * Key
- * Secret
- * Module
-* Policies
-* Permissions
-
-### Identity
-Identities represent an actor that can execute **actions** like editing or interacting with an object.
-
-Identities possess **permissions** and **properties** which can be accessed by policies.
-
-### Objects
-Objects are things that can be interacted with like keys, secrets or modules.
-
-Each object has a **policy** that handles authorization of **actions** performed on it.
-
-When an object is created a default policy is attached which forwards all decisions to the global policy.
-For the first iteration of the system this policy will not be modifiable.
-
-**WARNING**: by modifying a policy, an object could become inaccessible!
-
-### Permissions
-
-Permissions can be assigned to an identity.
-
-| Property | Description | Example |
-|----------|-------------|---------|
-| Allowed Action | Regex specifying the allowed actions | key:meta:edit |
-| Object | Regex specifying the objects this affects | keys:* |
-| Multisig | Number of approvals required | 2 |
-
-Optionally a permission can have a multisig flag that requires N approvals from identities with the same permission.
-
-### Policies
-
-Policies guard actions that are performed on an object.
-
-By default a global policy governs all objects and global actions using an AWS IAM like model.
-
-Potentially a dynamic model using attachable policies could be implemented in the future to allow
-for highly custom models.
-
-A potential graphical representation of a future policy:
-
-
-
-### Global Default Ruleset
-
-This default global policy defines an AWS IAM like permission system.
-
-The following actions are implemented on objects:
-
-| Category | Action | Description | Note |
-|----------|-------------|---------|---------|
-| Object | object:view | Allow to view the object | Cannot be scripted using the policy builder |
-| Object | object:delete | Allow to delete the object |
-| Object | object:attach:normal | Allow to attach the object to a module slot |
-| Object | object:attach:exclusive | Allow to attach the object to an exclusive module slot |
-| Object | object:policy:view | Allow to view the object's attached policy |
-| Object | object:policy:edit | Allow to edit the object's attached policy |
-| Object | object:audit:view | Allow to view the object's audit log |
-| Object:Key | key:sign:eddsa | Allow to sign using the key |
-| Object:Key | key:sign:ecdsa | Allow to sign using the key |
-| Object:Key | key:sign:rsa | Allow to sign using the key |
-| Object:Key | key:encrypt:rsa | Allow to encrypt using the key |
-| Object:Key | key:encrypt:des | Allow to encrypt using the key |
-| Object:Key | key:encrypt:3des| Allow to encrypt using the key |
-| Object:Key | key:encrypt:aes | Allow to encrypt using the key |
-| Object:Key | key:decrypt:rsa | Allow to decrypt using the key |
-| Object:Key | key:decrypt:des | Allow to decrypt using the key |
-| Object:Key | key:decrypt:3des| Allow to decrypt using the key |
-| Object:Key | key:decrypt:aes | Allow to decrypt using the key |
-| Object:Key | key:auth:hmac | Allow to auth messages using the key |
-| Object:Secret | secret:reveal | Allow to reveal a secret to the identity |
-| Object:Module | module:update | Allow to update a module's bytecode | Updates verify the module signature
-| Object:Module | module:config | Allow to configure a module | Assigning objects to slots requires additional permissions on that object
-| Object:Module | module:call:* | Allow to call a function of the module | Function names are defined in the module and vary between modules
-
-The following actions are implemented globally:
-
-| Category | Action | Description | Note |
-|----------|-------------|---------|---------|
-| Object | g:key:generate | Allow to generate a key |
-| Object | g:key:import | Allow to import a key |
-| Object | g:secret:import | Allow to import a secret |
-| Object | g:module:install | Allow to install a module |
-| Object | g:user:create | Allow to create a user |
-| Object | g:user:permission_remove | Allow to create a user | **Privilege Escalation Risk**: Recommend Multisig
-| Object | g:user:permission_add | Allow to create a user | **Privilege Escalation Risk**: Recommend Multisig
-| Object | g:cluster:view | Allow to view cluster nodes
-| Object | g:cluster:add | Allow to add a node to the cluster | **Dangerous**: Recommend Multisig
-| Object | g:cluster:remove | Allow to remove a node from the cluster | **Dangerous**: Recommend Multisig
-| Object | g:config:edit | Allow to edit the global config
-
diff --git a/core/internal/iam/capabilities.go b/core/internal/iam/capabilities.go
deleted file mode 100644
index 1c692a3..0000000
--- a/core/internal/iam/capabilities.go
+++ /dev/null
@@ -1,23 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package main
-
-type (
- Capability struct {
- Name string
- }
-)
diff --git a/core/internal/iam/policies.go b/core/internal/iam/policies.go
deleted file mode 100644
index b17b623..0000000
--- a/core/internal/iam/policies.go
+++ /dev/null
@@ -1,69 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package main
-
-import (
- "context"
- "fmt"
- "github.com/open-policy-agent/opa/ast"
- "github.com/open-policy-agent/opa/rego"
- "github.com/open-policy-agent/opa/util"
-)
-
-type dataSetProfile struct {
- numTokens int
- numPaths int
-}
-
-func main() {
- ctx := context.Background()
- compiler := ast.NewCompiler()
- module := ast.MustParseModule(policy)
-
- compiler.Compile(map[string]*ast.Module{"": module})
- if compiler.Failed() {
- }
-
- r := rego.New(
- rego.Compiler(compiler),
- rego.Input(util.MustUnmarshalJSON([]byte(`{
- "token_id": "deadbeef",
- "path": "mna",
- "method": "GET"
- }`))),
- rego.Query("data.restauthz"),
- )
-
- rs, err := r.Eval(ctx)
- if err != nil {
- panic(err)
- }
- fmt.Printf("%v", rs)
-}
-
-const policy = `package restauthz
-
-default allow = false
-
-allow {
- input.method == "GET"
-}
-
-allow {
- not input.method == "GET"
-}
-`
diff --git a/core/internal/network/BUILD.bazel b/core/internal/network/BUILD.bazel
index db6467a..e7f7d55 100644
--- a/core/internal/network/BUILD.bazel
+++ b/core/internal/network/BUILD.bazel
@@ -6,7 +6,7 @@
importpath = "git.monogon.dev/source/nexantic.git/core/internal/network",
visibility = ["//:__subpackages__"],
deps = [
- "//core/internal/common:go_default_library",
+ "//core/internal/common/service:go_default_library",
"@com_github_insomniacslk_dhcp//dhcpv4/nclient4:go_default_library",
"@com_github_vishvananda_netlink//:go_default_library",
"@org_golang_x_sys//unix:go_default_library",
diff --git a/core/internal/network/main.go b/core/internal/network/main.go
index ecb0d18..b45b8db 100644
--- a/core/internal/network/main.go
+++ b/core/internal/network/main.go
@@ -19,7 +19,7 @@
import (
"context"
"fmt"
- "git.monogon.dev/source/nexantic.git/core/internal/common"
+ "git.monogon.dev/source/nexantic.git/core/internal/common/service"
"net"
"os"
@@ -35,7 +35,7 @@
)
type Service struct {
- *common.BaseService
+ *service.BaseService
config Config
dhcp4Client *nclient4.Client
}
@@ -47,12 +47,12 @@
s := &Service{
config: config,
}
- s.BaseService = common.NewBaseService("network", logger, s)
+ s.BaseService = service.NewBaseService("network", logger, s)
return s, nil
}
func setResolvconf(nameservers []net.IP, searchDomains []string) error {
- os.Mkdir("/etc", 0755) // Error intentionally not checked
+ _ = os.Mkdir("/etc", 0755)
newResolvConf, err := os.Create(resolvConfSwapPath)
if err != nil {
return err
@@ -79,8 +79,8 @@
return err
}
if err := netlink.RouteAdd(&netlink.Route{
- Dst: &net.IPNet{IP: net.IPv4(0, 0, 0, 0), Mask: net.IPv4Mask(0, 0, 0, 0)},
- Gw: gw,
+ Dst: &net.IPNet{IP: net.IPv4(0, 0, 0, 0), Mask: net.IPv4Mask(0, 0, 0, 0)},
+ Gw: gw,
Scope: netlink.SCOPE_UNIVERSE,
}); err != nil {
diff --git a/core/internal/node/BUILD.bazel b/core/internal/node/BUILD.bazel
index 763df36..39b1aca 100644
--- a/core/internal/node/BUILD.bazel
+++ b/core/internal/node/BUILD.bazel
@@ -14,7 +14,6 @@
"//core/internal/common:go_default_library",
"//core/internal/consensus:go_default_library",
"//core/internal/storage:go_default_library",
- "@com_github_casbin_casbin//:go_default_library",
"@com_github_google_uuid//:go_default_library",
"@org_uber_go_zap//:go_default_library",
],
diff --git a/core/internal/node/main.go b/core/internal/node/main.go
index 7494f7a..76a5cf2 100644
--- a/core/internal/node/main.go
+++ b/core/internal/node/main.go
@@ -22,8 +22,8 @@
"git.monogon.dev/source/nexantic.git/core/internal/common"
"git.monogon.dev/source/nexantic.git/core/internal/consensus"
"git.monogon.dev/source/nexantic.git/core/internal/storage"
+ "os"
- "github.com/casbin/casbin"
"github.com/google/uuid"
"go.uber.org/zap"
)
@@ -35,9 +35,9 @@
Storage *storage.Manager
logger *zap.Logger
- ruleEnforcer *casbin.Enforcer
state common.SmalltownState
joinToken string
+ hostname string
}
)
@@ -45,6 +45,11 @@
flag.Parse()
logger.Info("Creating Smalltown node")
+ hostname, err := os.Hostname()
+ if err != nil {
+ panic(err)
+ }
+
storageManager, err := storage.Initialize(logger.With(zap.String("component", "storage")))
if err != nil {
logger.Error("Failed to initialize storage manager", zap.Error(err))
@@ -52,10 +57,9 @@
}
consensusService, err := consensus.NewConsensusService(consensus.Config{
- Name: "test",
- ExternalHost: "0.0.0.0",
- ListenPort: consensusPort,
- ListenHost: "0.0.0.0",
+ Name: hostname,
+ ListenPort: consensusPort,
+ ListenHost: "0.0.0.0",
}, logger.With(zap.String("module", "consensus")))
if err != nil {
return nil, err
@@ -63,8 +67,9 @@
s := &SmalltownNode{
Consensus: consensusService,
- logger: logger,
Storage: storageManager,
+ logger: logger,
+ hostname: hostname,
}
apiService, err := api.NewApiServer(&api.Config{
@@ -91,7 +96,7 @@
return err
}
} else {
- s.logger.Info("Consensus is not provisioned")
+ s.logger.Info("Consensus is not provisioned, starting provisioning...")
err := s.startForSetup()
if err != nil {
return err
diff --git a/core/internal/node/setup.go b/core/internal/node/setup.go
index 5d8953d..efc72d3 100644
--- a/core/internal/node/setup.go
+++ b/core/internal/node/setup.go
@@ -19,6 +19,7 @@
import (
"git.monogon.dev/source/nexantic.git/core/generated/api"
"git.monogon.dev/source/nexantic.git/core/internal/common"
+ "git.monogon.dev/source/nexantic.git/core/internal/storage"
"errors"
@@ -41,19 +42,18 @@
return s.joinToken
}
-func (s *SmalltownNode) SetupNewCluster(name string, externalHost string) error {
+func (s *SmalltownNode) SetupNewCluster() error {
if s.state == common.StateConfigured {
return ErrAlreadySetup
}
- dataPath, err := s.Storage.GetPathInPlace(common.PlaceData, "etcd")
- if err == common.ErrNotInitialized {
+ dataPath, err := s.Storage.GetPathInPlace(storage.PlaceData, "etcd")
+ if err == storage.ErrNotInitialized {
return ErrStorageNotInitialized
} else if err != nil {
return err
}
- s.logger.Info("Setting up a new cluster", zap.String("name", name), zap.String("external_host", externalHost))
-
+ s.logger.Info("Setting up a new cluster")
s.logger.Info("Provisioning consensus")
// Make sure etcd is not yet provisioned
@@ -64,11 +64,11 @@
// Spin up etcd
config := s.Consensus.GetConfig()
config.NewCluster = true
- config.Name = name
- config.ExternalHost = externalHost
+ config.Name = s.hostname
config.DataDir = dataPath
s.Consensus.SetConfig(config)
+ // Generate the cluster CA and store it to local storage.
if err := s.Consensus.PrecreateCA(); err != nil {
return err
}
@@ -78,6 +78,7 @@
return err
}
+ // Now that the cluster is up and running, we can persist the CA to the cluster.
if err := s.Consensus.InjectCA(); err != nil {
return err
}
@@ -101,12 +102,12 @@
return nil
}
-func (s *SmalltownNode) JoinCluster(name string, clusterString string, externalHost string, certs *api.ConsensusCertificates) error {
+func (s *SmalltownNode) JoinCluster(clusterString string, certs *api.ConsensusCertificates) error {
if s.state != common.StateClusterJoinMode {
return ErrNotInJoinMode
}
- s.logger.Info("Joining cluster", zap.String("cluster", clusterString), zap.String("name", name))
+ s.logger.Info("Joining cluster", zap.String("cluster", clusterString))
err := s.SetupBackend()
if err != nil {
@@ -114,11 +115,10 @@
}
config := s.Consensus.GetConfig()
- config.Name = name
+ config.Name = s.hostname
config.InitialCluster = clusterString
- config.ExternalHost = externalHost
s.Consensus.SetConfig(config)
- if err := s.Consensus.SetupCertificates(certs); err != nil {
+ if err := s.Consensus.WriteCertificateFiles(certs); err != nil {
return err
}
diff --git a/core/internal/storage/BUILD.bazel b/core/internal/storage/BUILD.bazel
index 08c27d6..545f9a9 100644
--- a/core/internal/storage/BUILD.bazel
+++ b/core/internal/storage/BUILD.bazel
@@ -6,12 +6,11 @@
"blockdev.go",
"data.go",
"find.go",
- "xfs.go",
+ "storage.go",
],
importpath = "git.monogon.dev/source/nexantic.git/core/internal/storage",
visibility = ["//:__subpackages__"],
deps = [
- "//core/internal/common:go_default_library",
"//core/pkg/devicemapper:go_default_library",
"//core/pkg/sysfs:go_default_library",
"//core/pkg/tpm:go_default_library",
diff --git a/core/internal/storage/data.go b/core/internal/storage/data.go
index e6df103..80af4c9 100644
--- a/core/internal/storage/data.go
+++ b/core/internal/storage/data.go
@@ -18,7 +18,6 @@
import (
"fmt"
- "git.monogon.dev/source/nexantic.git/core/internal/common"
"git.monogon.dev/source/nexantic.git/core/pkg/tpm"
"io/ioutil"
"os"
@@ -148,18 +147,18 @@
// GetPathInPlace returns a path in the given place
// It may return ErrNotInitialized if the place you're trying to access
// is not initialized or ErrUnknownPlace if the place is not known
-func (s *Manager) GetPathInPlace(place common.DataPlace, path string) (string, error) {
+func (s *Manager) GetPathInPlace(place DataPlace, path string) (string, error) {
s.mutex.RLock()
defer s.mutex.RUnlock()
switch place {
- case common.PlaceESP:
+ case PlaceESP:
return filepath.Join(espDataPath, path), nil
- case common.PlaceData:
+ case PlaceData:
if s.dataReady {
return filepath.Join(dataMountPath, path), nil
}
- return "", common.ErrNotInitialized
+ return "", ErrNotInitialized
default:
- return "", common.ErrUnknownPlace
+ return "", ErrUnknownPlace
}
}
diff --git a/core/internal/common/storage.go b/core/internal/storage/storage.go
similarity index 88%
rename from core/internal/common/storage.go
rename to core/internal/storage/storage.go
index caaa155..d0743e8 100644
--- a/core/internal/common/storage.go
+++ b/core/internal/storage/storage.go
@@ -14,7 +14,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-package common
+package storage
import "errors"
@@ -27,9 +27,9 @@
var (
// ErrNotInitialized will be returned when trying to access a place that's not yet initialized
- ErrNotInitialized = errors.New("This place is not initialized")
+ ErrNotInitialized = errors.New("this place is not initialized")
// ErrUnknownPlace will be returned when trying to access a place that's not known
- ErrUnknownPlace = errors.New("This place is not known")
+ ErrUnknownPlace = errors.New("this place is not known")
)
type StorageManager interface {
diff --git a/core/internal/storage/xfs.go b/core/internal/storage/xfs.go
deleted file mode 100644
index 30c6686..0000000
--- a/core/internal/storage/xfs.go
+++ /dev/null
@@ -1,18 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package storage
-
diff --git a/core/pkg/devicemapper/devicemapper.go b/core/pkg/devicemapper/devicemapper.go
index dec6260..2687e3a 100644
--- a/core/pkg/devicemapper/devicemapper.go
+++ b/core/pkg/devicemapper/devicemapper.go
@@ -14,6 +14,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
+// package devicemapper is a thin wrapper for the devicemapper ioctl API.
+// See: https://github.com/torvalds/linux/blob/master/include/uapi/linux/dm-ioctl.h
package devicemapper
import (
@@ -120,13 +122,15 @@
}
}
+// stringToDelimitedBuf copies src to dst and returns an error if len(src) > len(dst),
+// or when the string contains a null byte.
func stringToDelimitedBuf(dst []byte, src string) error {
if len(src) > len(dst)-1 {
- return fmt.Errorf("String longer than target buffer (%v > %v)", len(src), len(dst)-1)
+ return fmt.Errorf("string longer than target buffer (%v > %v)", len(src), len(dst)-1)
}
for i := 0; i < len(src); i++ {
if src[i] == 0x00 {
- return errors.New("String contains null byte, this is unsupported by DM")
+ return errors.New("string contains null byte, this is unsupported by DM")
}
dst[i] = src[i]
}
@@ -139,7 +143,7 @@
if fd == 0 {
f, err := os.Open("/dev/mapper/control")
if os.IsNotExist(err) {
- os.MkdirAll("/dev/mapper", 0755)
+ _ = os.MkdirAll("/dev/mapper", 0755)
if err := unix.Mknod("/dev/mapper/control", unix.S_IFCHR|0600, int(unix.Mkdev(10, 236))); err != nil {
return 0, err
}
@@ -283,11 +287,11 @@
return 0, fmt.Errorf("DM_DEV_CREATE failed: %w", err)
}
if err := LoadTable(name, targets); err != nil {
- RemoveDevice(name)
+ _ = RemoveDevice(name)
return 0, fmt.Errorf("DM_TABLE_LOAD failed: %w", err)
}
if err := Resume(name); err != nil {
- RemoveDevice(name)
+ _ = RemoveDevice(name)
return 0, fmt.Errorf("DM_DEV_SUSPEND failed: %w", err)
}
return dev, nil
diff --git a/core/pkg/tpm/tpm.go b/core/pkg/tpm/tpm.go
index 4638453..e3973d1 100644
--- a/core/pkg/tpm/tpm.go
+++ b/core/pkg/tpm/tpm.go
@@ -36,16 +36,51 @@
)
var (
- // SecureBootPCRs are all PCRs that measure the current Secure Boot configuration
+ // SecureBootPCRs are all PCRs that measure the current Secure Boot configuration.
+ // This is what we want if we rely on secure boot to verify boot integrity. The firmware
+ // hashes the secure boot policy and custom keys into the PCR.
+ //
+ // This requires an extra step that provisions the custom keys.
+ //
+ // Some background: https://mjg59.dreamwidth.org/48897.html?thread=1847297
+ // (the initramfs issue mentioned in the article has been solved by integrating
+ // it into the kernel binary, and we don't have a shim bootloader)
+ //
+ // PCR7 alone is not sufficient - it needs to be combined with firmware measurements.
SecureBootPCRs = []int{7}
// FirmwarePCRs are alle PCRs that contain the firmware measurements
// See https://trustedcomputinggroup.org/wp-content/uploads/TCG_EFI_Platform_1_22_Final_-v15.pdf
- FirmwarePCRs = []int{0, 2, 3}
+ FirmwarePCRs = []int{
+ 0, // platform firmware
+ 2, // option ROM code
+ 3, // option ROM configuration and data
+ }
- // FullSystemPCRs are all PCRs that contain any measurements up to the currently running EFI
- // payload.
- FullSystemPCRs = []int{0, 1, 2, 3, 4}
+ // FullSystemPCRs are all PCRs that contain any measurements up to the currently running EFI payload.
+ FullSystemPCRs = []int{
+ 0, // platform firmware
+ 1, // host platform configuration
+ 2, // option ROM code
+ 3, // option ROM configuration and data
+ 4, // EFI payload
+ }
+
+ // Using FullSystemPCRs is the most secure, but also the most brittle option since updating the EFI
+ // binary, updating the platform firmware, changing platform settings or updating the binary
+ // would invalidate the sealed data. It's annoying (but possible) to predict values for PCR4,
+ // and even more annoying for the firmware PCR (comparison to known values on similar hardware
+ // is the only thing that comes to mind).
+ //
+ // See also: https://github.com/mxre/sealkey (generates PCR4 from EFI image, BSD license)
+ //
+ // Using only SecureBootPCRs is the easiest and still reasonably secure, if we assume that the
+ // platform knows how to take care of itself (i.e. Intel Boot Guard), and that secure boot
+ // is implemented properly. It is, however, a much larger amount of code we need to trust.
+ //
+ // We do not care about PCR 5 (GPT partition table) since modifying it is harmless. All of
+ // the boot options and cmdline are hardcoded in the kernel image, and we use no bootloader,
+ // so there's no PCR for bootloader configuration or kernel cmdline.
)
var (