fietsje: implement
This introduces Fietsje, a little Go dependency manager.
For more information, see third_party/go/fietsje/README.md.
We also bump some dependencies while we're at it, notably, sqliboiler
now uses Go modules. If we weren't to do that, we'd have to add more
heuristics to Fietsje to handle the old version correctly.
Test Plan: fietsje is untested - I'll add some tests to it. Everything else is just regenerating basically the same repositories.bzl file, but with some bumped dependencies.
X-Origin-Diff: phab/D535
GitOrigin-RevId: 4fc919e1bd386bc3f3c1c53e672b1e3b9da17dfc
diff --git a/build/fietsje/BUILD.bazel b/build/fietsje/BUILD.bazel
new file mode 100644
index 0000000..b54b307
--- /dev/null
+++ b/build/fietsje/BUILD.bazel
@@ -0,0 +1,38 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
+
+go_library(
+ name = "go_default_library",
+ srcs = [
+ "dependency.go",
+ "deps_containerd.go",
+ "deps_gvisor.go",
+ "deps_kubernetes.go",
+ "deps_sqlboiler.go",
+ "main.go",
+ "planner.go",
+ "render.go",
+ "shelf.go",
+ "transitive.go",
+ ],
+ importpath = "git.monogon.dev/source/nexantic.git/build/fietsje",
+ visibility = ["//visibility:private"],
+ deps = [
+ "//build/fietsje/proto:go_default_library",
+ "@bazel_gazelle//label:go_default_library",
+ "@com_github_golang_protobuf//proto:go_default_library",
+ "@org_golang_x_mod//modfile:go_default_library",
+ ],
+)
+
+go_binary(
+ name = "fietsje",
+ embed = [":go_default_library"],
+ visibility = ["//visibility:public"],
+)
+
+exports_files(
+ [
+ "fietsje.bash.in",
+ ],
+ visibility = ["//visibility:public"],
+)
diff --git a/build/fietsje/README.md b/build/fietsje/README.md
new file mode 100644
index 0000000..ff990a3
--- /dev/null
+++ b/build/fietsje/README.md
@@ -0,0 +1,169 @@
+Fietsje
+=======
+
+The little Gazelle that could.
+
+Introduction
+------------
+
+Fietsje is a dependency management system for Go dependencies in nxt. It does
+not replace either gomods or Gazelle, but instead builds upon both on them in
+a way that makes sense for our particular usecase: pulling in a large set of
+dependency trees from third\_party projects, and sticking to those as much as
+possible.
+
+When run, Fietsje consults rules written themselves in Go (in `deps_.*go`
+files), and uses this high-level intent to write a `repositories.bzl` file
+that is then consumed by Gazelle. It caches 'locked' versions (ie. Go import
+path and version to a particular checksum) in the Shelf, a text proto file
+that lives alongside `repositories.bzl`. The Shelf should not be modified
+manually.
+
+The effective source of truth used for builds is still the `repositories.bzl`
+file in the actual build path. Definitions in Go are in turn the high-level
+intent that is used to build `repositories.bzl`.
+
+Running
+-------
+
+You should run Fietsje any time you want to update dependencies. The following
+should be a no-op if you haven't changed anything in `deps_*.go`:
+
+ scripts/bin/bazel run //:fietsje
+
+Otherwise, if any definition in build/fietsje/deps_*.go has been changed,
+third_party/go/repositories.bzl will now reflect that.
+
+Fietsje Definition DSL (collect/use/...)
+----------------------------------------
+
+Definitions are kept in pure Go source, with a light DSL focused around a
+'planner' builder.
+
+The builder allows for two kinds of actions:
+ - import a high level dependency (eg. Kubernetes, google/tpm) at a particular
+ version. This is done using the `collect()` call. The dependency will now
+ be part of the build, but its transitive dependencies will not. A special
+ flavor of collect() is collectOverride(), that explicitely allows for
+ overriding a dependency that has already been pulled in by another high
+ level dependency.
+ - enable a transitive dependency defined by a high-level definition using the `use()`
+ call. This can only be done in a `collection` builder context, ie. after a
+ `collect()`/`collectOverride()`call.
+
+In addition, the builder allows to augment a `collection` context with build flags
+(like enabled patches, build tags, etc) that will be applied to the next `.use()`
+call only. This is done by calling `.with()`.
+
+In general, `.collect()`/`.collectOverride()` calls should be limited only to
+dependencies 'we' (as developers) want. These 'high-level' dependencies are
+large projects like Kubernetes, or direct imports from nxt itself. Every
+transitive dependency of those should just be enabled by calling `.use()`,
+instead of another `.collectOverride()` call that might pin it to a wrong
+version.
+
+After updating definitions, run Fietsje as above.
+
+How to: add a new high-level dependency
+---------------------------------------
+
+To add a new high-level dependency, first consider making a new `deps_*.go`
+file for it. If you're pulling in a separate ecosystem of code (ie. a large
+third-party project like kubernetes), it should live in its own file for
+clarity. If you're just pulling in a simple dependency (eg. a library low on
+transitive dependencies) you can drop it into `main.go`.
+
+The first step is to pick a version of the dependency you want to use. If
+possible, pick a tag/release. Otherwise, pick the current master commit hash.
+You can find version information by visiting the project's git repository web
+viewer, or first cloning the repository locally.
+
+Once you've picked a version, add a line like this:
+
+ p.collect("github.com/example/foo", "1.2.3")
+
+If you now re-run Fietsje and rebuild your code, it should be able to link
+against the dependency directly. If this works, you're done. If not, you will
+start getting errors about the newly included library trying to link against
+missing dependencies (ie. external Bazel workspaces). This means you need to
+enable these transitive dependencies for the high-level dependency you've just
+included.
+
+If your high-level dependency contains a go.mod/go.sum file, you can call
+`.use` on the return of the `collect()` call to enable them. Only enable the
+ones that are necessary to build your code. In the future, audit flows might be
+implemented to find and eradicate unused transitive dependencies, while enabling
+ones that are needed - but for now this has to be done manually - usually by a
+cycle of:
+
+ - try to build your code
+ - find missing transitive library, enable via .use()
+ - repeat until code builds
+
+With our previous example, enabling transitive dependencies would look something
+like this:
+
+ p.collect(
+ "github.com/example/foo", "1.2.3",
+ ).use(
+ "github.com/example/libbar",
+ "github.com/example/libbaz",
+ "github.com/golang/glog",
+ )
+
+What this means is that github.com/{example/libbar,example/libbaz,golang/glog}
+will now be available to the build at whatever version example/foo defines them
+in its go.mod/go.sum.
+
+If your high-level dependency is not go.mod/go.sum compatible, you have
+different ways to proceed:
+
+ - if the project uses some alternative resolution/vendoring code, write
+ support for it in transitive.go/`getTransitiveDeps`
+ - otherwise, if you're not in a rush, try to convince and/or send a PR to
+ upstream to enable Go module support
+ - if the dependency has little transitive dependencies, use `.inject()` to
+ add transitive dependencies manually after your `.collect()` call
+ - otherwise, extend fietsje to allow for out-of-tree go.mod/go.sum files kept
+ within nxt, or come up with some other solution.
+
+Your new dependency might conflict with existing dependencies, which usually
+manifests in build failures due to incompatible types. If this happens, you
+will have to start digging to find a way to bring in compatible versions of
+the two dependencies that are interacting with eachother. Do also mention any
+such constraints in code comments near your `.collect()` call.
+
+How to: update a high-level dependency
+--------------------------------------
+
+If you want to update a .collect()/.collectOverride() call, find out the
+version you want to bump to and update it in the call. Re-running fietsje
+will automatically update all enable transitive dependencies. Build and test
+your code. Again, any possible conflicts will have to be resolved manually.
+
+In the future, an audit flow might be provided for checking what the newest
+available version of a high-level dependency is, to allow for easier,
+semi-automated version bumps.
+
+Version resolution conflicts
+----------------------------
+
+Any time a `.collect()`/`.collectOverride()` call is made, Fietsje will note
+what transitive dependencies did the specified high-level dependency request.
+Then, subsequent `.use()` calls will enable these dependencies in the build. On
+subsequent `.collect()`/`.collectOverride()` calls, any transitive dependency
+that already has been pulled in will be ignored, and the existing version will
+be kept.
+
+This means that Fietsje does not detect or handle version conflicts at a granular
+level comparable to gomod. However, it does perform 'well enough', and in general
+the Go ecosystem is stable enough that incompatibilites arise rarely - especially as
+everything moves forward to versioned to go modules, which allow for multiple
+incompatible versions to coexist as fully separate import paths.
+
+It is as such the programmer's job to understand the relationship between imported
+high-level dependencies. In the future, helper heuristics can be included that will
+help understand and reason about dependency relationships. For now, Fietsje will just
+help a user when they call `.use()` on the wrong dependency, ie. when the requested
+transitive dependency has not been pulled in by a given high-level dependency.
+
diff --git a/build/fietsje/def.bzl b/build/fietsje/def.bzl
new file mode 100644
index 0000000..65997f6
--- /dev/null
+++ b/build/fietsje/def.bzl
@@ -0,0 +1,73 @@
+# Copyright 2020 The Monogon Project Authors.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load(
+ "@io_bazel_rules_go//go:def.bzl",
+ _go_context = "go_context",
+ _go_rule = "go_rule",
+)
+load(
+ "@bazel_skylib//lib:shell.bzl",
+ "shell",
+)
+
+def _fietsje_runner_impl(ctx):
+ go = _go_context(ctx)
+ out_file = ctx.actions.declare_file(ctx.label.name + ".bash")
+ substitutions = {
+ "@@GOTOOL@@": shell.quote(go.go.path),
+ "@@FIETSJE_SHORT_PATH@@": shell.quote(ctx.executable.fietsje.short_path),
+ }
+ ctx.actions.expand_template(
+ template = ctx.file._template,
+ output = out_file,
+ substitutions = substitutions,
+ is_executable = True,
+ )
+ runfiles = ctx.runfiles(files = [
+ ctx.executable.fietsje,
+ go.go,
+ ])
+ return [DefaultInfo(
+ files = depset([out_file]),
+ runfiles = runfiles,
+ executable = out_file,
+ )]
+
+_fietsje_runner = _go_rule(
+ implementation = _fietsje_runner_impl,
+ attrs = {
+ "fietsje": attr.label(
+ default = "//build/fietsje",
+ executable = True,
+ cfg = 'host',
+ ),
+ "_template": attr.label(
+ default = "//build/fietsje:fietsje.bash.in",
+ allow_single_file = True,
+ ),
+ },
+)
+
+def fietsje(name):
+ runner_name = name + "-runner"
+ _fietsje_runner(
+ name = runner_name,
+ )
+ native.sh_binary(
+ name = name,
+ srcs = [runner_name],
+ )
diff --git a/build/fietsje/dependency.go b/build/fietsje/dependency.go
new file mode 100644
index 0000000..c6100e4
--- /dev/null
+++ b/build/fietsje/dependency.go
@@ -0,0 +1,155 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "log"
+ "os"
+ "os/exec"
+ "path/filepath"
+
+ "github.com/bazelbuild/bazel-gazelle/label"
+)
+
+// dependency is an external Go package/module, requested by the user of Fietsje directly or indirectly.
+type dependency struct {
+ // importpath is the Go import path that was used to import this dependency.
+ importpath string
+ // version at which this dependency has been requested. This can be in any form that `go get` or the go module
+ // system understands.
+ version string
+
+ // locked is the 'resolved' version of a dependency, containing information about the dependency's hash, etc.
+ locked *locked
+
+ // parent is the dependency that pulled in this one, or nil if pulled in by the user.
+ parent *dependency
+
+ shelf *shelf
+
+ // Build specific settings passed to gazelle.
+ disableProtoBuild bool
+ buildTags []string
+ patches []string
+}
+
+// locked is information about a dependency resolved from the go module system. It is expensive to get, and as such
+// it is cached both in memory (as .locked in a dependency) and in the shelf.
+type locked struct {
+ // bazelName is the external workspace name that Bazel should use for this dependency, eg. com_github_google_glog.
+ bazelName string
+ // sum is the gomod compatible checksum of the depdendency, egh1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs=.
+ sum string
+ // semver is the gomod-compatible version of this dependency. If the dependency was requested by git hash that does
+ // not resolve to a particular release, this will be in the form of v0.0.0-20200520133742-deadbeefcafe.
+ semver string
+}
+
+// child creates a new child dependence for this dependency, ie. one where the 'parent' pointer points to the dependency
+// on which this method is called.
+func (d *dependency) child(importpath, version string) *dependency {
+ return &dependency{
+ importpath: importpath,
+ version: version,
+ shelf: d.shelf,
+ parent: d,
+ }
+}
+
+func (d *dependency) String() string {
+ return fmt.Sprintf("%s@%s", d.importpath, d.version)
+}
+
+// lock ensures that this dependency is locked, which means that it has been resolved to a particular, stable version
+// and VCS details. We lock a dependency by either asking the go module subsystem (via a go module proxy or a download),
+// or by consulting the shelf as a cache.
+func (d *dependency) lock() error {
+ // If already locked in-memory, use that.
+ if d.locked != nil {
+ return nil
+ }
+
+ // If already locked in the shelf, use that.
+ if shelved := d.shelf.get(d.importpath, d.version); shelved != nil {
+ d.locked = shelved
+ return nil
+ }
+
+ // Otherwise, download module.
+ semver, _, sum, err := d.download()
+ if err != nil {
+ return fmt.Errorf("could not download: %v", err)
+ }
+
+ // And resolve its bazelName.
+ name := label.ImportPathToBazelRepoName(d.importpath)
+
+ // Hack for github.com/google/gvisor: it requests @com_github_opencontainers_runtime-spec.
+ // We fix the generated name for this repo so it conforms to what gvisor expects.
+ // TODO(q3k): instead of this, patch gvisor?
+ if name == "com_github_opencontainers_runtime_spec" {
+ name = "com_github_opencontainers_runtime-spec"
+ }
+
+ d.locked = &locked{
+ bazelName: name,
+ sum: sum,
+ semver: semver,
+ }
+ log.Printf("%s: locked to %s", d, d.locked)
+
+ // Save locked version to shelf.
+ d.shelf.put(d.importpath, d.version, d.locked)
+ return d.shelf.save()
+}
+
+func (l *locked) String() string {
+ return fmt.Sprintf("%s@%s", l.bazelName, l.sum)
+}
+
+// download ensures that this dependency is download locally, and returns the download location and the dependency's
+// gomod-compatible sum.
+func (d *dependency) download() (version, dir, sum string, err error) {
+ goroot := os.Getenv("GOROOT")
+ if goroot == "" {
+ err = fmt.Errorf("GOROOT must be set")
+ return
+ }
+ goTool := filepath.Join(goroot, "bin", "go")
+
+ query := fmt.Sprintf("%s@%s", d.importpath, d.version)
+ cmd := exec.Command(goTool, "mod", "download", "-json", "--", query)
+ out, err := cmd.Output()
+ if err != nil {
+ log.Printf("go mod returned: %q", out)
+ err = fmt.Errorf("go mod failed: %v", err)
+ return
+ }
+
+ var res struct{ Version, Sum, Dir string }
+ err = json.Unmarshal(out, &res)
+ if err != nil {
+ return
+ }
+
+ version = res.Version
+ dir = res.Dir
+ sum = res.Sum
+ return
+}
diff --git a/build/fietsje/deps_containerd.go b/build/fietsje/deps_containerd.go
new file mode 100644
index 0000000..0678725
--- /dev/null
+++ b/build/fietsje/deps_containerd.go
@@ -0,0 +1,109 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+func depsContainerd(p *planner) {
+ p.collectOverride(
+ "github.com/containerd/containerd", "8e685f78cf66e2901b2fbed2fdddd64449a74ab9",
+ buildTags("no_zfs", "no_aufs", "no_devicemapper", "no_btrfs"),
+ disabledProtoBuild,
+ ).use(
+ "github.com/BurntSushi/toml",
+ "github.com/Microsoft/go-winio",
+ "github.com/beorn7/perks",
+ "github.com/cespare/xxhash/v2",
+ "github.com/cilium/ebpf",
+ "github.com/containerd/btrfs",
+ "github.com/containerd/console",
+ "github.com/containerd/continuity",
+ "github.com/containerd/fifo",
+ "github.com/containerd/go-cni",
+ "github.com/containerd/go-runc",
+ "github.com/containerd/ttrpc",
+ "github.com/containerd/typeurl",
+ "github.com/containernetworking/cni",
+ "github.com/coreos/go-systemd/v22",
+ "github.com/cpuguy83/go-md2man",
+ "github.com/davecgh/go-spew",
+ "github.com/docker/distribution",
+ "github.com/docker/docker",
+ "github.com/docker/go-events",
+ "github.com/docker/go-metrics",
+ "github.com/docker/go-units",
+ "github.com/docker/spdystream",
+ "github.com/emicklei/go-restful",
+ "github.com/godbus/dbus/v5",
+ "github.com/gogo/protobuf",
+ "github.com/google/gofuzz",
+ "github.com/google/uuid",
+ "github.com/hashicorp/errwrap",
+ "github.com/hashicorp/go-multierror",
+ "github.com/hashicorp/golang-lru",
+ "github.com/imdario/mergo",
+ "github.com/json-iterator/go",
+ "github.com/konsorten/go-windows-terminal-sequences",
+ "github.com/matttproud/golang_protobuf_extensions",
+ "github.com/modern-go/concurrent",
+ "github.com/modern-go/reflect2",
+ "github.com/opencontainers/go-digest",
+ "github.com/opencontainers/image-spec",
+ "github.com/opencontainers/runc",
+ "github.com/opencontainers/runtime-spec",
+ "github.com/pkg/errors",
+ "github.com/prometheus/client_golang",
+ "github.com/prometheus/client_model",
+ "github.com/prometheus/common",
+ "github.com/prometheus/procfs",
+ "github.com/russross/blackfriday",
+ "github.com/seccomp/libseccomp-golang",
+ "github.com/sirupsen/logrus",
+ "github.com/stretchr/testify",
+ "github.com/syndtr/gocapability",
+ "github.com/tchap/go-patricia",
+ "github.com/urfave/cli",
+ "go.etcd.io/bbolt",
+ "go.opencensus.io",
+ "golang.org/x/crypto",
+ "golang.org/x/oauth2",
+ "golang.org/x/sync",
+ "golang.org/x/sys",
+ "google.golang.org/genproto",
+ "gopkg.in/inf.v0",
+ "gopkg.in/yaml.v2",
+ "gotest.tools",
+ "k8s.io/klog",
+ "sigs.k8s.io/yaml",
+ ).with(disabledProtoBuild).use(
+ "github.com/Microsoft/hcsshim",
+ "github.com/containerd/cgroups",
+ "github.com/containerd/cri",
+ "github.com/gogo/googleapis",
+ ).with(buildTags("selinux")).use(
+ "github.com/opencontainers/selinux",
+ )
+
+ // containernetworking/plugins
+ p.collectOverride(
+ "github.com/containernetworking/plugins", "v0.8.2",
+ patches("cni-plugins-build.patch"),
+ ).use(
+ "github.com/alexflint/go-filemutex",
+ "github.com/coreos/go-iptables",
+ "github.com/j-keck/arping",
+ "github.com/safchain/ethtool",
+ )
+}
diff --git a/build/fietsje/deps_gvisor.go b/build/fietsje/deps_gvisor.go
new file mode 100644
index 0000000..1856aa7
--- /dev/null
+++ b/build/fietsje/deps_gvisor.go
@@ -0,0 +1,41 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+func depsGVisor(p *planner) {
+ p.collect(
+ "github.com/google/gvisor", "release-20200511.0",
+ patches("gvisor.patch"),
+ ).use(
+ "github.com/cenkalti/backoff",
+ "github.com/gofrs/flock",
+ "github.com/google/subcommands",
+ "github.com/kr/pretty",
+ "github.com/kr/pty",
+ "golang.org/x/time",
+ )
+ // gRPC is used by gvisor's bazel machinery, but not present in go.sum. Include it manually.
+ p.collect("github.com/grpc/grpc", "v1.26.0")
+
+ p.collect(
+ "github.com/google/gvisor-containerd-shim", "v0.0.4",
+ patches(
+ "gvisor-containerd-shim.patch", "gvisor-containerd-shim-build.patch",
+ "gvisor-containerd-shim-nogo.patch", "gvisor-shim-root.patch",
+ ),
+ )
+}
diff --git a/build/fietsje/deps_kubernetes.go b/build/fietsje/deps_kubernetes.go
new file mode 100644
index 0000000..fe976b1
--- /dev/null
+++ b/build/fietsje/deps_kubernetes.go
@@ -0,0 +1,151 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+func depsKubernetes(p *planner) {
+ // containerd and its deps
+ p.collect(
+ "k8s.io/kubernetes", "v1.19.0-alpha.2",
+ buildTags("providerless"),
+ disabledProtoBuild,
+ patches("k8s-kubernetes.patch", "k8s-kubernetes-build.patch"),
+ ).inject(
+ // repo infra, not requested by k8s, but used with bazel
+ "k8s.io/repo-infra", "df02ded38f9506e5bbcbf21702034b4fef815f2f",
+ ).with(patches("k8s-client-go.patch", "k8s-client-go-build.patch")).use(
+ "k8s.io/client-go",
+ ).use(
+ "k8s.io/cli-runtime",
+ "k8s.io/client-go",
+ "k8s.io/cloud-provider",
+ "k8s.io/cluster-bootstrap",
+ "k8s.io/component-base",
+ "k8s.io/csi-translation-lib",
+ "k8s.io/kube-controller-manager",
+ "k8s.io/kube-proxy",
+ "k8s.io/kube-scheduler",
+ "k8s.io/kubectl",
+ "k8s.io/legacy-cloud-providers",
+ "k8s.io/sample-apiserver",
+ ).with(disabledProtoBuild).use(
+ "k8s.io/api",
+ "k8s.io/apiextensions-apiserver",
+ "k8s.io/apimachinery",
+ "k8s.io/apiserver",
+ "k8s.io/cri-api",
+ "k8s.io/kube-aggregator",
+ "k8s.io/kubelet",
+ "k8s.io/metrics",
+ ).use(
+ "cloud.google.com/go",
+ "github.com/Azure/go-ansiterm",
+ "github.com/MakeNowJust/heredoc",
+ "github.com/NYTimes/gziphandler",
+ "github.com/PuerkitoBio/purell",
+ "github.com/PuerkitoBio/urlesc",
+ "github.com/armon/circbuf",
+ "github.com/asaskevich/govalidator",
+ "github.com/bgentry/speakeasy",
+ "github.com/blang/semver",
+ "github.com/chai2010/gettext-go",
+ "github.com/checkpoint-restore/go-criu",
+ "github.com/container-storage-interface/spec",
+ "github.com/coreos/go-oidc",
+ "github.com/coreos/go-semver",
+ "github.com/coreos/go-systemd",
+ "github.com/coreos/pkg",
+ "github.com/cyphar/filepath-securejoin",
+ "github.com/daviddengcn/go-colortext",
+ "github.com/dgrijalva/jwt-go",
+ "github.com/docker/go-connections",
+ "github.com/dustin/go-humanize",
+ "github.com/euank/go-kmsg-parser",
+ "github.com/evanphx/json-patch",
+ "github.com/exponent-io/jsonpath",
+ "github.com/fatih/camelcase",
+ "github.com/fatih/color",
+ "github.com/ghodss/yaml",
+ "github.com/go-openapi/analysis",
+ "github.com/go-openapi/errors",
+ "github.com/go-openapi/jsonpointer",
+ "github.com/go-openapi/jsonreference",
+ "github.com/go-openapi/loads",
+ "github.com/go-openapi/runtime",
+ "github.com/go-openapi/spec",
+ "github.com/go-openapi/strfmt",
+ "github.com/go-openapi/swag",
+ "github.com/go-openapi/validate",
+ "github.com/go-stack/stack",
+ "github.com/golang/groupcache",
+ "github.com/google/btree",
+ "github.com/google/go-cmp",
+ "github.com/googleapis/gnostic",
+ "github.com/gorilla/websocket",
+ "github.com/gregjones/httpcache",
+ "github.com/grpc-ecosystem/go-grpc-middleware",
+ "github.com/grpc-ecosystem/go-grpc-prometheus",
+ "github.com/grpc-ecosystem/grpc-gateway",
+ "github.com/jonboulle/clockwork",
+ "github.com/karrick/godirwalk",
+ "github.com/liggitt/tabwriter",
+ "github.com/lithammer/dedent",
+ "github.com/mailru/easyjson",
+ "github.com/mattn/go-colorable",
+ "github.com/mattn/go-isatty",
+ "github.com/mattn/go-runewidth",
+ "github.com/mindprince/gonvml",
+ "github.com/mitchellh/go-wordwrap",
+ "github.com/mitchellh/mapstructure",
+ "github.com/moby/term",
+ "github.com/morikuni/aec",
+ "github.com/mrunalp/fileutils",
+ "github.com/munnerz/goautoneg",
+ "github.com/mxk/go-flowrate",
+ "github.com/olekukonko/tablewriter",
+ "github.com/peterbourgon/diskv",
+ "github.com/pquerna/cachecontrol",
+ "github.com/robfig/cron",
+ "github.com/soheilhy/cmux",
+ "github.com/spf13/afero",
+ "github.com/spf13/cobra",
+ "github.com/spf13/pflag",
+ "github.com/tmc/grpc-websocket-proxy",
+ "github.com/vishvananda/netlink",
+ "github.com/vishvananda/netns",
+ "github.com/xiang90/probing",
+ "go.mongodb.org/mongo-driver",
+ "go.uber.org/atomic",
+ "go.uber.org/multierr",
+ "go.uber.org/zap",
+ "golang.org/x/xerrors",
+ "gonum.org/v1/gonum",
+ "gopkg.in/natefinch/lumberjack.v2",
+ "gopkg.in/square/go-jose.v2",
+ "k8s.io/gengo",
+ "k8s.io/heapster",
+ "k8s.io/kube-openapi",
+ "k8s.io/utils",
+ "sigs.k8s.io/apiserver-network-proxy/konnectivity-client",
+ "sigs.k8s.io/kustomize",
+ "sigs.k8s.io/structured-merge-diff/v3",
+ "vbom.ml/util",
+ ).with(patches("cadvisor.patch", "cadvisor-build.patch")).use(
+ "github.com/google/cadvisor",
+ ).with(disabledProtoBuild).use(
+ "go.etcd.io/etcd",
+ )
+}
diff --git a/build/fietsje/deps_sqlboiler.go b/build/fietsje/deps_sqlboiler.go
new file mode 100644
index 0000000..3b167a8
--- /dev/null
+++ b/build/fietsje/deps_sqlboiler.go
@@ -0,0 +1,57 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+func depsSQLBoiler(p *planner) {
+ p.collect(
+ "github.com/volatiletech/sqlboiler/v4", "v4.1.1",
+ ).use(
+ "github.com/denisenkom/go-mssqldb",
+ "github.com/ericlagergren/decimal",
+ "github.com/friendsofgo/errors",
+ "github.com/go-sql-driver/mysql",
+ "github.com/golang-sql/civil",
+ "github.com/hashicorp/hcl",
+ "github.com/lib/pq",
+ "github.com/magiconair/properties",
+ "github.com/spf13/cast",
+ "github.com/spf13/cobra",
+ "github.com/spf13/jwalterweatherman",
+ "github.com/spf13/viper",
+ "github.com/subosito/gotenv",
+ "github.com/volatiletech/inflect",
+ "github.com/volatiletech/null/v8",
+ "github.com/volatiletech/randomize",
+ "github.com/volatiletech/strmangle",
+ "gopkg.in/ini.v1",
+ )
+ // required by //build/sqlboiler autogeneration
+ p.collect(
+ "github.com/glerchundi/sqlboiler-crdb/v4", "d540ee52783ebbbfe010acc5d91a9043d88de3fd",
+ ).use(
+ "github.com/gofrs/uuid",
+ )
+ p.collect(
+ "github.com/rubenv/sql-migrate", "ae26b214fa431c314a5a9b986d5c90fb1719c68d",
+ ).use(
+ "github.com/armon/go-radix",
+ "github.com/mattn/go-sqlite3",
+ "github.com/mitchellh/cli",
+ "github.com/posener/complete",
+ "gopkg.in/gorp.v1",
+ )
+}
diff --git a/build/fietsje/fietsje.bash.in b/build/fietsje/fietsje.bash.in
new file mode 100644
index 0000000..bad5f8e
--- /dev/null
+++ b/build/fietsje/fietsje.bash.in
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# Most of this comes from github.com/bazelbuild/bazel-gazelle/internal/gazelle.bash.in.
+# It's very hacky.
+
+# Copyright 2017 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FIETSJE_SHORT_PATH=@@FIETSJE_SHORT_PATH@@
+GOTOOL=@@GOTOOL@@
+
+# find_runfile prints the location of a runfile in the source workspace,
+# either by reading the symbolic link or reading the runfiles manifest.
+function find_runfile {
+ local runfile=$1
+ if [ -f "$runfile" ]; then
+ readlink "$runfile"
+ return
+ fi
+ runfile=$(echo "$runfile" | sed -e 's!^\(\.\./\|external/\)!!')
+ if grep -q "^$runfile" MANIFEST; then
+ grep "^$runfile" MANIFEST | head -n 1 | cut -d' ' -f2
+ return
+ fi
+ # printing nothing indicates failure
+}
+
+# bazel_build_get_path builds a given target and prints the absolute path
+# to the generated binary. This only works for rules that produce a single file.
+function bazel_build_get_path {
+ local build_log=$(mktemp fietsje_build.XXXX.json --tmpdir)
+ bazel build --build_event_json_file="$build_log" "$1"
+ grep "^{\"id\":{\"targetCompleted\":{\"label\":\"$1\"" "$build_log" | \
+ sed -e 's!^.*file://\([^"]*\).*$!\1!'
+ rm -f "$build_log"
+}
+
+# set_goroot attempts to set GOROOT to the SDK used by rules_go. fietsje
+# invokes tools inside the Go SDK for dependency management. It's good to
+# use the SDK used by the workspace in case the Go SDK is not installed
+# on the host system or is a different version.
+function set_goroot {
+ local gotool=$(find_runfile "$GOTOOL")
+ if [ -z "$gotool" ]; then
+ echo "$0: warning: could not locate GOROOT used by rules_go" >&2
+ return
+ fi
+ export GOROOT=$(cd "$(dirname "$gotool")/.."; pwd)
+ if type cygpath >/dev/null 2>&1; then
+ # On Windows, convert the path to something usable outside of bash.
+ GOROOT=$(cygpath -w "$GOROOT")
+ fi
+}
+
+set_goroot
+fietsje_short_path=$(find_runfile "$FIETSJE_SHORT_PATH")
+if [ -z "$fietsje_short_path" ]; then
+ echo "error: could not locate fietsje binary" >&2
+ exit 1
+fi
+if [ -z "${BUILD_WORKSPACE_DIRECTORY-}" ]; then
+ echo "error: BUILD_WORKSPACE_DIRECOTRY not set" >&2
+ exit 1
+fi
+cd "$BUILD_WORKSPACE_DIRECTORY"
+"$fietsje_short_path" -shelf_path third_party/go/shelf.pb.text -repositories_bzl third_party/go/repositories.bzl
diff --git a/build/fietsje/main.go b/build/fietsje/main.go
new file mode 100644
index 0000000..a5f8e46
--- /dev/null
+++ b/build/fietsje/main.go
@@ -0,0 +1,110 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "bytes"
+ "flag"
+ "io/ioutil"
+ "log"
+)
+
+var (
+ flagShelfPath string
+ flagRepositoresBzlPath string
+)
+
+func main() {
+ flag.StringVar(&flagShelfPath, "shelf_path", "", "Path to shelf (cache/lockfile)")
+ flag.StringVar(&flagRepositoresBzlPath, "repositories_bzl", "", "Path to output repositories.bzl file")
+ flag.Parse()
+
+ if flagShelfPath == "" {
+ log.Fatalf("shelf_path must be set")
+ }
+ if flagRepositoresBzlPath == "" {
+ log.Fatalf("repositories_bzl must be set")
+ }
+
+ shelf, err := shelfLoad(flagShelfPath)
+ if err != nil {
+ log.Fatalf("could not load shelf: %v", err)
+ }
+
+ p := &planner{
+ available: make(map[string]*dependency),
+ enabled: make(map[string]bool),
+ seen: make(map[string]string),
+
+ shelf: shelf,
+ }
+
+ // gRPC/proto deps (https://github.com/bazelbuild/rules_go/blob/master/go/workspace.rst#id8)
+ // bump down from 1.28.1 to 1.26.0 because https://github.com/etcd-io/etcd/issues/11563
+ p.collect(
+ "google.golang.org/grpc", "v1.26.0",
+ ).use(
+ "golang.org/x/net",
+ "golang.org/x/text",
+ )
+
+ depsKubernetes(p)
+ depsContainerd(p)
+ depsGVisor(p)
+ depsSQLBoiler(p)
+
+ // our own deps, common
+ p.collectOverride("go.uber.org/zap", "v1.15.0")
+ p.collectOverride("golang.org/x/mod", "v0.3.0")
+ p.collect("github.com/cenkalti/backoff/v4", "v4.0.2")
+
+ p.collect("github.com/google/go-tpm", "ae6dd98980d4")
+ p.collect("github.com/google/go-tpm-tools", "f8c04ff88181")
+ p.collect("github.com/insomniacslk/dhcp", "5dd7202f19711228cb4a51aa8b3415421c2edefe")
+ p.collect("github.com/mdlayher/ethernet", "0394541c37b7f86a10e0b49492f6d4f605c34163").use(
+ "github.com/mdlayher/raw",
+ )
+ p.collect("github.com/rekby/gpt", "a930afbc6edcc89c83d39b79e52025698156178d")
+ p.collect("github.com/yalue/native_endian", "51013b03be4fd97b0aabf29a6923e60359294186")
+
+ // used by insomniacslk/dhcp for pkg/uio
+ p.collect("github.com/u-root/u-root", "v6.0.0")
+
+ // used by //core/cmd/mkimage
+ p.collect("github.com/diskfs/go-diskfs", "v1.0.0").use(
+ "gopkg.in/djherbis/times.v1",
+ )
+ // used by //build/bindata
+ p.collect("github.com/kevinburke/go-bindata", "v3.16.0")
+
+ // used by deltagen
+ p.collect("github.com/lyft/protoc-gen-star", "v0.4.14")
+
+ // First generate the repositories starlark rule into memory. This is because rendering will lock all unlocked
+ // dependencies, which might take a while. If a use were to interrupt it now, they would end up with an incomplete
+ // repositories.bzl and would have to restore from git.
+ buf := bytes.NewBuffer(nil)
+ err = p.render(buf)
+ if err != nil {
+ log.Fatalf("could not render deps: %v", err)
+ }
+
+ err = ioutil.WriteFile(flagRepositoresBzlPath, buf.Bytes(), 0666)
+ if err != nil {
+ log.Fatalf("could not write deps: %v", err)
+ }
+}
diff --git a/build/fietsje/planner.go b/build/fietsje/planner.go
new file mode 100644
index 0000000..4e67c2d
--- /dev/null
+++ b/build/fietsje/planner.go
@@ -0,0 +1,194 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "fmt"
+)
+
+// The Planner provides the main DSL and high-level control logic for resolving dependencies. It is the main API that
+// fietsje users should consume.
+
+// planner is a builder for a single world of Go package dependencies, and what is then emitted into a Starlark file
+// containing gazelle go_repository rules.
+// The planner's builder system covers three increasingly specific contextx:
+// - planner (this structure, allows for 'collecting' in high-level dependencies. ie. collections)
+// - collection (represents what has been pulled in by a high-level dependency, and allows for 'using' transitive
+// dependencies from a collection)
+// - optionized (represents a collection with extra build flags, eg. disabled proto builds)
+type planner struct {
+ // available is a map of importpaths to dependencies that the planner knows. This is a flat structure that is the
+ // main source of truth of actual dependency data, like a registry of everything that the planner knows about.
+ // The available dependency for a given importpath, as the planner progresses, might change, ie. when there is a
+ // version conflict. As such, code should use importpaths as atoms describing dependencies, instead of holding
+ // dependency pointers.
+ available map[string]*dependency
+ // enabled is a map of dependencies that will be emitted by the planner into the build via Gazelle.
+ enabled map[string]bool
+ // seen is a map of 'dependency' -> 'parent' importpaths, ie. returns what higher-level dependency (ie. one enabled
+ // with .collect()) pulled in a given dependency. This is only used for error messages to help the user find what
+ // a transitive dependency has been pulled in by.
+ seen map[string]string
+
+ shelf *shelf
+}
+
+func (p *planner) collect(importpath, version string, opts ...buildOpt) *collection {
+ return p.collectInternal(importpath, version, false, opts...)
+}
+
+func (p *planner) collectOverride(importpath, version string, opts ...buildOpt) *collection {
+ return p.collectInternal(importpath, version, true, opts...)
+}
+
+// collectInternal pulls in a high-level dependency into the planner and
+// enables it. It also parses all of its transitive // dependencies (not just
+// directly transitive, but recursively transitive) and makes the planner aware
+// of them. It does not enable these transitive dependencies, but returns a
+// collection builder, which can be used to do se by calling .use().
+func (p *planner) collectInternal(importpath, version string, override bool, opts ...buildOpt) *collection {
+ // Ensure overrides are explicit and minimal.
+ by, ok := p.seen[importpath]
+ if ok && !override {
+ panic(fmt.Errorf("%s is being collected, but has already been declared by %s; replace it by a use(%q) call on %s or use collectOverride", importpath, by, importpath, by))
+ }
+ if !ok && override {
+ panic(fmt.Errorf("%s is being collected with override, but has not been seen as a dependency previously - use .collect(%q, %q) instead", importpath, importpath, version))
+ }
+
+ d := &dependency{
+ shelf: p.shelf,
+ importpath: importpath,
+ version: version,
+ }
+ for _, o := range opts {
+ o(d)
+ }
+
+ // automatically enable direct import
+ p.enabled[d.importpath] = true
+ p.available[d.importpath] = d
+
+ td, err := d.getTransitiveDeps()
+ if err != nil {
+ panic(fmt.Errorf("could not get transitive deps for %q: %v", d.importpath, err))
+ }
+ // add transitive deps to 'available' map
+ for k, v := range td {
+ // skip dependencies that have already been enabled, dependencies are 'first enabled version wins'.
+ if _, ok := p.available[k]; ok && p.enabled[k] {
+ continue
+ }
+
+ p.available[k] = v
+
+ // make note of the high-level dependency that pulled in the dependency.
+ p.seen[v.importpath] = d.importpath
+ }
+
+ return &collection{
+ p: p,
+ highlevel: d,
+ transitive: td,
+ }
+}
+
+// collection represents the context of the planner after pulling/collecting in a high-level dependency. In this state,
+// the planner can be used to enable transitive dependencies of the high-level dependency.
+type collection struct {
+ p *planner
+
+ highlevel *dependency
+ transitive map[string]*dependency
+}
+
+// use enables given dependencies defined in the collection by a high-level dependency.
+func (c *collection) use(paths ...string) *collection {
+ return c.with().use(paths...)
+}
+
+// inject adds a dependency to a collection as if requested by the high-level dependency of the collection. This should
+// be used sparingly, for instance when high-level dependencies contain bazel code that uses some external workspaces
+// from Go modules, and those workspaces are not defined in parsed transitive dependency definitions like go.mod/sum.
+func (c *collection) inject(importpath, version string) *collection {
+ d := c.highlevel.child(importpath, version)
+ c.transitive[importpath] = d
+ c.p.available[importpath] = d
+ c.p.enabled[importpath] = true
+
+ return c
+}
+
+// with transforms a collection into an optionized, by setting some build options.
+func (c *collection) with(o ...buildOpt) *optionized {
+ return &optionized{
+ c: c,
+ opts: o,
+ }
+}
+
+// optionized is a collection that has some build options set, that will be applied to all dependencies 'used' in this
+// context
+type optionized struct {
+ c *collection
+ opts []buildOpt
+}
+
+// buildOpt is a build option passed to Gazelle.
+type buildOpt func(d *dependency)
+
+// buildTags sets the given buildTags in affected dependencies.
+func buildTags(tags ...string) buildOpt {
+ return func(d *dependency) {
+ d.buildTags = tags
+ }
+}
+
+// disabledProtoBuild disables protobuf builds in affected dependencies.
+func disabledProtoBuild(d *dependency) {
+ d.disableProtoBuild = true
+}
+
+// patches applies patches in affected dependencies after BUILD file generation.
+func patches(patches ...string) buildOpt {
+ return func(d *dependency) {
+ d.patches = patches
+ }
+}
+
+// use enables given dependencies defined in the collection by a high-level dependency, with any set build options.
+// After returning, the builder degrades to a collection - ie, all build options are reset.
+func (o *optionized) use(paths ...string) *collection {
+ for _, path := range paths {
+ el, ok := o.c.transitive[path]
+ if !ok {
+ msg := fmt.Sprintf("dependency %q not found in %q", path, o.c.highlevel.importpath)
+ if alternative, ok := o.c.p.seen[path]; ok {
+ msg += fmt.Sprintf(" (but found in %q)", alternative)
+ } else {
+ msg += " or any other collected library"
+ }
+ panic(msg)
+ }
+ for _, o := range o.opts {
+ o(el)
+ }
+ o.c.p.enabled[path] = true
+ }
+
+ return o.c
+}
diff --git a/build/fietsje/proto/BUILD.bazel b/build/fietsje/proto/BUILD.bazel
new file mode 100644
index 0000000..816164b
--- /dev/null
+++ b/build/fietsje/proto/BUILD.bazel
@@ -0,0 +1,23 @@
+load("@rules_proto//proto:defs.bzl", "proto_library")
+load("@io_bazel_rules_go//go:def.bzl", "go_library")
+load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")
+
+proto_library(
+ name = "proto_proto",
+ srcs = ["shelf.proto"],
+ visibility = ["//visibility:public"],
+)
+
+go_proto_library(
+ name = "proto_go_proto",
+ importpath = "git.monogon.dev/source/nexantic.git/build/fietsje/proto",
+ proto = ":proto_proto",
+ visibility = ["//visibility:public"],
+)
+
+go_library(
+ name = "go_default_library",
+ embed = [":proto_go_proto"],
+ importpath = "git.monogon.dev/source/nexantic.git/build/fietsje/proto",
+ visibility = ["//visibility:public"],
+)
diff --git a/build/fietsje/proto/shelf.proto b/build/fietsje/proto/shelf.proto
new file mode 100644
index 0000000..394e5f0
--- /dev/null
+++ b/build/fietsje/proto/shelf.proto
@@ -0,0 +1,33 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto3";
+package nexantic.build.fietsje;
+option go_package = "git.monogon.dev/source/nexantic.git/build/fietsje/proto";
+
+// The Shelf is a cache/lockfile of Fietsje dependencies. See //build/fietsje/shelf.go for more information.
+// This proto definition is used to generate text proto files. As such, repeated fields have singular names.
+
+message Shelf {
+ message Entry {
+ string import_path = 1;
+ string version = 2;
+ string bazel_name = 3;
+ string sum = 4;
+ string semver = 5;
+ }
+ repeated Entry entry = 1;
+}
diff --git a/build/fietsje/render.go b/build/fietsje/render.go
new file mode 100644
index 0000000..d06e530
--- /dev/null
+++ b/build/fietsje/render.go
@@ -0,0 +1,72 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "fmt"
+ "io"
+ "sort"
+)
+
+// render writes a gazelle-compatible starlark file based on the enabled dependencies in this planner.
+func (p *planner) render(w io.Writer) error {
+ fmt.Fprintln(w, `load("@bazel_gazelle//:deps.bzl", "go_repository")`)
+ fmt.Fprintln(w, ``)
+ fmt.Fprintln(w, `def go_repositories():`)
+
+ // Get and sort all enabled importpaths.
+ var enabled []string
+ for importpath, _ := range p.enabled {
+ enabled = append(enabled, importpath)
+ }
+ sort.Slice(enabled, func(i, j int) bool { return enabled[i] < enabled[j] })
+
+ // Render all importpaths.
+ for _, importpath := range enabled {
+ d := p.available[importpath]
+ if err := d.lock(); err != nil {
+ return fmt.Errorf("could not lock %q: %v", importpath, err)
+ }
+
+ fmt.Fprintf(w, " go_repository(\n")
+ fmt.Fprintf(w, " name = %q,\n", d.locked.bazelName)
+ fmt.Fprintf(w, " importpath = %q,\n", d.importpath)
+ fmt.Fprintf(w, " version = %q,\n", d.locked.semver)
+ fmt.Fprintf(w, " sum = %q,\n", d.locked.sum)
+ if d.disableProtoBuild {
+ fmt.Fprintf(w, " build_file_proto_mode = %q,\n", "disable")
+ }
+ if d.buildTags != nil {
+ fmt.Fprintf(w, " build_tags = [\n")
+ for _, tag := range d.buildTags {
+ fmt.Fprintf(w, " %q,\n", tag)
+ }
+ fmt.Fprintf(w, " ],\n")
+ }
+ if d.patches != nil {
+ fmt.Fprintf(w, " patches = [\n")
+ for _, patch := range d.patches {
+ fmt.Fprintf(w, " %q,\n", "//third_party/go/patches:"+patch)
+ }
+ fmt.Fprintf(w, " ],\n")
+ fmt.Fprintf(w, " patch_args = [%q],\n", "-p1")
+ }
+
+ fmt.Fprintf(w, " )\n")
+ }
+ return nil
+}
diff --git a/build/fietsje/shelf.go b/build/fietsje/shelf.go
new file mode 100644
index 0000000..d3a1a1a
--- /dev/null
+++ b/build/fietsje/shelf.go
@@ -0,0 +1,157 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "bytes"
+ "fmt"
+ "io/ioutil"
+ "log"
+ "os"
+ "sort"
+
+ "github.com/golang/protobuf/proto"
+
+ pb "git.monogon.dev/source/nexantic.git/build/fietsje/proto"
+)
+
+// The Shelf is a combined cache and dependency lockfile, not unlike go.sum. It's implemented as a text proto file on
+// disk, and currently stores a single mapping of shelfKeys to shelfValues, which are in order a (importpath, version)
+// tuple and the `locked` structure of a dependency.
+// The resulting shelf file should be commited to the nxt repository. It can be freely deleted to force recreation from
+// scratch, which can be useful as there is no garbage collection implemented for it.
+// The 'lockfile' aspect of the Shelf is counter-intuitive to what readers might be used to from other dependency
+// management systems. It does not lock a third-party dependency to a particular version, but only locks a well defined
+// version to its checksum. As such, recreating the shelf from scratch should not bump any dependencies, unless some
+// upstream-project retagged a release to a different VCS commit, or a fietsje user pinned to 'master' instead of a
+// particular commit. The effective changes will always be reflected in the resulting starlark repository ruleset,
+// which (also being commited to source control) can be used as a canary of a version being effectively bumped.
+
+// shelfKey is the key into the shelf map structure.
+type shelfKey struct {
+ importpath string
+ version string
+}
+
+// shelfValue is the entry of a shelf map structure.
+type shelfValue struct {
+ l *locked
+}
+
+// shelf is an in-memory representation of the shelf loaded from disk.
+type shelf struct {
+ path string
+ data map[shelfKey]shelfValue
+}
+
+func shelfLoad(path string) (*shelf, error) {
+ var data []byte
+ var err error
+
+ if _, err := os.Stat(path); os.IsNotExist(err) {
+ log.Printf("Creating new shelf file at %q, this run will be slow.", path)
+ } else {
+ data, err = ioutil.ReadFile(path)
+ if err != nil {
+ return nil, fmt.Errorf("could not read shelf: %v", err)
+ }
+ }
+ var shelfProto pb.Shelf
+ err = proto.UnmarshalText(string(data), &shelfProto)
+ if err != nil {
+ return nil, fmt.Errorf("could not unmarshal shelf: %v", err)
+ }
+
+ res := &shelf{
+ path: path,
+ data: make(map[shelfKey]shelfValue),
+ }
+
+ for _, e := range shelfProto.Entry {
+ k := shelfKey{
+ importpath: e.ImportPath,
+ version: e.Version,
+ }
+ v := shelfValue{
+ l: &locked{
+ bazelName: e.BazelName,
+ sum: e.Sum,
+ semver: e.Semver,
+ },
+ }
+ res.data[k] = v
+ }
+ return res, nil
+}
+
+// get retrieves a given lock entry from the in-memory shelf.
+func (s *shelf) get(importpath, version string) *locked {
+ res, ok := s.data[shelfKey{importpath: importpath, version: version}]
+ if !ok {
+ return nil
+ }
+ return res.l
+}
+
+// put stores a given locked entry in memory. This will not be commited to disk until .save() is called.
+func (s *shelf) put(importpath, version string, l *locked) {
+ s.data[shelfKey{importpath: importpath, version: version}] = shelfValue{l: l}
+}
+
+// save commits the shelf to disk (to the same location it was loaded from), fully overwriting from in-memory data.
+func (s *shelf) save() error {
+ // Build proto representation of shelf data.
+ var shelfProto pb.Shelf
+ for k, v := range s.data {
+ shelfProto.Entry = append(shelfProto.Entry, &pb.Shelf_Entry{
+ ImportPath: k.importpath,
+ Version: k.version,
+ BazelName: v.l.bazelName,
+ Sum: v.l.sum,
+ Semver: v.l.semver,
+ })
+ }
+
+ // Sort shelf keys by importpath, then by version.
+ sort.Slice(shelfProto.Entry, func(i, j int) bool {
+ a := shelfProto.Entry[i]
+ b := shelfProto.Entry[j]
+
+ if a.ImportPath < b.ImportPath {
+ return true
+ }
+ if a.ImportPath > b.ImportPath {
+ return false
+ }
+ return a.Version < b.Version
+ })
+
+ // Make an in-memory representation of the marshaled shelf.
+ buf := bytes.NewBuffer(nil)
+ err := proto.MarshalText(buf, &shelfProto)
+ if err != nil {
+ return fmt.Errorf("could not serialize shelf: %v", err)
+ }
+
+ // And write it out.
+ err = ioutil.WriteFile(s.path, buf.Bytes(), 0644)
+ if err != nil {
+ return fmt.Errorf("could not write shelf: %v", err)
+ }
+
+ return nil
+}
diff --git a/build/fietsje/transitive.go b/build/fietsje/transitive.go
new file mode 100644
index 0000000..f4a6e1d
--- /dev/null
+++ b/build/fietsje/transitive.go
@@ -0,0 +1,192 @@
+// Copyright 2020 The Monogon Project Authors.
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "fmt"
+ "io/ioutil"
+ "log"
+ "os"
+ "strings"
+
+ "golang.org/x/mod/modfile"
+)
+
+// getTransitiveDeps is a hairy ball of heuristic used to find all recursively transitive dependencies of a given
+// dependency.
+// It downloads a given dependency using `go get`, and performs analysis of standard (go.mod/go.sum) and project-
+// specific dependency management configuration/lock files in order to build a full view of all known, versioned
+// transitive dependencies.
+func (d *dependency) getTransitiveDeps() (map[string]*dependency, error) {
+ // First, lock the dependency. Downloading it later will also return a sum, and we want to ensure both are the
+ // same.
+ err := d.lock()
+ if err != nil {
+ return nil, fmt.Errorf("could not lock: %v", err)
+ }
+
+ _, path, sum, err := d.download()
+ if err != nil {
+ return nil, fmt.Errorf("could not download: %v", err)
+ }
+
+ if sum != d.locked.sum {
+ return nil, fmt.Errorf("inconsistent sum: %q downloaded, %q in shelf/lock", sum, d.locked.sum)
+ }
+
+ exists := func(p string) bool {
+ full := fmt.Sprintf("%s/%s", path, p)
+ if _, err := os.Stat(full); err == nil {
+ return true
+ }
+ if err != nil && !os.IsExist(err) {
+ panic(fmt.Sprintf("checking file %q: %v", full, err))
+ }
+ return false
+ }
+
+ read := func(p string) []byte {
+ full := fmt.Sprintf("%s/%s", path, p)
+ data, err := ioutil.ReadFile(full)
+ if err != nil {
+ panic(fmt.Sprintf("reading file %q: %v", full, err))
+ }
+ return data
+ }
+
+ requirements := make(map[string]*dependency)
+
+ // Read & parse go.mod if present.
+ var mf *modfile.File
+ if exists("go.mod") {
+ log.Printf("%q: parsing go.mod\n", d.importpath)
+ data := read("go.mod")
+ mf, err = modfile.Parse("go.mod", data, nil)
+ if err != nil {
+ return nil, fmt.Errorf("parsing go.mod in %s: %v", d.importpath, err)
+ }
+ }
+
+ // If a go.mod file was present, interpret it to populate dependencies.
+ if mf != nil {
+ for _, req := range mf.Require {
+ requirements[req.Mod.Path] = d.child(req.Mod.Path, req.Mod.Version)
+ }
+ for _, rep := range mf.Replace {
+ // skip filesystem rewrites
+ if rep.New.Version == "" {
+ continue
+ }
+
+ requirements[rep.New.Path] = d.child(rep.New.Path, rep.New.Version)
+ }
+ }
+
+ // Read parse, and interpret. go.sum if present.
+ // This should bring into view all recursively transitive dependencies.
+ if exists("go.sum") {
+ log.Printf("%q: parsing go.sum", d.importpath)
+ data := read("go.sum")
+ for _, line := range strings.Split(string(data), "\n") {
+ line = strings.TrimSpace(line)
+ if line == "" {
+ continue
+ }
+
+ parts := strings.Fields(line)
+ if len(parts) != 3 {
+ return nil, fmt.Errorf("parsing go.sum: unparseable line %q", line)
+ }
+
+ importpath, version := parts[0], parts[1]
+
+ // Skip if already created from go.mod.
+ // TODO(q3k): error if go.sum and go.mod disagree?
+ if _, ok := requirements[importpath]; ok {
+ continue
+ }
+
+ if strings.HasSuffix(version, "/go.mod") {
+ version = strings.TrimSuffix(version, "/go.mod")
+ }
+ requirements[importpath] = d.child(importpath, version)
+ }
+ }
+
+ // Special case: root Kubernetes repo - rewrite staging/ deps to k8s.io/ at correct versions, quit early.
+ // Kubernetes vendors all dependencies into vendor/, and also contains sub-projects (components) in staging/.
+ // This converts all staging dependencies into appropriately versioned k8s.io/<dep> paths.
+ if d.importpath == "k8s.io/kubernetes" {
+ log.Printf("%q: special case for Kubernetes main repository", d.importpath)
+ if mf == nil {
+ return nil, fmt.Errorf("k8s.io/kubernetes needs a go.mod")
+ }
+ // extract the version, turn into component version
+ version := d.version
+ if !strings.HasPrefix(version, "v") {
+ return nil, fmt.Errorf("invalid version format for k8s: %q", version)
+ }
+ version = version[1:]
+ componentVersion := fmt.Sprintf("kubernetes-%s", version)
+
+ // find all k8s.io 'components'
+ components := make(map[string]bool)
+ for _, rep := range mf.Replace {
+ if !strings.HasPrefix(rep.Old.Path, "k8s.io/") || !strings.HasPrefix(rep.New.Path, "./staging/src/") {
+ continue
+ }
+ components[rep.Old.Path] = true
+ }
+
+ // add them to planner at the 'kubernetes-$ver' tag
+ for component, _ := range components {
+ requirements[component] = d.child(component, componentVersion)
+ }
+ return requirements, nil
+ }
+
+ // Special case: github.com/containerd/containerd: read vendor.conf.
+ if d.importpath == "github.com/containerd/containerd" {
+ log.Printf("%q: special case for containerd", d.importpath)
+ if !exists("vendor.conf") {
+ panic("containerd needs vendor.conf")
+ }
+ data := read("vendor.conf")
+ for _, line := range strings.Split(string(data), "\n") {
+ // strip comments
+ parts := strings.SplitN(line, "#", 2)
+ line = parts[0]
+
+ // skip empty contents
+ line = strings.TrimSpace(line)
+ if line == "" {
+ continue
+ }
+
+ // read dep/version pairs
+ parts = strings.Fields(line)
+ if len(parts) < 2 {
+ return nil, fmt.Errorf("unparseable line in containerd vendor.conf: %q", line)
+ }
+ importpath, version := parts[0], parts[1]
+ requirements[importpath] = d.child(importpath, version)
+ }
+ return requirements, nil
+ }
+
+ return requirements, nil
+}
diff --git a/build/sqlboiler/sqlboiler.bzl b/build/sqlboiler/sqlboiler.bzl
index c9ed7d1..bfb4bf9 100644
--- a/build/sqlboiler/sqlboiler.bzl
+++ b/build/sqlboiler/sqlboiler.bzl
@@ -76,7 +76,7 @@
command += "cp \"{}\" migrations/\n".format(f.path)
# Copy in adapter
- command += "cp \"{}\" .\n".format(ctx.file.adapter.path)
+ command += "cp \"{}\" sqlboiler-crdb\n".format(ctx.file.adapter.path)
# Apply sql-migrate
command += "{} up --config \"{}\" \n".format(ctx.file.migrate.path, ctx.file.migrate_config.basename)
@@ -109,8 +109,8 @@
"tables": attr.string_list(default = []),
"migrate_config": attr.label(allow_single_file = True, default = Label("//build/sqlboiler:dbconfig.yml")),
"boiler_config": attr.label(allow_single_file = True, default = Label("//build/sqlboiler:sqlboiler.toml")),
- "boiler": attr.label(allow_single_file = True, default = Label("@com_github_volatiletech_sqlboiler//:sqlboiler")),
- "adapter": attr.label(allow_single_file = True, default = Label("@com_github_glerchundi_sqlboiler_crdb//:sqlboiler-crdb")),
+ "boiler": attr.label(allow_single_file = True, default = Label("@com_github_volatiletech_sqlboiler_v4//:v4")),
+ "adapter": attr.label(allow_single_file = True, default = Label("@com_github_glerchundi_sqlboiler_crdb_v4//:v4")),
"migrate": attr.label(allow_single_file = True, default = Label("@com_github_rubenv_sql_migrate//sql-migrate:sql-migrate")),
},
)
@@ -149,15 +149,14 @@
Label("@com_github_lib_pq//:go_default_library"),
Label("@com_github_pkg_errors//:go_default_library"),
Label("@com_github_spf13_viper//:go_default_library"),
- Label("@com_github_volatiletech_sqlboiler//boil:go_default_library"),
- Label("@com_github_volatiletech_sqlboiler//drivers:go_default_library"),
- Label("@com_github_volatiletech_sqlboiler//queries:go_default_library"),
- Label("@com_github_volatiletech_sqlboiler//queries/qm:go_default_library"),
- Label("@com_github_volatiletech_sqlboiler//randomize:go_default_library"),
- Label("@com_github_volatiletech_sqlboiler//strmangle:go_default_library"),
- Label("@com_github_volatiletech_sqlboiler//types:go_default_library"),
- Label("@com_github_volatiletech_sqlboiler//queries/qmhelper:go_default_library"),
- Label("@com_github_volatiletech_null//:go_default_library"),
+ Label("@com_github_volatiletech_strmangle//:go_default_library"),
+ Label("@com_github_volatiletech_sqlboiler_v4//boil:go_default_library"),
+ Label("@com_github_volatiletech_sqlboiler_v4//drivers:go_default_library"),
+ Label("@com_github_volatiletech_sqlboiler_v4//queries:go_default_library"),
+ Label("@com_github_volatiletech_sqlboiler_v4//queries/qm:go_default_library"),
+ Label("@com_github_volatiletech_sqlboiler_v4//types:go_default_library"),
+ Label("@com_github_volatiletech_sqlboiler_v4//queries/qmhelper:go_default_library"),
+ Label("@com_github_volatiletech_null_v8//:go_default_library"),
],
),
},