build/fietsje: remove
We've moved back to using go.mod as a source of truth, as our main large
depdency (Kubernetes) is now mostly gomod compatible.
Change-Id: Ie6215b7330a7dcec7681fa3081437efb2be5bf77
Reviewed-on: https://review.monogon.dev/c/monogon/+/651
Reviewed-by: Lorenz Brun <lorenz@monogon.tech>
diff --git a/build/ci/jenkins-presubmit.groovy b/build/ci/jenkins-presubmit.groovy
index b26cdd7..37278ba 100644
--- a/build/ci/jenkins-presubmit.groovy
+++ b/build/ci/jenkins-presubmit.groovy
@@ -45,7 +45,7 @@
gerritCheck checks: ['jenkins:gazelle': 'RUNNING'], message: "Running on ${env.NODE_NAME}"
echo "Gerrit change: ${GERRIT_CHANGE_URL}"
sh "git clean -fdx -e '/bazel-*'"
- sh "JENKINS_NODE_COOKIE=dontKillMe bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=third_party/go/repositories.bzl%go_repositories -prune"
+ sh "JENKINS_NODE_COOKIE=dontKillMe bazel run //:gazelle-update-repos"
sh "JENKINS_NODE_COOKIE=dontKillMe bazel run //:gazelle -- update"
script {
@@ -56,7 +56,7 @@
Unclean working directory after running gazelle.
Please run:
- \$ bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=third_party/go/repositories.bzl%go_repositories -prune
+ \$ bazel run //:gazelle-update-repos
\$ bazel run //:gazelle -- update
In your git checkout and amend the resulting diff to this changelist.
diff --git a/build/fietsje/BUILD.bazel b/build/fietsje/BUILD.bazel
deleted file mode 100644
index bb3c38e..0000000
--- a/build/fietsje/BUILD.bazel
+++ /dev/null
@@ -1,32 +0,0 @@
-load("@io_bazel_rules_go//go:def.bzl", "go_library")
-
-go_library(
- name = "fietsje",
- srcs = [
- "dependency.go",
- "deps_containerd.go",
- "deps_delve.go",
- "deps_gvisor.go",
- "deps_kubernetes.go",
- "deps_monogon.go",
- "planner.go",
- "render.go",
- "shelf.go",
- "transitive.go",
- ],
- importpath = "source.monogon.dev/build/fietsje",
- visibility = ["//build:__subpackages__"],
- deps = [
- "//build/fietsje/proto",
- "@bazel_gazelle//label:go_default_library",
- "@com_github_golang_protobuf//proto:go_default_library",
- "@org_golang_x_mod//modfile",
- ],
-)
-
-exports_files(
- [
- "fietsje.bash.in",
- ],
- visibility = ["//visibility:public"],
-)
diff --git a/build/fietsje/README.md b/build/fietsje/README.md
deleted file mode 100644
index 723813f..0000000
--- a/build/fietsje/README.md
+++ /dev/null
@@ -1,169 +0,0 @@
-Fietsje
-=======
-
-The little Gazelle that could.
-
-Introduction
-------------
-
-Fietsje is a dependency management system for Go dependencies in monogon. It
-does not replace either gomods or Gazelle, but instead builds upon both on them
-in a way that makes sense for our particular usecase: pulling in a large set of
-dependency trees from third\_party projects, and sticking to those as much as
-possible.
-
-When run, Fietsje consults rules written themselves in Go (in `deps_.*go`
-files), and uses this high-level intent to write a `repositories.bzl` file
-that is then consumed by Gazelle. It caches 'locked' versions (ie. Go import
-path and version to a particular checksum) in the Shelf, a text proto file
-that lives alongside `repositories.bzl`. The Shelf should not be modified
-manually.
-
-The effective source of truth used for builds is still the `repositories.bzl`
-file in the actual build path. Definitions in Go are in turn the high-level
-intent that is used to build `repositories.bzl`.
-
-Running
--------
-
-You should run Fietsje any time you want to update dependencies. The following
-should be a no-op if you haven't changed anything in `deps_*.go`:
-
- scripts/bin/bazel run //:fietsje
-
-Otherwise, if any definition in build/fietsje/deps_*.go has been changed,
-third_party/go/repositories.bzl will now reflect that.
-
-Fietsje Definition DSL (collect/use/...)
-----------------------------------------
-
-Definitions are kept in pure Go source, with a light DSL focused around a
-'planner' builder.
-
-The builder allows for two kinds of actions:
- - import a high level dependency (eg. Kubernetes, google/tpm) at a particular
- version. This is done using the `collect()` call. The dependency will now
- be part of the build, but its transitive dependencies will not. A special
- flavor of collect() is collectOverride(), that explicitely allows for
- overriding a dependency that has already been pulled in by another high
- level dependency.
- - enable a transitive dependency defined by a high-level definition using the `use()`
- call. This can only be done in a `collection` builder context, ie. after a
- `collect()`/`collectOverride()`call.
-
-In addition, the builder allows to augment a `collection` context with build flags
-(like enabled patches, build tags, etc) that will be applied to the next `.use()`
-call only. This is done by calling `.with()`.
-
-In general, `.collect()`/`.collectOverride()` calls should be limited only to
-dependencies 'we' (as developers) want. These 'high-level' dependencies are
-large projects like Kubernetes, or direct imports from monogon itself. Every
-transitive dependency of those should just be enabled by calling `.use()`,
-instead of another `.collectOverride()` call that might pin it to a wrong
-version.
-
-After updating definitions, run Fietsje as above.
-
-How to: add a new high-level dependency
----------------------------------------
-
-To add a new high-level dependency, first consider making a new `deps_*.go`
-file for it. If you're pulling in a separate ecosystem of code (ie. a large
-third-party project like kubernetes), it should live in its own file for
-clarity. If you're just pulling in a simple dependency (eg. a library low on
-transitive dependencies) you can drop it into `main.go`.
-
-The first step is to pick a version of the dependency you want to use. If
-possible, pick a tag/release. Otherwise, pick the current master commit hash.
-You can find version information by visiting the project's git repository web
-viewer, or first cloning the repository locally.
-
-Once you've picked a version, add a line like this:
-
- p.collect("github.com/example/foo", "1.2.3")
-
-If you now re-run Fietsje and rebuild your code, it should be able to link
-against the dependency directly. If this works, you're done. If not, you will
-start getting errors about the newly included library trying to link against
-missing dependencies (ie. external Bazel workspaces). This means you need to
-enable these transitive dependencies for the high-level dependency you've just
-included.
-
-If your high-level dependency contains a go.mod/go.sum file, you can call
-`.use` on the return of the `collect()` call to enable them. Only enable the
-ones that are necessary to build your code. In the future, audit flows might be
-implemented to find and eradicate unused transitive dependencies, while enabling
-ones that are needed - but for now this has to be done manually - usually by a
-cycle of:
-
- - try to build your code
- - find missing transitive library, enable via .use()
- - repeat until code builds
-
-With our previous example, enabling transitive dependencies would look something
-like this:
-
- p.collect(
- "github.com/example/foo", "1.2.3",
- ).use(
- "github.com/example/libbar",
- "github.com/example/libbaz",
- "github.com/golang/glog",
- )
-
-What this means is that github.com/{example/libbar,example/libbaz,golang/glog}
-will now be available to the build at whatever version example/foo defines them
-in its go.mod/go.sum.
-
-If your high-level dependency is not go.mod/go.sum compatible, you have
-different ways to proceed:
-
- - if the project uses some alternative resolution/vendoring code, write
- support for it in transitive.go/`getTransitiveDeps`
- - otherwise, if you're not in a rush, try to convince and/or send a PR to
- upstream to enable Go module support
- - if the dependency has little transitive dependencies, use `.inject()` to
- add transitive dependencies manually after your `.collect()` call
- - otherwise, extend fietsje to allow for out-of-tree go.mod/go.sum files kept
- within monogon, or come up with some other solution.
-
-Your new dependency might conflict with existing dependencies, which usually
-manifests in build failures due to incompatible types. If this happens, you
-will have to start digging to find a way to bring in compatible versions of
-the two dependencies that are interacting with eachother. Do also mention any
-such constraints in code comments near your `.collect()` call.
-
-How to: update a high-level dependency
---------------------------------------
-
-If you want to update a .collect()/.collectOverride() call, find out the
-version you want to bump to and update it in the call. Re-running fietsje
-will automatically update all enable transitive dependencies. Build and test
-your code. Again, any possible conflicts will have to be resolved manually.
-
-In the future, an audit flow might be provided for checking what the newest
-available version of a high-level dependency is, to allow for easier,
-semi-automated version bumps.
-
-Version resolution conflicts
-----------------------------
-
-Any time a `.collect()`/`.collectOverride()` call is made, Fietsje will note
-what transitive dependencies did the specified high-level dependency request.
-Then, subsequent `.use()` calls will enable these dependencies in the build. On
-subsequent `.collect()`/`.collectOverride()` calls, any transitive dependency
-that already has been pulled in will be ignored, and the existing version will
-be kept.
-
-This means that Fietsje does not detect or handle version conflicts at a granular
-level comparable to gomod. However, it does perform 'well enough', and in general
-the Go ecosystem is stable enough that incompatibilites arise rarely - especially as
-everything moves forward to versioned to go modules, which allow for multiple
-incompatible versions to coexist as fully separate import paths.
-
-It is as such the programmer's job to understand the relationship between imported
-high-level dependencies. In the future, helper heuristics can be included that will
-help understand and reason about dependency relationships. For now, Fietsje will just
-help a user when they call `.use()` on the wrong dependency, ie. when the requested
-transitive dependency has not been pulled in by a given high-level dependency.
-
diff --git a/build/fietsje/cmd/BUILD.bazel b/build/fietsje/cmd/BUILD.bazel
deleted file mode 100644
index fec62fe..0000000
--- a/build/fietsje/cmd/BUILD.bazel
+++ /dev/null
@@ -1,19 +0,0 @@
-load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
-
-go_library(
- name = "cmd_lib",
- srcs = ["main.go"],
- importpath = "source.monogon.dev/build/fietsje/cmd",
- visibility = ["//visibility:private"],
- deps = [
- "//build/fietsje",
- "//build/toolbase",
- "//build/toolbase/gotoolchain",
- ],
-)
-
-go_binary(
- name = "cmd",
- embed = [":cmd_lib"],
- visibility = ["//visibility:public"],
-)
diff --git a/build/fietsje/cmd/main.go b/build/fietsje/cmd/main.go
deleted file mode 100644
index b9e1553..0000000
--- a/build/fietsje/cmd/main.go
+++ /dev/null
@@ -1,49 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-// fietsje is the standalone command line tool which calls into the Fietsje
-// library (//build/fietsje) to perform actual work. The split between cli and
-// main library is used so that Fietsje can be called from other Go tooling
-// without having to shell out to a binary.
-package main
-
-import (
- "log"
- "os"
- "path"
-
- "source.monogon.dev/build/fietsje"
- "source.monogon.dev/build/toolbase"
- "source.monogon.dev/build/toolbase/gotoolchain"
-)
-
-func main() {
- // Get absolute path of Monogon workspace directory currently operating on
- // (either via bazel run or by running it directly in the root of a checkout),
- // use it to build paths to shelf.pb.txt and repositories.bzl.
- wd, err := toolbase.WorkspaceDirectory()
- if err != nil {
- log.Fatalf("%v", err)
- }
- shelfPath := path.Join(wd, "third_party/go/shelf.pb.text")
- repositoriesBzlPath := path.Join(wd, "third_party/go/repositories.bzl")
- // Set GOROOT as required by fietsje/go-the-tool.
- os.Setenv("GOROOT", gotoolchain.Root)
-
- if err := fietsje.Monogon(shelfPath, repositoriesBzlPath); err != nil {
- log.Fatal(err)
- }
-}
diff --git a/build/fietsje/dependency.go b/build/fietsje/dependency.go
deleted file mode 100644
index 1eeb52c..0000000
--- a/build/fietsje/dependency.go
+++ /dev/null
@@ -1,175 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-import (
- "encoding/json"
- "fmt"
- "log"
- "os"
- "os/exec"
- "path/filepath"
-
- "github.com/bazelbuild/bazel-gazelle/label"
-)
-
-// dependency is an external Go package/module, requested by the user of Fietsje
-// directly or indirectly.
-type dependency struct {
- // importpath is the Go import path that was used to import this dependency.
- importpath string
- // version at which this dependency has been requested. This can be in any form
- // that `go get` or the go module system understands.
- version string
-
- // locked is the 'resolved' version of a dependency, containing information about
- // the dependency's hash, etc.
- locked *locked
-
- // parent is the dependency that pulled in this one, or nil if pulled in by the
- // user.
- parent *dependency
-
- shelf *shelf
-
- // Build specific settings passed to gazelle.
- disableProtoBuild bool
- forceBazelGeneration bool
- buildTags []string
- patches []string
- prePatches []string
- buildExtraArgs []string
- useImportAliasNaming bool
- // replace is an importpath that this dependency will replace. If this is set, this
- // dependency will be visible in the build as 'importpath', but downloaded at
- // 'replace'/'version'. This might be slighly confusing, but follows the semantics
- // of what Gazelle exposes via 'replace' in 'go_repository'.
- replace string
-}
-
-func (d *dependency) remoteImportpath() string {
- if d.replace != "" {
- return d.replace
- }
- return d.importpath
-}
-
-// locked is information about a dependency resolved from the go module system. It
-// is expensive to get, and as such it is cached both in memory (as .locked in a
-// dependency) and in the shelf.
-type locked struct {
- // bazelName is the external workspace name that Bazel should use for this
- // dependency, eg. com_github_google_glog.
- bazelName string
- // sum is the gomod compatible checksum of the depdendency,
- // egh1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs=.
- sum string
- // semver is the gomod-compatible version of this dependency. If the dependency was
- // requested by git hash that does not resolve to a particular release, this will
- // be in the form of v0.0.0-20200520133742-deadbeefcafe.
- semver string
-}
-
-// child creates a new child dependence for this dependency, ie. one where the
-// 'parent' pointer points to the dependency on which this method is called.
-func (d *dependency) child(importpath, version string) *dependency {
- return &dependency{
- importpath: importpath,
- version: version,
- shelf: d.shelf,
- parent: d,
- }
-}
-
-func (d *dependency) String() string {
- if d.replace != "" {
- return fmt.Sprintf("%s@%s (replacing %s)", d.replace, d.version, d.importpath)
- }
- return fmt.Sprintf("%s@%s", d.importpath, d.version)
-}
-
-// lock ensures that this dependency is locked, which means that it has been
-// resolved to a particular, stable version and VCS details. We lock a dependency
-// by either asking the go module subsystem (via a go module proxy or a download),
-// or by consulting the shelf as a cache.
-func (d *dependency) lock() error {
- // If already locked in-memory, use that.
- if d.locked != nil {
- return nil
- }
-
- // If already locked in the shelf, use that.
- if shelved := d.shelf.get(d.remoteImportpath(), d.version); shelved != nil {
- d.locked = shelved
- return nil
- }
-
- // Otherwise, download module.
- semver, _, sum, err := d.download()
- if err != nil {
- return fmt.Errorf("could not download: %v", err)
- }
-
- // And resolve its bazelName.
- name := label.ImportPathToBazelRepoName(d.importpath)
-
- d.locked = &locked{
- bazelName: name,
- sum: sum,
- semver: semver,
- }
- log.Printf("%s: locked to %s", d, d.locked)
-
- // Save locked version to shelf.
- d.shelf.put(d.remoteImportpath(), d.version, d.locked)
- return d.shelf.save()
-}
-
-func (l *locked) String() string {
- return fmt.Sprintf("%s@%s", l.bazelName, l.sum)
-}
-
-// download ensures that this dependency is download locally, and returns the
-// download location and the dependency's gomod-compatible sum.
-func (d *dependency) download() (version, dir, sum string, err error) {
- goroot := os.Getenv("GOROOT")
- if goroot == "" {
- err = fmt.Errorf("GOROOT must be set")
- return
- }
- goTool := filepath.Join(goroot, "bin", "go")
-
- query := fmt.Sprintf("%s@%s", d.remoteImportpath(), d.version)
- cmd := exec.Command(goTool, "mod", "download", "-json", "--", query)
- out, err := cmd.Output()
- if err != nil {
- log.Printf("go mod returned: %q", out)
- err = fmt.Errorf("go mod failed: %v", err)
- return
- }
-
- var res struct{ Version, Sum, Dir string }
- err = json.Unmarshal(out, &res)
- if err != nil {
- return
- }
-
- version = res.Version
- dir = res.Dir
- sum = res.Sum
- return
-}
diff --git a/build/fietsje/deps_containerd.go b/build/fietsje/deps_containerd.go
deleted file mode 100644
index 9461cdb..0000000
--- a/build/fietsje/deps_containerd.go
+++ /dev/null
@@ -1,144 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-// deps_containerd.go contains all dependencies required by containerd/runc.
-
-func depsContainerd(p *planner) {
- p.collectOverride(
- "github.com/containerd/containerd", "v1.4.3",
- buildTags("no_zfs", "no_aufs", "no_devicemapper", "no_btrfs"),
- disabledProtoBuild,
- ).use(
- "github.com/BurntSushi/toml",
- "github.com/Microsoft/go-winio",
- "github.com/beorn7/perks",
- "github.com/cespare/xxhash/v2",
- "github.com/cilium/ebpf",
- "github.com/containerd/btrfs",
- "github.com/containerd/console",
- "github.com/containerd/continuity",
- "github.com/containerd/fifo",
- "github.com/containerd/go-cni",
- "github.com/containerd/go-runc",
- "github.com/containerd/imgcrypt",
- "github.com/containers/ocicrypt",
- "github.com/containerd/typeurl",
- "github.com/coreos/go-systemd/v22",
- "github.com/cpuguy83/go-md2man/v2",
- "github.com/davecgh/go-spew",
- "github.com/docker/docker",
- "github.com/docker/go-events",
- "github.com/docker/go-metrics",
- "github.com/docker/go-units",
- "github.com/docker/spdystream",
- "github.com/emicklei/go-restful",
- "github.com/fullsailor/pkcs7",
- "github.com/godbus/dbus/v5",
- "github.com/gogo/protobuf",
- "github.com/go-logr/logr",
- "github.com/google/gofuzz",
- "github.com/google/uuid",
- "github.com/hashicorp/errwrap",
- "github.com/hashicorp/go-multierror",
- "github.com/hashicorp/golang-lru",
- "github.com/imdario/mergo",
- "github.com/json-iterator/go",
- "github.com/konsorten/go-windows-terminal-sequences",
- "github.com/matttproud/golang_protobuf_extensions",
- "github.com/modern-go/concurrent",
- "github.com/modern-go/reflect2",
- "github.com/opencontainers/go-digest",
- "github.com/opencontainers/image-spec",
- "github.com/opencontainers/runc",
- "github.com/opencontainers/runtime-spec",
- "github.com/pkg/errors",
- "github.com/prometheus/client_golang",
- "github.com/prometheus/client_model",
- "github.com/prometheus/common",
- "github.com/prometheus/procfs",
- "github.com/russross/blackfriday/v2",
- "github.com/shurcooL/sanitized_anchor_name",
- "github.com/sirupsen/logrus",
- "github.com/syndtr/gocapability",
- "github.com/tchap/go-patricia",
- "github.com/urfave/cli",
- "go.etcd.io/bbolt",
- "go.opencensus.io",
- "golang.org/x/crypto",
- "golang.org/x/oauth2",
- "google.golang.org/genproto",
- "gopkg.in/inf.v0",
- "gopkg.in/yaml.v2",
- "k8s.io/klog/v2",
- "sigs.k8s.io/yaml",
- ).with(useImportAliasNaming).use(
- "golang.org/x/sync",
- ).with(disabledProtoBuild, patches("containerd-netns-statedir.patch")).use(
- "github.com/containerd/cri",
- ).with(disabledProtoBuild).use(
- "github.com/Microsoft/hcsshim",
- "github.com/containerd/cgroups",
-
- "github.com/gogo/googleapis",
- ).with(buildTags("selinux")).use(
- "github.com/opencontainers/selinux",
- "github.com/willf/bitset",
- ).with(patches(
- "ttrpc-hacks.patch",
- )).use(
- "github.com/containerd/ttrpc",
- ).with(patches(
- "cni-fix-cachepath.patch",
- )).use("github.com/containernetworking/cni").replace(
- // ttrpc is broken by go protobuf v2, this is a tentative PR that's
- // not yet merged by upstream.
- // See: https://github.com/containerd/ttrpc/pull/67
- //
- // It also contains our own fix that builds up on the above and allows services to
- // return the original status error library values. This is required for ttrpc to
- // actually work from runsc and for results to be correctly interpreted by
- // containerd. See:
- // https://github.com/monogon-dev/ttrpc/commit/222b428f008e3ecb11cfff12e3fd92e3143a2f01
- //
- // Note: this is not a good fix, and has known issues, like not being
- // able to return Details in gRPC status errors. However, with the
- // limited usage within gvisor/containerd it works. In the future
- // upstream will have to resolve this properly, eg. port ttrpc away
- // from gogo, or fix gogo to work with the new protobuf APU.
- "github.com/containerd/ttrpc",
- "github.com/monogon-dev/ttrpc", "222b428f008e3ecb11cfff12e3fd92e3143a2f01",
- )
-
- // This is depended on by github.com/containerd/containerd, but not mentioned in
- // their vendor.conf. They seem to be moving off of vendoring to gomod, so this
- // should be reverted on the next containerd bump (when fietsje will panic about
- // vendor.conf missing).
- p.collectOverride(
- "github.com/checkpoint-restore/go-criu/v4", "v4.1.0",
- )
-
- // containernetworking/plugins
- p.collectOverride(
- "github.com/containernetworking/plugins", "v0.8.2",
- ).use(
- "github.com/alexflint/go-filemutex",
- "github.com/coreos/go-iptables",
- "github.com/j-keck/arping",
- "github.com/safchain/ethtool",
- )
-}
diff --git a/build/fietsje/deps_delve.go b/build/fietsje/deps_delve.go
deleted file mode 100644
index 69258a8..0000000
--- a/build/fietsje/deps_delve.go
+++ /dev/null
@@ -1,29 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-// deps_delve.go contains all dependencies required by the Delve debugger.
-
-func depsDelve(p *planner) {
- p.collect("github.com/go-delve/delve", "v1.4.1").use(
- "github.com/google/go-dap",
- "github.com/peterh/liner",
- "github.com/cosiner/argv",
- "golang.org/x/arch",
- "go.starlark.net",
- )
-}
diff --git a/build/fietsje/deps_gvisor.go b/build/fietsje/deps_gvisor.go
deleted file mode 100644
index 8810a83..0000000
--- a/build/fietsje/deps_gvisor.go
+++ /dev/null
@@ -1,41 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-// deps_gvisor.go contains all dependencies required by gVisor/runsc.
-
-func depsGVisor(p *planner) {
- p.collect(
- "github.com/google/gvisor", "release-20211101.0",
- patches(
- "gvisor.patch",
- "gvisor-build-against-newer-runtime-specs.patch",
- "gvisor-cgroup-fix.patch",
- ),
- ).use(
- "github.com/cenkalti/backoff",
- "github.com/gofrs/flock",
- "github.com/google/subcommands",
- "github.com/kr/pretty",
- "github.com/kr/pty",
- "github.com/mohae/deepcopy",
- "golang.org/x/time",
- )
- // gRPC is used by gvisor's bazel machinery, but not present in go.sum. Include it
- // manually.
- p.collect("github.com/grpc/grpc", "v1.29.1")
-}
diff --git a/build/fietsje/deps_kubernetes.go b/build/fietsje/deps_kubernetes.go
deleted file mode 100644
index e3b00cf..0000000
--- a/build/fietsje/deps_kubernetes.go
+++ /dev/null
@@ -1,186 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-// deps_kubernetes.go contains all dependencies required by Kubernetes.
-
-func depsKubernetes(p *planner) {
- // containerd and its deps
- p.collect(
- "k8s.io/kubernetes", "v1.19.7",
- buildTags("providerless"),
- disabledProtoBuild,
- patches(
- "k8s-kubernetes.patch",
- "k8s-kubernetes-build.patch",
- "k8s-native-metrics.patch",
- "k8s-use-native.patch",
- "k8s-revert-seccomp-runtime-default.patch",
- "k8s-removed-block-device-pseudo-locks.patch",
- ),
- prePatches(
- "k8s-e2e-tests-providerless.patch",
- "k8s-fix-paths.patch",
- "k8s-fix-logs-path.patch",
- "k8s-drop-legacy-log-path.patch",
- ),
- ).inject(
- // repo infra, not requested by k8s, but used with bazel
- "k8s.io/repo-infra", "a3483874bd37251c629c92df6d82a226b0e6ad92",
- prePatches("k8s-infra-bzl4-compat.patch", "k8s-infra-fix-go116.patch"),
- ).with(prePatches("k8s-client-go.patch")).use(
- "k8s.io/client-go",
- ).with(patches("k8s-native-mounter.patch")).use(
- "k8s.io/utils",
- ).use(
- "k8s.io/cli-runtime",
- "k8s.io/client-go",
- "k8s.io/cloud-provider",
- "k8s.io/cluster-bootstrap",
- "k8s.io/code-generator",
- "k8s.io/component-base",
- "k8s.io/csi-translation-lib",
- "k8s.io/kube-controller-manager",
- "k8s.io/kube-proxy",
- "k8s.io/kube-scheduler",
- "k8s.io/kubectl",
- "k8s.io/legacy-cloud-providers",
- "k8s.io/sample-apiserver",
- ).with(disabledProtoBuild).use(
- "k8s.io/api",
- "k8s.io/apiextensions-apiserver",
- "k8s.io/apimachinery",
- "k8s.io/apiserver",
- "k8s.io/cri-api",
- "k8s.io/kube-aggregator",
- "k8s.io/kubelet",
- "k8s.io/metrics",
- ).use(
- "cloud.google.com/go",
- "github.com/Azure/go-ansiterm",
- "github.com/MakeNowJust/heredoc",
- "github.com/NYTimes/gziphandler",
- "github.com/PuerkitoBio/purell",
- "github.com/PuerkitoBio/urlesc",
- "github.com/armon/circbuf",
- "github.com/asaskevich/govalidator",
- "github.com/bgentry/speakeasy",
- "github.com/blang/semver",
- "github.com/chai2010/gettext-go",
- "github.com/container-storage-interface/spec",
- "github.com/coreos/go-oidc",
- "github.com/coreos/go-semver",
- "github.com/coreos/go-systemd",
- "github.com/coreos/pkg",
- "github.com/cyphar/filepath-securejoin",
- "github.com/daviddengcn/go-colortext",
- "github.com/dgrijalva/jwt-go",
- "github.com/docker/go-connections",
- "github.com/docker/distribution",
- "github.com/dustin/go-humanize",
- "github.com/elazarl/goproxy",
- "github.com/euank/go-kmsg-parser",
- "github.com/evanphx/json-patch",
- "github.com/exponent-io/jsonpath",
- "github.com/fatih/camelcase",
- "github.com/fatih/color",
- "github.com/ghodss/yaml",
- "github.com/go-openapi/analysis",
- "github.com/go-openapi/errors",
- "github.com/go-openapi/jsonpointer",
- "github.com/go-openapi/jsonreference",
- "github.com/go-openapi/loads",
- "github.com/go-openapi/runtime",
- "github.com/go-openapi/spec",
- "github.com/go-openapi/strfmt",
- "github.com/go-openapi/swag",
- "github.com/go-openapi/validate",
- "github.com/go-stack/stack",
- "github.com/golang/groupcache",
- "github.com/google/btree",
- "github.com/google/go-cmp",
- "github.com/gorilla/websocket",
- "github.com/gregjones/httpcache",
- "github.com/grpc-ecosystem/go-grpc-middleware",
- "github.com/grpc-ecosystem/go-grpc-prometheus",
- "github.com/grpc-ecosystem/grpc-gateway",
- "github.com/hashicorp/hcl",
- "github.com/hpcloud/tail",
- "github.com/inconshreveable/mousetrap",
- "github.com/jonboulle/clockwork",
- "github.com/karrick/godirwalk",
- "github.com/liggitt/tabwriter",
- "github.com/lithammer/dedent",
- "github.com/mailru/easyjson",
- "github.com/magiconair/properties",
- "github.com/mattn/go-colorable",
- "github.com/mattn/go-isatty",
- "github.com/mattn/go-runewidth",
- "github.com/mindprince/gonvml",
- "github.com/mistifyio/go-zfs",
- "github.com/mitchellh/go-wordwrap",
- "github.com/mitchellh/mapstructure",
- "github.com/moby/term",
- "github.com/moby/sys/mountinfo",
- "github.com/morikuni/aec",
- "github.com/mrunalp/fileutils",
- "github.com/munnerz/goautoneg",
- "github.com/mxk/go-flowrate",
- "github.com/olekukonko/tablewriter",
- "github.com/onsi/ginkgo",
- "github.com/onsi/gomega",
- "github.com/peterbourgon/diskv",
- "github.com/pquerna/cachecontrol",
- "github.com/robfig/cron",
- "github.com/russross/blackfriday",
- "github.com/soheilhy/cmux",
- "github.com/spf13/afero",
- "github.com/spf13/cast",
- "github.com/spf13/jwalterweatherman",
- "github.com/spf13/cobra",
- "github.com/spf13/pflag",
- "github.com/spf13/viper",
- "github.com/stretchr/testify",
- "github.com/tmc/grpc-websocket-proxy",
- "github.com/vishvananda/netlink",
- "github.com/vishvananda/netns",
- "github.com/xiang90/probing",
- "go.mongodb.org/mongo-driver",
- "go.uber.org/atomic",
- "go.uber.org/multierr",
- "go.uber.org/zap",
- "golang.org/x/net",
- "golang.org/x/xerrors",
- "gonum.org/v1/gonum",
- "gopkg.in/fsnotify.v1",
- "gopkg.in/natefinch/lumberjack.v2",
- "gopkg.in/square/go-jose.v2",
- "gopkg.in/tomb.v1",
- "k8s.io/gengo",
- "k8s.io/heapster",
- "k8s.io/kube-openapi",
- "sigs.k8s.io/apiserver-network-proxy/konnectivity-client",
- "sigs.k8s.io/kustomize",
- "sigs.k8s.io/structured-merge-diff/v4",
- "vbom.ml/util",
- ).use(
- "github.com/google/cadvisor",
- ).with(disabledProtoBuild).use(
- "go.etcd.io/etcd",
- "github.com/googleapis/gnostic",
- )
-}
diff --git a/build/fietsje/deps_monogon.go b/build/fietsje/deps_monogon.go
deleted file mode 100644
index 7ab3705..0000000
--- a/build/fietsje/deps_monogon.go
+++ /dev/null
@@ -1,152 +0,0 @@
-package fietsje
-
-// deps_monogon.go contains the main entrypoint for Monogon-specific
-// dependencies to be handled by fietsje, and calls out to all other functions
-// in deps_*.go.
-
-import (
- "bytes"
- "fmt"
- "os"
-)
-
-// Monogon runs fietsje for all Monogon transitive dependencies.
-func Monogon(shelfPath, repositoriesBzlPath string) error {
- shelf, err := shelfLoad(shelfPath)
- if err != nil {
- return fmt.Errorf("could not load shelf: %w", err)
- }
-
- p := &planner{
- available: make(map[string]*dependency),
- enabled: make(map[string]bool),
- seen: make(map[string]string),
-
- shelf: shelf,
- }
-
- depsKubernetes(p)
- depsContainerd(p)
- depsGVisor(p)
-
- // our own deps, common
- p.collectOverride("go.uber.org/zap", "v1.15.0")
- p.collectOverride("golang.org/x/mod", "v0.3.0", useImportAliasNaming)
- p.collectOverride("github.com/spf13/viper", "v1.9.0").use(
- "gopkg.in/ini.v1",
- "github.com/subosito/gotenv",
- )
- p.collectOverride("github.com/spf13/cobra", "v1.2.1")
- p.collect("github.com/cenkalti/backoff/v4", "v4.0.2")
-
- p.collect("github.com/google/go-tpm", "ae6dd98980d4")
- p.collect("github.com/google/go-tpm-tools", "f8c04ff88181")
- p.collect("github.com/google/certificate-transparency-go", "v1.1.0")
- p.collect("github.com/insomniacslk/dhcp", "67c425063dcad32c5d14ce9a520c8865240dc945").use(
- "github.com/mdlayher/ethernet",
- "github.com/mdlayher/raw",
- "github.com/u-root/u-root",
- )
- p.collect("github.com/rekby/gpt", "a930afbc6edcc89c83d39b79e52025698156178d")
- p.collect("github.com/yalue/native_endian", "51013b03be4fd97b0aabf29a6923e60359294186")
-
- // Used by //build/bazel_cc_fix, override to make sure we use the latest version
- p.collectOverride("github.com/mattn/go-shellwords", "v1.0.11")
-
- // Used by //metropolis/build/mkimage
- p.collect("github.com/diskfs/go-diskfs", "v1.2.0").use(
- "gopkg.in/djherbis/times.v1",
- "github.com/pkg/xattr",
- "github.com/pierrec/lz4",
- "github.com/ulikunitz/xz",
- )
-
- p.collect("github.com/pierrec/lz4/v4", "v4.1.12")
-
- // Used by //metropolis/build/genosrelease
- p.collect("github.com/joho/godotenv", "v1.3.0")
-
- // used by //build/bindata
- p.collect("github.com/kevinburke/go-bindata", "v3.16.0")
-
- // for interactive debugging during development (//:dlv alias)
- depsDelve(p)
-
- // Used by //metropolis/test/nanoswitch
- p.collect("github.com/google/nftables", "7127d9d22474b437f0e8136ddb21855df29790bf").use(
- "github.com/koneu/natend",
- )
-
- // Used by //metropolis/node/core/network/dhcp4c
- p.collect("github.com/google/gopacket", "v1.1.17")
-
- // used by //core//kubernetes/clusternet
- p.collect("golang.zx2c4.com/wireguard/wgctrl", "ec7f26be9d9e47a32a2789f8c346031978485cbf").use(
- "github.com/mdlayher/netlink",
- "github.com/mdlayher/genetlink",
- )
-
- p.collect(
- "github.com/sbezverk/nfproxy", "7fac5f39824e7f34228b08ba8b7640770ca6a9f4",
- patches("nfproxy.patch"),
- ).use(
- "github.com/sbezverk/nftableslib",
- )
-
- p.collect("github.com/coredns/coredns", "v1.7.0",
- prePatches("coredns-remove-unused-plugins.patch"),
- ).use(
- "github.com/caddyserver/caddy",
- "github.com/dnstap/golang-dnstap",
- "github.com/farsightsec/golang-framestream",
- "github.com/flynn/go-shlex",
- "github.com/grpc-ecosystem/grpc-opentracing",
- "github.com/infobloxopen/go-trees",
- "github.com/miekg/dns",
- "github.com/opentracing/opentracing-go",
- )
-
- // goimports with import grouping patch
- // https://go-review.googlesource.com/c/tools/+/321409/
- p.collectOverride("golang.org/x/tools", "v0.1.2-0.20210518182153-17b346669257", patches("goimports-group-merging.patch"))
-
- // commentwrap is used as a nogo analyzer to stick to a maximum line
- // length for comments.
- p.collect("github.com/corverroos/commentwrap", "2926638be44ce0c6c0ee2471e9b5ad9473c984cd").use(
- "github.com/muesli/reflow",
- )
-
- // Used by metroctl to resolve XDG directories
- p.collect("github.com/adrg/xdg", "v0.4.0")
-
- // Used for generating proto docs in //build/proto_docs
- p.collect("github.com/pseudomuto/protoc-gen-doc", "v1.5.0", patches("protoc-gen-doc-no-gogo.patch")).use(
- "github.com/Masterminds/sprig",
- "github.com/Masterminds/semver",
- "github.com/aokoli/goutils",
- "github.com/huandu/xstrings",
- ).with(
- disabledProtoBuild,
- ).use(
- "github.com/pseudomuto/protokit",
- )
-
- // Used by //metropolis/node/build/mkcpio
- p.collect("github.com/cavaliergopher/cpio", "b4d3577d12deb76a6d52f184acd341f875574785")
-
- // First generate the repositories starlark rule into memory. This is because
- // rendering will lock all unlocked dependencies, which might take a while. If a
- // use were to interrupt it now, they would end up with an incomplete
- // repositories.bzl and would have to restore from git.
- buf := bytes.NewBuffer(nil)
- err = p.render(buf)
- if err != nil {
- return fmt.Errorf("could not render deps: %w", err)
- }
-
- err = os.WriteFile(repositoriesBzlPath, buf.Bytes(), 0666)
- if err != nil {
- return fmt.Errorf("could not write deps: %w", err)
- }
- return nil
-}
diff --git a/build/fietsje/planner.go b/build/fietsje/planner.go
deleted file mode 100644
index 9e4889c..0000000
--- a/build/fietsje/planner.go
+++ /dev/null
@@ -1,251 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-import (
- "fmt"
-)
-
-// The Planner provides the main DSL and high-level control logic for resolving
-// dependencies. It is the main API that fietsje users should consume.
-
-// planner is a builder for a single world of Go package dependencies, and what is
-// then emitted into a Starlark file containing gazelle go_repository rules. The
-// planner's builder system covers three increasingly specific contexts:
-// - planner (this structure, allows for 'collecting' in high-level dependencies. ie. collections)
-// - collection (represents what has been pulled in by a high-level dependency, and allows for 'using' transitive
-// dependencies from a collection)
-// - optionized (represents a collection with extra build flags, eg. disabled proto builds)
-type planner struct {
- // available is a map of importpaths to dependencies that the planner knows. This
- // is a flat structure that is the main source of truth of actual dependency data,
- // like a registry of everything that the planner knows about. The available
- // dependency for a given importpath, as the planner progresses, might change, ie.
- // when there is a version conflict. As such, code should use importpaths as atoms
- // describing dependencies, instead of holding dependency pointers.
- available map[string]*dependency
- // enabled is a map of dependencies that will be emitted by the planner into the
- // build via Gazelle.
- enabled map[string]bool
- // seen is a map of 'dependency' -> 'parent' importpaths, ie. returns what higher-
- // level dependency (ie. one enabled with .collect()) pulled in a given dependency.
- // This is only used for error messages to help the user find what a transitive
- // dependency has been pulled in by.
- seen map[string]string
-
- shelf *shelf
-}
-
-func (p *planner) collect(importpath, version string, opts ...buildOpt) *collection {
- return p.collectInternal(importpath, version, false, opts...)
-}
-
-func (p *planner) collectOverride(importpath, version string, opts ...buildOpt) *collection {
- return p.collectInternal(importpath, version, true, opts...)
-}
-
-// collectInternal pulls in a high-level dependency into the planner and
-// enables it. It also parses all of its transitive // dependencies (not just
-// directly transitive, but recursively transitive) and makes the planner aware
-// of them. It does not enable these transitive dependencies, but returns a
-// collection builder, which can be used to do se by calling .use().
-func (p *planner) collectInternal(importpath, version string, override bool, opts ...buildOpt) *collection {
- // Ensure overrides are explicit and minimal.
- by, ok := p.seen[importpath]
- if ok && !override {
- panic(fmt.Errorf("%s is being collected, but has already been declared by %s; replace it by a use(%q) call on %s or use collectOverride", importpath, by, importpath, by))
- }
- if !ok && override {
- panic(fmt.Errorf("%s is being collected with override, but has not been seen as a dependency previously - use .collect(%q, %q) instead", importpath, importpath, version))
- }
-
- d := &dependency{
- shelf: p.shelf,
- importpath: importpath,
- version: version,
- }
- for _, o := range opts {
- o(d)
- }
-
- // automatically enable direct import
- p.enabled[d.importpath] = true
- p.available[d.importpath] = d
-
- td, err := d.getTransitiveDeps()
- if err != nil {
- panic(fmt.Errorf("could not get transitive deps for %q: %v", d.importpath, err))
- }
- // add transitive deps to 'available' map
- for k, v := range td {
- // skip dependencies that have already been enabled, dependencies are 'first
- // enabled version wins'.
- if _, ok := p.available[k]; ok && p.enabled[k] {
- continue
- }
-
- p.available[k] = v
-
- // make note of the high-level dependency that pulled in the dependency.
- p.seen[v.importpath] = d.importpath
- }
-
- return &collection{
- p: p,
- highlevel: d,
- transitive: td,
- }
-}
-
-// collection represents the context of the planner after pulling/collecting in a
-// high-level dependency. In this state, the planner can be used to enable
-// transitive dependencies of the high-level dependency.
-type collection struct {
- p *planner
-
- highlevel *dependency
- transitive map[string]*dependency
-}
-
-// use enables given dependencies defined in the collection by a high-level
-// dependency.
-func (c *collection) use(paths ...string) *collection {
- return c.with().use(paths...)
-}
-
-// replace injects a new dependency with a replacement importpath. This is used to
-// reflect 'replace' stanzas in go.mod files of third-party dependencies. This is
-// not done automatically by Fietsje, as a replacement is global to the entire
-// build tree, and should be done knowingly and explicitly by configuration. The
-// 'oldpath' importpath will be visible to the build system, but will be backed at
-// 'newpath' locked at 'version'.
-func (c *collection) replace(oldpath, newpath, version string) *collection {
- // Ensure oldpath is in use. We want as little replacements as possible, and if
- // it's not being used by anything, it means that we likely don't need it.
- c.use(oldpath)
-
- d := c.highlevel.child(oldpath, version)
- d.replace = newpath
- c.transitive[oldpath] = d
- c.p.available[oldpath] = d
- c.p.enabled[oldpath] = true
-
- return c
-}
-
-// inject adds a dependency to a collection as if requested by the high-level
-// dependency of the collection. This should be used sparingly, for instance when
-// high-level dependencies contain bazel code that uses some external workspaces
-// from Go modules, and those workspaces are not defined in parsed transitive
-// dependency definitions like go.mod/sum.
-func (c *collection) inject(importpath, version string, opts ...buildOpt) *collection {
- d := c.highlevel.child(importpath, version)
- c.transitive[importpath] = d
- c.p.available[importpath] = d
- c.p.enabled[importpath] = true
- for _, o := range opts {
- o(c.transitive[importpath])
- }
-
- return c
-}
-
-// with transforms a collection into an optionized, by setting some build options.
-func (c *collection) with(o ...buildOpt) *optionized {
- return &optionized{
- c: c,
- opts: o,
- }
-}
-
-// optionized is a collection that has some build options set, that will be applied
-// to all dependencies 'used' in this context
-type optionized struct {
- c *collection
- opts []buildOpt
-}
-
-// buildOpt is a build option passed to Gazelle.
-type buildOpt func(d *dependency)
-
-// buildTags sets the given buildTags in affected dependencies.
-func buildTags(tags ...string) buildOpt {
- return func(d *dependency) {
- d.buildTags = tags
- }
-}
-
-// disabledProtoBuild disables protobuf builds in affected dependencies.
-func disabledProtoBuild(d *dependency) {
- d.disableProtoBuild = true
-}
-
-// patches applies patches in affected dependencies after BUILD file generation.
-func patches(patches ...string) buildOpt {
- return func(d *dependency) {
- d.patches = patches
- }
-}
-
-// prePatches applies patches in affected dependencies before BUILD file
-// generation.
-func prePatches(patches ...string) buildOpt {
- return func(d *dependency) {
- d.prePatches = patches
- }
-}
-
-// useImportAliasNaming instructs Gazelle to name the generated targets in a way
-// which is both compatible with the old go_default_library convention as well
-// as the import convention. See `go_naming_convention=import_alias` in Gazelle.
-func useImportAliasNaming(d *dependency) {
- d.useImportAliasNaming = true
-}
-
-func forceBazelGeneration(d *dependency) {
- d.forceBazelGeneration = true
-}
-
-func buildExtraArgs(args ...string) buildOpt {
- return func(d *dependency) {
- d.buildExtraArgs = args
- }
-}
-
-// use enables given dependencies defined in the collection by a high-level
-// dependency, with any set build options. After returning, the builder degrades to
-// a collection - ie, all build options are reset.
-func (o *optionized) use(paths ...string) *collection {
- for _, path := range paths {
- el, ok := o.c.transitive[path]
- if !ok {
- msg := fmt.Sprintf("dependency %q not found in %q", path, o.c.highlevel.importpath)
- if alternative, ok := o.c.p.seen[path]; ok {
- msg += fmt.Sprintf(" (but found in %q)", alternative)
- } else {
- msg += " or any other collected library"
- }
- panic(msg)
- }
- for _, o := range o.opts {
- o(el)
- }
- o.c.p.enabled[path] = true
- }
-
- return o.c
-}
diff --git a/build/fietsje/proto/BUILD.bazel b/build/fietsje/proto/BUILD.bazel
deleted file mode 100644
index d1a3b83..0000000
--- a/build/fietsje/proto/BUILD.bazel
+++ /dev/null
@@ -1,23 +0,0 @@
-load("@rules_proto//proto:defs.bzl", "proto_library")
-load("@io_bazel_rules_go//go:def.bzl", "go_library")
-load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")
-
-proto_library(
- name = "proto_proto",
- srcs = ["shelf.proto"],
- visibility = ["//visibility:public"],
-)
-
-go_proto_library(
- name = "proto_go_proto",
- importpath = "source.monogon.dev/build/fietsje/proto",
- proto = ":proto_proto",
- visibility = ["//visibility:public"],
-)
-
-go_library(
- name = "proto",
- embed = [":proto_go_proto"],
- importpath = "source.monogon.dev/build/fietsje/proto",
- visibility = ["//visibility:public"],
-)
diff --git a/build/fietsje/proto/shelf.proto b/build/fietsje/proto/shelf.proto
deleted file mode 100644
index fa454bb..0000000
--- a/build/fietsje/proto/shelf.proto
+++ /dev/null
@@ -1,33 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-syntax = "proto3";
-package build.fietsje;
-option go_package = "source.monogon.dev/build/fietsje/proto";
-
-// The Shelf is a cache/lockfile of Fietsje dependencies. See //build/fietsje/shelf.go for more information.
-// This proto definition is used to generate text proto files. As such, repeated fields have singular names.
-
-message Shelf {
- message Entry {
- string import_path = 1;
- string version = 2;
- string bazel_name = 3;
- string sum = 4;
- string semver = 5;
- }
- repeated Entry entry = 1;
-}
diff --git a/build/fietsje/render.go b/build/fietsje/render.go
deleted file mode 100644
index 5ad26d0..0000000
--- a/build/fietsje/render.go
+++ /dev/null
@@ -1,99 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-import (
- "fmt"
- "io"
- "sort"
-)
-
-// render writes a gazelle-compatible starlark file based on the enabled
-// dependencies in this planner.
-func (p *planner) render(w io.Writer) error {
- fmt.Fprintln(w, `load("@bazel_gazelle//:deps.bzl", "go_repository")`)
- fmt.Fprintln(w, ``)
- fmt.Fprintln(w, `def go_repositories():`)
-
- // Get and sort all enabled importpaths.
- var enabled []string
- for importpath, _ := range p.enabled {
- enabled = append(enabled, importpath)
- }
- sort.Slice(enabled, func(i, j int) bool { return enabled[i] < enabled[j] })
-
- // Render all importpaths.
- for _, importpath := range enabled {
- d := p.available[importpath]
- if err := d.lock(); err != nil {
- return fmt.Errorf("could not lock %q: %v", importpath, err)
- }
-
- fmt.Fprintf(w, " go_repository(\n")
- fmt.Fprintf(w, " name = %q,\n", d.locked.bazelName)
- fmt.Fprintf(w, " importpath = %q,\n", d.importpath)
- fmt.Fprintf(w, " version = %q,\n", d.locked.semver)
- fmt.Fprintf(w, " sum = %q,\n", d.locked.sum)
- if d.replace != "" {
- fmt.Fprintf(w, " replace = %q,\n", d.replace)
- }
- if d.disableProtoBuild {
- fmt.Fprintf(w, " build_file_proto_mode = %q,\n", "disable")
- }
- if d.forceBazelGeneration {
- fmt.Fprintf(w, " build_file_generation = %q,\n", "on")
- }
- if d.buildTags != nil {
- fmt.Fprintf(w, " build_tags = [\n")
- for _, tag := range d.buildTags {
- fmt.Fprintf(w, " %q,\n", tag)
- }
- fmt.Fprintf(w, " ],\n")
- }
- if d.patches != nil {
- fmt.Fprintf(w, " patches = [\n")
- for _, patch := range d.patches {
- fmt.Fprintf(w, " %q,\n", "//third_party/go/patches:"+patch)
- }
- fmt.Fprintf(w, " ],\n")
- }
- if d.prePatches != nil {
- fmt.Fprintf(w, " pre_patches = [\n")
- for _, patch := range d.prePatches {
- fmt.Fprintf(w, " %q,\n", "//third_party/go/patches:"+patch)
- }
- fmt.Fprintf(w, " ],\n")
- }
- if d.patches != nil || d.prePatches != nil {
- fmt.Fprintf(w, " patch_args = [%q],\n", "-p1")
- }
- fmt.Fprintf(w, " build_extra_args = [\n")
- if d.useImportAliasNaming {
- fmt.Fprintf(w, " %q,\n", "-go_naming_convention=import_alias")
- fmt.Fprintf(w, " %q,\n", "-go_naming_convention_external=import_alias")
- } else {
- fmt.Fprintf(w, " %q,\n", "-go_naming_convention=go_default_library")
- fmt.Fprintf(w, " %q,\n", "-go_naming_convention_external=go_default_library")
- }
- for _, arg := range d.buildExtraArgs {
- fmt.Fprintf(w, " %q,\n", arg)
- }
- fmt.Fprintf(w, " ],\n")
- fmt.Fprintf(w, " )\n")
- }
- return nil
-}
diff --git a/build/fietsje/shelf.go b/build/fietsje/shelf.go
deleted file mode 100644
index c377186..0000000
--- a/build/fietsje/shelf.go
+++ /dev/null
@@ -1,162 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-import (
- "bytes"
- "fmt"
- "log"
- "os"
- "sort"
-
- "github.com/golang/protobuf/proto"
-
- pb "source.monogon.dev/build/fietsje/proto"
-)
-
-// The Shelf is a combined cache and dependency lockfile, not unlike go.sum. It's
-// implemented as a text proto file on disk, and currently stores a single mapping
-// of shelfKeys to shelfValues, which are in order a (importpath, version) tuple
-// and the `locked` structure of a dependency. The resulting shelf file should be
-// commited to the monogon repository. It can be freely deleted to force recreation
-// from scratch, which can be useful as there is no garbage collection implemented
-// for it. The 'lockfile' aspect of the Shelf is counter-intuitive to what readers
-// might be used to from other dependency management systems. It does not lock a
-// third-party dependency to a particular version, but only locks a well defined
-// version to its checksum. As such, recreating the shelf from scratch should not
-// bump any dependencies, unless some upstream-project retagged a release to a
-// different VCS commit, or a fietsje user pinned to 'master' instead of a
-// particular commit. The effective changes will always be reflected in the
-// resulting starlark repository ruleset, which (also being commited to source
-// control) can be used as a canary of a version being effectively bumped.
-
-// shelfKey is the key into the shelf map structure.
-type shelfKey struct {
- importpath string
- version string
-}
-
-// shelfValue is the entry of a shelf map structure.
-type shelfValue struct {
- l *locked
-}
-
-// shelf is an in-memory representation of the shelf loaded from disk.
-type shelf struct {
- path string
- data map[shelfKey]shelfValue
-}
-
-func shelfLoad(path string) (*shelf, error) {
- var data []byte
- var err error
-
- if _, err := os.Stat(path); os.IsNotExist(err) {
- log.Printf("Creating new shelf file at %q, this run will be slow.", path)
- } else {
- data, err = os.ReadFile(path)
- if err != nil {
- return nil, fmt.Errorf("could not read shelf: %v", err)
- }
- }
- var shelfProto pb.Shelf
- err = proto.UnmarshalText(string(data), &shelfProto)
- if err != nil {
- return nil, fmt.Errorf("could not unmarshal shelf: %v", err)
- }
-
- res := &shelf{
- path: path,
- data: make(map[shelfKey]shelfValue),
- }
-
- for _, e := range shelfProto.Entry {
- k := shelfKey{
- importpath: e.ImportPath,
- version: e.Version,
- }
- v := shelfValue{
- l: &locked{
- bazelName: e.BazelName,
- sum: e.Sum,
- semver: e.Semver,
- },
- }
- res.data[k] = v
- }
- return res, nil
-}
-
-// get retrieves a given lock entry from the in-memory shelf.
-func (s *shelf) get(importpath, version string) *locked {
- res, ok := s.data[shelfKey{importpath: importpath, version: version}]
- if !ok {
- return nil
- }
- return res.l
-}
-
-// put stores a given locked entry in memory. This will not be commited to disk
-// until .save() is called.
-func (s *shelf) put(importpath, version string, l *locked) {
- s.data[shelfKey{importpath: importpath, version: version}] = shelfValue{l: l}
-}
-
-// save commits the shelf to disk (to the same location it was loaded from), fully
-// overwriting from in-memory data.
-func (s *shelf) save() error {
- // Build proto representation of shelf data.
- var shelfProto pb.Shelf
- for k, v := range s.data {
- shelfProto.Entry = append(shelfProto.Entry, &pb.Shelf_Entry{
- ImportPath: k.importpath,
- Version: k.version,
- BazelName: v.l.bazelName,
- Sum: v.l.sum,
- Semver: v.l.semver,
- })
- }
-
- // Sort shelf keys by importpath, then by version.
- sort.Slice(shelfProto.Entry, func(i, j int) bool {
- a := shelfProto.Entry[i]
- b := shelfProto.Entry[j]
-
- if a.ImportPath < b.ImportPath {
- return true
- }
- if a.ImportPath > b.ImportPath {
- return false
- }
- return a.Version < b.Version
- })
-
- // Make an in-memory representation of the marshaled shelf.
- buf := bytes.NewBuffer(nil)
- err := proto.MarshalText(buf, &shelfProto)
- if err != nil {
- return fmt.Errorf("could not serialize shelf: %v", err)
- }
-
- // And write it out.
- err = os.WriteFile(s.path, buf.Bytes(), 0644)
- if err != nil {
- return fmt.Errorf("could not write shelf: %v", err)
- }
-
- return nil
-}
diff --git a/build/fietsje/transitive.go b/build/fietsje/transitive.go
deleted file mode 100644
index 6ffd594..0000000
--- a/build/fietsje/transitive.go
+++ /dev/null
@@ -1,192 +0,0 @@
-// Copyright 2020 The Monogon Project Authors.
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package fietsje
-
-import (
- "fmt"
- "log"
- "os"
- "strings"
-
- "golang.org/x/mod/modfile"
-)
-
-// getTransitiveDeps is a hairy ball of heuristic used to find all recursively
-// transitive dependencies of a given dependency. It downloads a given dependency
-// using `go get`, and performs analysis of standard (go.mod/go.sum) and project-
-// specific dependency management configuration/lock files in order to build a full
-// view of all known, versioned transitive dependencies.
-func (d *dependency) getTransitiveDeps() (map[string]*dependency, error) {
- // First, lock the dependency. Downloading it later will also return a sum, and we
- // want to ensure both are the same.
- err := d.lock()
- if err != nil {
- return nil, fmt.Errorf("could not lock: %v", err)
- }
-
- _, path, sum, err := d.download()
- if err != nil {
- return nil, fmt.Errorf("could not download: %v", err)
- }
-
- if sum != d.locked.sum {
- return nil, fmt.Errorf("inconsistent sum: %q downloaded, %q in shelf/lock", sum, d.locked.sum)
- }
-
- exists := func(p string) bool {
- full := fmt.Sprintf("%s/%s", path, p)
- if _, err := os.Stat(full); err == nil {
- return true
- }
- if err != nil && !os.IsExist(err) {
- panic(fmt.Sprintf("checking file %q: %v", full, err))
- }
- return false
- }
-
- read := func(p string) []byte {
- full := fmt.Sprintf("%s/%s", path, p)
- data, err := os.ReadFile(full)
- if err != nil {
- panic(fmt.Sprintf("reading file %q: %v", full, err))
- }
- return data
- }
-
- requirements := make(map[string]*dependency)
-
- // Read & parse go.mod if present.
- var mf *modfile.File
- if exists("go.mod") {
- log.Printf("%q: parsing go.mod\n", d.importpath)
- data := read("go.mod")
- mf, err = modfile.Parse("go.mod", data, nil)
- if err != nil {
- return nil, fmt.Errorf("parsing go.mod in %s: %v", d.importpath, err)
- }
- }
-
- // If a go.mod file was present, interpret it to populate dependencies.
- if mf != nil {
- for _, req := range mf.Require {
- requirements[req.Mod.Path] = d.child(req.Mod.Path, req.Mod.Version)
- }
- for _, rep := range mf.Replace {
- // skip filesystem rewrites
- if rep.New.Version == "" {
- continue
- }
-
- requirements[rep.New.Path] = d.child(rep.New.Path, rep.New.Version)
- }
- }
-
- // Read parse, and interpret. go.sum if present.
- // This should bring into view all recursively transitive dependencies.
- if exists("go.sum") {
- log.Printf("%q: parsing go.sum", d.importpath)
- data := read("go.sum")
- for _, line := range strings.Split(string(data), "\n") {
- line = strings.TrimSpace(line)
- if line == "" {
- continue
- }
-
- parts := strings.Fields(line)
- if len(parts) != 3 {
- return nil, fmt.Errorf("parsing go.sum: unparseable line %q", line)
- }
-
- importpath, version := parts[0], parts[1]
-
- // Skip if already created from go.mod.
- // TODO(q3k): error if go.sum and go.mod disagree?
- if _, ok := requirements[importpath]; ok {
- continue
- }
-
- if strings.HasSuffix(version, "/go.mod") {
- version = strings.TrimSuffix(version, "/go.mod")
- }
- requirements[importpath] = d.child(importpath, version)
- }
- }
-
- // Special case: root Kubernetes repo - rewrite staging/ deps to k8s.io/ at correct
- // versions, quit early. Kubernetes vendors all dependencies into vendor/, and also
- // contains sub-projects (components) in staging/. This converts all staging
- // dependencies into appropriately versioned k8s.io/<dep> paths.
- if d.importpath == "k8s.io/kubernetes" {
- log.Printf("%q: special case for Kubernetes main repository", d.importpath)
- if mf == nil {
- return nil, fmt.Errorf("k8s.io/kubernetes needs a go.mod")
- }
- // extract the version, turn into component version
- version := d.version
- if !strings.HasPrefix(version, "v") {
- return nil, fmt.Errorf("invalid version format for k8s: %q", version)
- }
- version = version[1:]
- componentVersion := fmt.Sprintf("kubernetes-%s", version)
-
- // find all k8s.io 'components'
- components := make(map[string]bool)
- for _, rep := range mf.Replace {
- if !strings.HasPrefix(rep.Old.Path, "k8s.io/") || !strings.HasPrefix(rep.New.Path, "./staging/src/") {
- continue
- }
- components[rep.Old.Path] = true
- }
-
- // add them to planner at the 'kubernetes-$ver' tag
- for component, _ := range components {
- requirements[component] = d.child(component, componentVersion)
- }
- return requirements, nil
- }
-
- // Special case: github.com/containerd/containerd: read vendor.conf.
- if d.importpath == "github.com/containerd/containerd" {
- log.Printf("%q: special case for containerd", d.importpath)
- if !exists("vendor.conf") {
- panic("containerd needs vendor.conf")
- }
- data := read("vendor.conf")
- for _, line := range strings.Split(string(data), "\n") {
- // strip comments
- parts := strings.SplitN(line, "#", 2)
- line = parts[0]
-
- // skip empty contents
- line = strings.TrimSpace(line)
- if line == "" {
- continue
- }
-
- // read dep/version pairs
- parts = strings.Fields(line)
- if len(parts) < 2 {
- return nil, fmt.Errorf("unparseable line in containerd vendor.conf: %q", line)
- }
- importpath, version := parts[0], parts[1]
- requirements[importpath] = d.child(importpath, version)
- }
- return requirements, nil
- }
-
- return requirements, nil
-}