*: migrate to CC toolchains and Bazel 5.4.0
Change-Id: Iff3c0ddda4413dd0c5fa657a5b7813223e98611e
Reviewed-on: https://review.monogon.dev/c/monogon/+/1079
Tested-by: Jenkins CI
Reviewed-by: Lorenz Brun <lorenz@monogon.tech>
diff --git a/build/ci/jenkins-presubmit.groovy b/build/ci/jenkins-presubmit.groovy
index 54d89be..cf7b705 100644
--- a/build/ci/jenkins-presubmit.groovy
+++ b/build/ci/jenkins-presubmit.groovy
@@ -3,7 +3,7 @@
// open Gerrit change request.
// TODO(leo): remove once CI image has been updated.
-def gazelle_build = "bazel --noworkspace_rc run go install github.com/bazelbuild/bazelisk@v1.15.0"
+def gazelle_build = "curl -o ~/bazelisk https://storage.googleapis.com/monogon-infra-public/bazelisk-v1.15.0 && chmod +x ~/bazelisk"
pipeline {
agent none
@@ -25,8 +25,8 @@
echo "Gerrit change: ${GERRIT_CHANGE_URL}"
sh "git clean -fdx -e '/bazel-*'"
sh gazelle_build
- sh "JENKINS_NODE_COOKIE=dontKillMe ~/go/bin/bazelisk test //..."
- sh "JENKINS_NODE_COOKIE=dontKillMe ~/go/bin/bazelisk test -c dbg //..."
+ sh "JENKINS_NODE_COOKIE=dontKillMe ~/bazelisk test //..."
+ sh "JENKINS_NODE_COOKIE=dontKillMe ~/bazelisk test -c dbg //..."
}
post {
success {
@@ -50,8 +50,8 @@
echo "Gerrit change: ${GERRIT_CHANGE_URL}"
sh "git clean -fdx -e '/bazel-*'"
sh gazelle_build
- sh "JENKINS_NODE_COOKIE=dontKillMe ~/go/bin/bazelisk run //:gazelle-update-repos"
- sh "JENKINS_NODE_COOKIE=dontKillMe ~/go/bin/bazelisk run //:gazelle -- update"
+ sh "JENKINS_NODE_COOKIE=dontKillMe ~/bazelisk run //:gazelle-update-repos"
+ sh "JENKINS_NODE_COOKIE=dontKillMe ~/bazelisk run //:gazelle -- update"
script {
def diff = sh script: "git status --porcelain", returnStdout: true
diff --git a/build/platforms/BUILD.bazel b/build/platforms/BUILD.bazel
new file mode 100644
index 0000000..e612a9c
--- /dev/null
+++ b/build/platforms/BUILD.bazel
@@ -0,0 +1,28 @@
+# Generic platform for Linux x86_64 targets.
+platform(
+ name = "linux_amd64",
+ constraint_values = [
+ "@platforms//os:linux",
+ "@platforms//cpu:x86_64",
+ ],
+)
+
+# EFI preboot environment for x86_64 machines.
+platform(
+ name = "efi_amd64",
+ constraint_values = [
+ "//build/platforms/os:efi",
+ "@platforms//cpu:x86_64",
+ ],
+)
+
+# Linux x86_64 platform with static linking
+# (i.e. Metropolis node, scratch containers...).
+platform(
+ name = "linux_amd64_static",
+ constraint_values = [
+ "@platforms//os:linux",
+ "@platforms//cpu:x86_64",
+ "//build/platforms/linkmode:musl-static",
+ ],
+)
diff --git a/build/platforms/linkmode/BUILD.bazel b/build/platforms/linkmode/BUILD.bazel
new file mode 100644
index 0000000..8a7f99c
--- /dev/null
+++ b/build/platforms/linkmode/BUILD.bazel
@@ -0,0 +1,14 @@
+package(
+ default_visibility = ["//visibility:public"],
+)
+
+# Minimal Metropolis node environment. No dynamic linker,
+# so everything has to be statically linked.
+constraint_value(
+ name = "musl-static",
+ constraint_setting = ":linkmode",
+)
+
+constraint_setting(
+ name = "linkmode",
+)
diff --git a/build/platforms/os/BUILD.bazel b/build/platforms/os/BUILD.bazel
new file mode 100644
index 0000000..3775b39
--- /dev/null
+++ b/build/platforms/os/BUILD.bazel
@@ -0,0 +1,9 @@
+package(
+ default_visibility = ["//visibility:public"],
+)
+
+# EFI preboot environment
+constraint_value(
+ name = "efi",
+ constraint_setting = "@platforms//os:os",
+)
diff --git a/build/proto_docs/proto_docs.bzl b/build/proto_docs/proto_docs.bzl
index 98661a2..d929649 100644
--- a/build/proto_docs/proto_docs.bzl
+++ b/build/proto_docs/proto_docs.bzl
@@ -25,7 +25,7 @@
# "duplicate" types.
# Since generating documentation for well-known types is not that useful just
# skip them.
- if src.path.find("/bin/external/com_google_protobuf/_virtual_imports/") != -1:
+ if src.path.find("/bin/external/com_github_protocolbuffers_protobuf/_virtual_imports/") != -1:
continue
args.append(src.path)
diff --git a/build/toolbase/gotoolchain/def.bzl b/build/toolbase/gotoolchain/def.bzl
index 5c421e6..2ff5fe0 100644
--- a/build/toolbase/gotoolchain/def.bzl
+++ b/build/toolbase/gotoolchain/def.bzl
@@ -17,7 +17,7 @@
template = ctx.file._template,
output = out,
substitutions = {
- "GOROOT": go.root,
+ "GOROOT": go.sdk.root_file.dirname,
"GOTOOL": go.go.path,
},
)
diff --git a/build/toolchain/BUILD b/build/toolchain/BUILD
index bd8c307..4a12064 100644
--- a/build/toolchain/BUILD
+++ b/build/toolchain/BUILD
@@ -4,33 +4,25 @@
#
# We currently define two toolchains:
#
-# - //build/toolchain:host_cc_suite , which is a fully unhermetic host toolchain,
-# that can be used to build tools for the host.
-# - //build/toolchain/musl-host-gcc:musl_host_cc_suite , which combines the host's
+# - //build/toolchain:host_cc_toolchain , which points to our hermetic sandbox
+# sysroot default compiler toolchain. It is mainly used to target the execution platform
+# inside the sandbox (i.e. build tooling).
+# - //build/toolchain/musl-host-gcc:musl_host_toolchain , which combines the sandbox sysroot
# gcc compiler with a sysroot tarball that targets the Metropolis node
# runtime. This can be used to build C libraries/tools running within the
-# Metropolis node image.
-#
+# Metropolis node image or on the (unknown) host operating system outside the sandbox.
-# This file defines //build/toolchain:host_cc_suite.
+# This file defines //build/toolchain:host_cc_toolchain.
#
-# This is a C++ toolchain that uses GCC from the host at hardcoded paths. We
-# can get away with this, as currently the entire build is performed in a known
-# container (see: //scripts:create_container.sh). We define this toolchain so
+# This is a C++ toolchain that uses GCC from the sandbox sysroot at hardcoded paths. We
+# can get away with this, as currently the entire build is performed in a hermetic
+# sandbox sysroot (see: //third_party/sandboxroot). We define this toolchain so
# that we have full control over all configuration of it, which we need as we
# are building some fairly odd C binaries (notably, a qboot bootloader for
# testing).
#
-# The host_cc toolchain suite is enabled for all cc_* targets that aren't
-# building host tools by setting --crosstool_top in .bazelrc. In the future,
-# this should only be triggered by transitions where necessary.
-#
-# In the future, the host_cc toolchains should be replaced by a hermetic
-# toolchain that's built locally, or downloaded from the Internet - as
-# github.com/bazelbuild/bazel-toolchains does it. As that's being built, we
-# should then also have another toolchain definition for C binaries that
-# target static binaries for Metropolis nodes, so that mkfs.xfs can be built
-# using native cc_* rules, too.
+# The host_cc toolchain suite is enabled for all cc_* targets whose
+# platform isn't matching a more specific toolchain.
#
# This, and :cc_toolchain_config.bzl is based on the following tutorial:
# https://docs.bazel.build/versions/master/tutorial/cc-toolchain-config.html
@@ -39,16 +31,11 @@
filegroup(name = "empty")
-cc_toolchain_suite(
- name = "host_cc_suite",
- toolchains = {
- "k8": ":host_cc_k8_toolchain",
- },
-)
-
cc_toolchain(
name = "host_cc_k8_toolchain",
all_files = ":empty",
+ ar_files = ":empty",
+ as_files = ":empty",
compiler_files = ":empty",
dwp_files = ":empty",
linker_files = ":empty",
@@ -60,3 +47,17 @@
)
host_cc_toolchain_config(name = "host_cc_k8_toolchain_config")
+
+toolchain(
+ name = "host_cc_toolchain",
+ exec_compatible_with = [
+ "@bazel_tools//platforms:linux",
+ "@bazel_tools//platforms:x86_64",
+ ],
+ target_compatible_with = [
+ "@bazel_tools//platforms:linux",
+ "@bazel_tools//platforms:x86_64",
+ ],
+ toolchain = ":host_cc_k8_toolchain",
+ toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
+)
diff --git a/build/toolchain/cc_toolchain_config.bzl b/build/toolchain/cc_toolchain_config.bzl
index cebf18f..7a949fa 100644
--- a/build/toolchain/cc_toolchain_config.bzl
+++ b/build/toolchain/cc_toolchain_config.bzl
@@ -167,6 +167,10 @@
name = "strip",
path = "/bin/false",
),
+ tool_path(
+ name = "objcopy",
+ path = "/usr/bin/objcopy",
+ ),
]
return cc_common.create_cc_toolchain_config_info(
diff --git a/build/toolchain/llvm-efi/BUILD b/build/toolchain/llvm-efi/BUILD
index 3cfa67f..5feb67f 100644
--- a/build/toolchain/llvm-efi/BUILD
+++ b/build/toolchain/llvm-efi/BUILD
@@ -4,13 +4,6 @@
filegroup(name = "empty")
-cc_toolchain_suite(
- name = "efi_cc_suite",
- toolchains = {
- "k8": ":efi_k8_cc_toolchain",
- },
-)
-
filegroup(
name = "fltused",
srcs = ["fltused.o"],
@@ -19,6 +12,8 @@
cc_toolchain(
name = "efi_k8_cc_toolchain",
all_files = ":empty",
+ ar_files = ":empty",
+ as_files = ":empty",
compiler_files = ":empty",
dwp_files = ":empty",
linker_files = ":fltused",
@@ -30,3 +25,17 @@
)
efi_k8_cc_toolchain_config(name = "efi_k8_cc_toolchain_config")
+
+toolchain(
+ name = "efi_k8_toolchain",
+ exec_compatible_with = [
+ "@platforms//cpu:x86_64",
+ "@platforms//os:linux",
+ ],
+ target_compatible_with = [
+ "@platforms//cpu:x86_64",
+ "//build/platforms/os:efi",
+ ],
+ toolchain = ":efi_k8_cc_toolchain",
+ toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
+)
diff --git a/build/toolchain/llvm-efi/README.md b/build/toolchain/llvm-efi/README.md
index 5f1b9c8..0360ec5 100644
--- a/build/toolchain/llvm-efi/README.md
+++ b/build/toolchain/llvm-efi/README.md
@@ -1,19 +1,18 @@
llvm-efi
========
-llvm-efi is a Bazel cc toolchain that uses the machine's host LLVM/clang with flags targeting freestanding EFI.
+llvm-efi is a Bazel cc toolchain that uses the sandbox sysroot LLVM/clang with flags targeting freestanding EFI.
EFI headers are not shipped as part of the toolchain, but are available as a cc_library from `@gnuefi//:gnuefi`.
-At some point, this toolchain should be replaced by a fully hermetic toolchain that doesn't depend on the host environment.
-
Usage
-----
To use this toolchain explicitly while building a `cc_binary`, do:
- bazel build --crosstool_top=//build/toolchain/llvm-efi:efi_cc_suite //foo/bar
+ bazel build --platforms=//build/platforms:efi_amd64 //foo/bar
-During an actual build however, the right toolchain should be selected using aspects or other Bazel configurability features, instead of a hardcoded `--crosstool_top`.
+During an actual build however, the right toolchain should be selected using transitions
+or other configuration mechanisms.
fltused
-------
diff --git a/build/toolchain/llvm-efi/transition.bzl b/build/toolchain/llvm-efi/transition.bzl
index e2f7a8f..1edb86b 100644
--- a/build/toolchain/llvm-efi/transition.bzl
+++ b/build/toolchain/llvm-efi/transition.bzl
@@ -3,13 +3,13 @@
Transition that enables building for an EFI environment. Currently only supports C code.
"""
return {
- "//command_line_option:crosstool_top": "//build/toolchain/llvm-efi:efi_cc_suite",
+ "//command_line_option:platforms": "//build/platforms:efi_amd64"
}
build_efi_transition = transition(
implementation = _build_efi_transition_impl,
inputs = [],
outputs = [
- "//command_line_option:crosstool_top",
+ "//command_line_option:platforms",
],
)
diff --git a/build/toolchain/musl-host-gcc/BUILD b/build/toolchain/musl-host-gcc/BUILD
index 656bb45..5b83901 100644
--- a/build/toolchain/musl-host-gcc/BUILD
+++ b/build/toolchain/musl-host-gcc/BUILD
@@ -1,23 +1,18 @@
load("//build/toolchain:cc_toolchain_config.bzl", "host_cc_toolchain_config")
-# This file defines //build/toolchain/musl-host-gcc:musl_host_cc_suite.
+# This file defines //build/toolchain/musl-host-gcc:musl_host_toolchain.
#
-# This is a C++ toolchain that uses GCC from the host at hardcoded paths, with
+# This is a C++ toolchain that uses GCC from the sandbox sysroot at hardcoded paths, with
# a pre-built sysroot tarball that targets Metropolis nodes with musl and Linux
-# headers. It's a superset of //build/toolchain:host_cc_suite.
+# headers. It's a superset of //build/toolchain:host_cc_toolchain.
+#
# For more information, see README.md.
-cc_toolchain_suite(
- name = "musl_host_cc_suite",
- toolchains = {
- "k8": ":musl_host_cc_k8_toolchain",
- },
- visibility = ["//visibility:public"],
-)
-
cc_toolchain(
name = "musl_host_cc_k8_toolchain",
all_files = ":musl_toolchain_files",
+ ar_files = ":musl_toolchain_files",
+ as_files = ":musl_toolchain_files",
compiler_files = ":musl_toolchain_files",
dwp_files = ":musl_toolchain_files",
linker_files = ":musl_toolchain_files",
@@ -45,3 +40,18 @@
"@musl_sysroot//:all",
],
)
+
+toolchain(
+ name = "musl_host_toolchain",
+ exec_compatible_with = [
+ "@platforms//cpu:x86_64",
+ "@platforms//os:linux",
+ ],
+ target_compatible_with = [
+ "@platforms//cpu:x86_64",
+ "@platforms//os:linux",
+ "//build/platforms/linkmode:musl-static",
+ ],
+ toolchain = ":musl_host_cc_k8_toolchain",
+ toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
+)
diff --git a/build/toolchain/musl-host-gcc/README.md b/build/toolchain/musl-host-gcc/README.md
index 367b4a7..a5e9222 100644
--- a/build/toolchain/musl-host-gcc/README.md
+++ b/build/toolchain/musl-host-gcc/README.md
@@ -1,20 +1,21 @@
musl-host-gcc
=============
-musl-host-gcc is a Bazel C++ toolchain that uses the machine's host gcc in combination with a pre-built musl, musl headers, and Linux headers.
+musl-host-gcc is a Bazel C++ toolchain that uses the sandbox sysroot gcc in combination with a pre-built musl, musl headers, and Linux headers.
It is currently used to build the few C binaries we need on Metropolis nodes.
-At some point, this toolchain should be replaced by a fully hermetic toolchain that doesn't depend on the host environment.
+At some point, this toolchain should be improved to directly consume a static compiler toolchain and sysroot, so we can eventually get rid of the sandbox (like Aspect's [gcc-toolchain](https://github.com/aspect-build/gcc-toolchain) is doing).
Usage
-----
To use this toolchain explicitly while building a `cc_binary`, do:
- bazel build --crosstool_top=//build/toolchain/musl-host-gcc:musl_host_cc_suite //foo/bar
+ bazel build --platforms=//build/platforms:linux_amd64_static //foo/bar
-During an actual build however, the right toolchain should be selected using aspects or other Bazel configurability features, instead of a hardcoded `--crosstool_top`.
+During an actual build however, the right toolchain should be selected using transitions
+or other configuration mechanisms.
Building Toolchain Sysroot Tarball
----------------------------------
@@ -36,7 +37,7 @@
1. `//build/toolchain/musl-host-gcc/sysroot` is used to build `//build/toolchain/musl-host-gcc/sysroot.tar.xz` which is a tarball that contains all include and binary library files for building against musl for Metropolis nodes (x86\_64 / k8) - these are musl headers, musl libraries, and linux headers. This tarball is committed to source control.
1. When building a target that uses the toolchain, the `sysroot.tar.xz` tarball is extracted into an external repository `@musl_sysroot`, via `sysroot.bzl` and `sysroot_repository.bzl`.
1. A toolchain config is built using `//build/toolchain:cc_toolchain_config.bzl`, which points at `gcc-wrapper.sh` as its gcc entrypoint. `gcc-wrapper.sh` expects to be able to call the host gcc with `musl.spec`.
-1. A toolchain is built in `//build/toolchain/musl-host-gcc:musl_host_cc_suite`, which uses the previously mentioned config, and builds it to contain `gcc-wrapper.sh`, `musl.spec`, and the sysroot tarball.
+1. A toolchain is defined in `//build/toolchain/musl-host-gcc:musl_host_toolchain` with a `//build/platforms/linkmode:musl-static` constraint, which is selected by the `//build/platforms:linux_amd64_static` platform.
Quirks
------