*: fully hermetic builds and nix shell support
This change is a slightly more polished version of Serge's experiment:
- https://review.monogon.dev/c/monogon/+/1148
- https://bin.monogon.dev/pasta/sloth-parrot-ant
- https://bin.monogon.dev/pasta/eel-seal-wolf
There are two execution environments we have to support:
- Most builds run inside a sandbox, which is a Fedora
environment and does not require any host dependencies at all.
- Bazel itself and the tooling we require to bootstrap
the sandbox (mainly, Go and Proto toolchains). This has to
work directly on the host.
We first make the sandbox fully hermetic by setting
--experimental_use_hermetic_linux_sandbox, which set up an empty /
instead of mounting over individual directories, removing any remaining
host paths from the sandbox (except /proc and /dev/shm, which are
required by some toolchains). We also force static values for the shell,
$TMPDIR and $PATH, which would otherwise leak into the sandbox.
For the host, we use buildFHSUserEnv to build an environment which
supports our static toolchains, and well as a clean Bazel build
without all the nixpkgs patches which would otherwise break our custom
toolchains and sandbox implementation.
This allows us to use the exact same toolchains on NixOS and other
distros for perfect reproducibility.
Fixes https://github.com/monogon-dev/monogon/issues/174.
Fixes https://github.com/monogon-dev/monogon/issues/175.
Co-authored-by: Serge Bazanski <serge@monogon.tech>
Change-Id: I665471a45b315ce7e93ef16d9d056d7622886959
Reviewed-on: https://review.monogon.dev/c/monogon/+/1929
Tested-by: Jenkins CI
Reviewed-by: Serge Bazanski <serge@monogon.tech>
diff --git a/tools/bazel b/tools/bazel
index 849bb9b..67de8e4 100755
--- a/tools/bazel
+++ b/tools/bazel
@@ -22,7 +22,8 @@
fi
# Recommend using Bazelisk instead of Bazel's "bazel.sh" wrapper.
- if [[ -z "${BAZELISK_SKIP_WRAPPER:-}" ]]; then
+ # Skip if we're inside the Nix shell (which uses a customized Bazel build).
+ if [[ -z "${BAZELISK_SKIP_WRAPPER:-}" && -z "${MONOGON_NIXOS:-}" ]]; then
echo "############################################################" >&2
echo "# Please use Bazelisk to build the Monorepo. Using Bazel #" >&2
echo "# directly may work, but is not recommended or supported. #" >&2
@@ -132,6 +133,7 @@
# Autogenerated by tools/bazel. Manual changes can result in stale caches.
# Modify the generator instead.
+# Mount directories from the generated Fedora sandbox root.
build --sandbox_add_mount_pair=${ROOT}/.bazeldnf/sandbox/default/root/etc:/etc
build --sandbox_add_mount_pair=${ROOT}/.bazeldnf/sandbox/default/root/usr:/usr
build --sandbox_add_mount_pair=${ROOT}/.bazeldnf/sandbox/default/root/var:/var
@@ -141,6 +143,19 @@
build --sandbox_add_mount_pair=${ROOT}/.bazeldnf/sandbox/default/root/lib:/lib
build --sandbox_add_mount_pair=${ROOT}/.bazeldnf/sandbox/default/root/bin:/bin
build --sandbox_add_mount_pair=${ROOT}/.bazeldnf/sandbox/default/root/sbin:/sbin
+
+# Needed for the Go SDK shipped by rules_go to resolve its own GOROOT via /proc/self/exe.
+build --sandbox_add_mount_pair=/proc
+
+# Needed for python's multiprocessing lock implementation
+# (_multiprocessing.SemLock for eg. mp.Queue), as used in EDK2's build system.
+build --sandbox_add_mount_pair=/dev/shm
+
+# Needed for qemu for tests.
+build --sandbox_add_mount_pair=/dev/kvm
+
+# Put a tmpfs on /tmp for better performance.
+build --sandbox_tmpfs_path=/tmp
EOF
echo "Done regenerating sysroot." >&2
@@ -168,4 +183,8 @@
# This is strictly necessary to guarantee correctness.
export MONOGON_SANDBOX_DIGEST="$(cat "${SANDBOX}/checksum")"
+# Ignore the host TMPDIR - it might be something funny like /run/user/1000,
+# we want it to be /tmp inside the sandbox.
+export TMPDIR=/tmp
+
exec -a "$0" "${BAZEL_REAL}" "$@"