musl-host-gcc

musl-host-gcc is a Bazel C++ toolchain that uses the sandbox sysroot gcc in combination with a pre-built musl, musl headers, and Linux headers.

It is currently used to build the few C binaries we need on Metropolis nodes.

At some point, this toolchain should be improved to directly consume a static compiler toolchain and sysroot, so we can eventually get rid of the sandbox (like Aspect's gcc-toolchain is doing).

Usage

To use this toolchain explicitly while building a cc_binary, do:

bazel build --platforms=//build/platforms:linux_amd64_static //foo/bar

During an actual build however, the right toolchain should be selected using transitions or other configuration mechanisms.

Building Toolchain Sysroot Tarball

The toolchain's musl/linux components are currently built ahead of time and committed to this repository as //build/toolchain/musl-host-gcc/toolchain.tar.xz. This is the 'sysroot' tarball, that contains all headers and libraries required to build for Metropolis nodes.

To build this tarball, run the following commands:

bazel build //build/toolchain/musl-host-gcc/sysroot
cp -f bazel-bin/build/toolchain/musl-host-gcc/sysroot/sysroot.tar.xz build/toolchain/musl-host-gcc/sysroot.tar.xz

As a temporary hack the compiler-specific headers of our current development container have been manually merged in. This is expected to be replaced by a proper LLVM-based toolchain.

Internals

The toolchain is implemented in the following way:

1. //build/toolchain/musl-host-gcc/sysroot is used to build //build/toolchain/musl-host-gcc/sysroot.tar.xz which is a tarball that contains all include and binary library files for building against musl for Metropolis nodes (x86_64 / k8) - these are musl headers, musl libraries, and linux headers. This tarball is committed to source control.
2. When building a target that uses the toolchain, the sysroot.tar.xz tarball is extracted into an external repository @musl_sysroot, via sysroot.bzl and sysroot_repository.bzl.
3. A toolchain config is built using //build/toolchain:cc_toolchain_config.bzl, which points at gcc-wrapper.sh as its gcc entrypoint. gcc-wrapper.sh expects to be able to call the host gcc with musl.spec.
4. A toolchain is defined in //build/toolchain/musl-host-gcc:musl_host_toolchain with a //build/platforms/linkmode:musl-static constraint, which is selected by the //build/platforms:linux_amd64_static platform.

Quirks

As mentioned above, the musl sysroot is kept in a tarball in this repository. This is obviously suboptimal, but on the other hand gives us an effectively pre-built part of a toolchain. In the future, once we have a hermetic toolchain, a similar tarball might actually contain a fully hermetic toolchain pre-built for k8.