Improve Bazel Fedora build container handling and cache repository downloads
Adds lifecycle management scripts for the dev container and a "bazel" wrapper script, which sets container-only startup options.
Replaces /dev/null bind mounts by SELinux contexts for container breakup prevention, since newer podman versions managed to somehow break the ordering of mounts and mounting on top of a volume gives ENOENT. This requires a placeholder .arcconfig.
On Fedora, SELinux prevents the container from accessing /dev/kvm, which requires a custom policy (see rWa716c988d69e).
Design considerations:
- The build cache is on a tmpfs. This avoids fuse-overlayfs overhead. If the container is recreated, we want to drop the build cache - Bazel does not track ambient dependencies, so we do not know if we need to rebuild anything (like after upgrading a compiler).
- The repository cache contains just workspace dependencies and is mounted as a volume.
The repository caches does not work terribly well yet, we probably need to mount parts ~/.cache/bazel as well. podman always mounts volumes as noexec, so this is not as straight-forward as it looks.
Test Plan:
Ran the commands from the README as my unprivileged workstation user.
Smalltown was built and launched successfully.
X-Origin-Diff: phab/D198
GitOrigin-RevId: aff720d2862cdf5d1df67813d842d221d69a84c0
diff --git a/README.md b/README.md
index 4fa5e8f..303b602 100644
--- a/README.md
+++ b/README.md
@@ -3,28 +3,18 @@
## Run build
The build uses a Fedora 30 base image with a set of dependencies.
+Guide has been tested on a Fedora 30 host, with latest rW deployed.
-Start container shell:
+Build the base image:
```
-modprobe kvm
-
podman build -t smalltown-builder .
-
-podman run -it --rm \
- -v $(pwd):/work \
- -v /dev/null:/work/.git \
- -v /dev/null:/work/.idea \
- -v /dev/null:/work/.arcconfig \
- --device /dev/kvm \
- --net=host \
- smalltown-builder bash
```
Launch the VM:
- bazelisk run scripts:launch
+```
+scripts/bin/bazel run scripts:launch
+```
Exit qemu using the monitor console: `Ctrl-A c quit`.
-
-If your host is low on entropy, consider running rngd from rng-tools for development.
diff --git a/scripts/bin/bazel b/scripts/bin/bazel
new file mode 100755
index 0000000..799edd8
--- /dev/null
+++ b/scripts/bin/bazel
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+scripts/run_in_container.sh bazelisk $@ \
+ --repository_cache=/root/repo-cache
diff --git a/scripts/create_container.sh b/scripts/create_container.sh
new file mode 100755
index 0000000..6d284a1
--- /dev/null
+++ b/scripts/create_container.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+set -euo pipefail
+
+# Our local user needs write access to /dev/kvm (best accomplished by
+# adding your user to the kvm group).
+if ! touch /dev/kvm; then
+ echo "Cannot write to /dev/kvm - please verify permissions."
+ exit 1
+fi
+
+# The KVM module needs to be loaded, since our container is unprivileged
+# and won't be able to do it itself.
+if ! [[ -d /sys/module/kvm ]]; then
+ echo "kvm module not loaded - please modprobe kvm"
+ exit 1
+fi
+
+# Rebuild base image
+podman build -t smalltown-builder .
+
+# Set up SELinux contexts to prevent the container from writing to
+# files that would allow for easy breakouts via tools ran on the host.
+chcon -R system_u:object_r:container_file_t:s0 .
+chcon -R unconfined_u:object_r:user_home_t:s0 \
+ .arcconfig .idea .git
+
+# Create cache volume if it does not yet exist
+! podman volume create repo-cache
+
+podman run -it -d \
+ -v $(pwd):/work \
+ -v repo-cache:/root/repo-cache \
+ --tmpfs=/root/.cache/bazel:exec \
+ --device /dev/kvm \
+ --net=host \
+ --name=smalltown-dev \
+ smalltown-builder
diff --git a/scripts/destroy_container.sh b/scripts/destroy_container.sh
new file mode 100755
index 0000000..13fefe3
--- /dev/null
+++ b/scripts/destroy_container.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+podman stop smalltown-dev
+podman rm smalltown-dev --force
diff --git a/scripts/run_in_container.sh b/scripts/run_in_container.sh
new file mode 100755
index 0000000..8b60bd6
--- /dev/null
+++ b/scripts/run_in_container.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+set -euo pipefail
+
+podman exec -it smalltown-dev $@
\ No newline at end of file