treewide: stop using LZ4 for initrd compression
There are two issues at play here: One is a bug in pierrec/lz4 when
using the legacy framing format [1]. This bit us when we hit a broken
size region with CL:2130, taking hours to debug.
The other is the fact that the Linux LZ4 frame format has significant
design issues [2], especially with concatenanted initrds.
The first issue could be fixed by switching to a different LZ4
implementation (we do even have the reference impl in the monorepo) but
there is no API to generate the legacy frame format and things like [3],
a patch carried by Ubuntu to fix more edge cases just do not inspire
confidence in such a solution.
Thus, this CL switches over to using zstd for compressing initrds.
Zstd is slower than LZ4 for decompressing, but it still decompresses at
multiple GB/s per core while having a much better compression ratio.
It also doesn't have any Linux-specific bits and Linux uses the
reference implementation for decoding, which should make it much more
robust. So overall I think this is a good tradeoff.
[1] https://github.com/pierrec/lz4/issues/156
[2] https://github.com/lz4/lz4/issues/956#issuecomment-736705712
[3] https://launchpadlibrarian.net/507407918/0001-unlz4-Handle-0-size-chunks-discard-trailing-padding-.patch
Change-Id: I69cf69f2f361de325f4b39f2d3644ee729643716
Reviewed-on: https://review.monogon.dev/c/monogon/+/2313
Tested-by: Jenkins CI
Reviewed-by: Serge Bazanski <serge@monogon.tech>
diff --git a/cloud/takeover/BUILD.bazel b/cloud/takeover/BUILD.bazel
index 21b891a..103ea2d 100644
--- a/cloud/takeover/BUILD.bazel
+++ b/cloud/takeover/BUILD.bazel
@@ -19,7 +19,7 @@
"//net/dump",
"//net/proto",
"@com_github_cavaliergopher_cpio//:cpio",
- "@com_github_pierrec_lz4_v4//:lz4",
+ "@com_github_klauspost_compress//zstd",
"@org_golang_google_protobuf//proto",
"@org_golang_x_sys//unix",
],
diff --git a/cloud/takeover/takeover.go b/cloud/takeover/takeover.go
index 9de39f5..e265b30 100644
--- a/cloud/takeover/takeover.go
+++ b/cloud/takeover/takeover.go
@@ -27,7 +27,7 @@
"time"
"github.com/cavaliergopher/cpio"
- "github.com/pierrec/lz4/v4"
+ "github.com/klauspost/compress/zstd"
"golang.org/x/sys/unix"
"google.golang.org/protobuf/proto"
@@ -44,7 +44,7 @@
//go:embed ucode.cpio
var ucode []byte
-//go:embed cloud/agent/initramfs.cpio.lz4
+//go:embed cloud/agent/initramfs.cpio.zst
var initramfs []byte
// newMemfile creates a new file which is not located on a specific filesystem,
@@ -124,9 +124,11 @@
}
// Append AgentInit spec to initramfs
- compressedOut := lz4.NewWriter(initramfsFile)
- compressedOut.Apply(lz4.LegacyOption(true))
- cpioW := cpio.NewWriter(compressedOut)
+ compressedW, err := zstd.NewWriter(initramfsFile, zstd.WithEncoderLevel(1))
+ if err != nil {
+ return nil, fmt.Errorf("while creating zstd writer: %w", err)
+ }
+ cpioW := cpio.NewWriter(compressedW)
cpioW.WriteHeader(&cpio.Header{
Name: "/init.pb",
Size: int64(len(agentInitRaw)),
@@ -134,7 +136,7 @@
})
cpioW.Write(agentInitRaw)
cpioW.Close()
- compressedOut.Close()
+ compressedW.Close()
agentParams := bootparam.Params{
bootparam.Param{Param: "quiet"},