treewide: stop using LZ4 for initrd compression
There are two issues at play here: One is a bug in pierrec/lz4 when
using the legacy framing format [1]. This bit us when we hit a broken
size region with CL:2130, taking hours to debug.
The other is the fact that the Linux LZ4 frame format has significant
design issues [2], especially with concatenanted initrds.
The first issue could be fixed by switching to a different LZ4
implementation (we do even have the reference impl in the monorepo) but
there is no API to generate the legacy frame format and things like [3],
a patch carried by Ubuntu to fix more edge cases just do not inspire
confidence in such a solution.
Thus, this CL switches over to using zstd for compressing initrds.
Zstd is slower than LZ4 for decompressing, but it still decompresses at
multiple GB/s per core while having a much better compression ratio.
It also doesn't have any Linux-specific bits and Linux uses the
reference implementation for decoding, which should make it much more
robust. So overall I think this is a good tradeoff.
[1] https://github.com/pierrec/lz4/issues/156
[2] https://github.com/lz4/lz4/issues/956#issuecomment-736705712
[3] https://launchpadlibrarian.net/507407918/0001-unlz4-Handle-0-size-chunks-discard-trailing-padding-.patch
Change-Id: I69cf69f2f361de325f4b39f2d3644ee729643716
Reviewed-on: https://review.monogon.dev/c/monogon/+/2313
Tested-by: Jenkins CI
Reviewed-by: Serge Bazanski <serge@monogon.tech>
diff --git a/go.mod b/go.mod
index cc88cd0..9de22c8 100644
--- a/go.mod
+++ b/go.mod
@@ -108,7 +108,6 @@
github.com/mitchellh/go-wordwrap v1.0.0
github.com/opencontainers/runc v1.1.3
github.com/packethost/packngo v0.29.0
- github.com/pierrec/lz4/v4 v4.1.14
github.com/pkg/errors v0.9.1
github.com/pkg/sftp v1.10.1
github.com/prometheus/node_exporter v1.3.1
@@ -310,7 +309,7 @@
github.com/josharian/native v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/karrick/godirwalk v1.16.1 // indirect
- github.com/klauspost/compress v1.14.2 // indirect
+ github.com/klauspost/compress v1.17.2
github.com/kr/fs v0.1.0 // indirect
github.com/kr/pty v1.1.8 // indirect
github.com/libopenstorage/openstorage v1.0.0 // indirect