Tarballs we deliver should be compressed better

[nagisa]

Today’s rust-nightly-x86_64-unknown-linux-gnu.tar.gz is 125MiB in size. I did a make dist-tar-bins which output the same tarball, but only 88MiB in size. This is 70% of whatever we publish to s3.

I took liberty to also test:

• xz (the default level, -6) → 69MiB (55% original);
• xz -9 → 59MiB (47% original, but has high memory requirements to decompress)
• bz2 → 82MiB (65% original);
• lzma → 69MiB, but took longer than xz.

I strongly propose to either migrate to a more modern compression algorithm (xz) or at least investigating why gzip does such a bad job on the build bots.

cc @brson

[nagisa]

Apparently our s3 distribution includes docs, which my make dist-tar-bins didn’t include for some reason. That’s the cause of discrepancy between gzip sizes on my system and what’s shipped through s3.

I downloaded the tarball we ship and recompressed it for a fairer comparison:

• xz → 75MiB;
• xz -9 → 73MiB (honestly, surprised about how small the improvement is over xz -6);
• bz2 → 95MiB;

[nagisa]

The tarball (2015-03-03) has since increased to 138MiB in size. Recompressing that to xz results in 82MiB .tar.xz, for a 40% gain.

[alexcrichton]

Do we know of statistics of how many systems can decompress a xz archive? I'm mostly just curious as I suspect that we would continue to produce both and then systems like multirust could figure out which to download based on the host system.

[nagisa]

As I and others discussed on IRC:

• Macs don’t appear to have this by default, unsure, recipe on Homebrew;
• Windows have neither gzip, nor xz, nor tar by default, so it doesn’t matter either way;

On linux systems some very common packages (with defaultish configure options) depend on liblzma5/xz (gdb and systemd come to mind as examples) so it is very likely it will be available on a standard linux system.

[comex]

OS X does support xz by default in its 'tar' command (which is bsdtar - not sure exactly when support was introduced, but I think it was in 10.9) and Archive Utility (apparently newly in 10.10). This works via a library rather than the xz command line utility, which is not provided.

[nagisa]

Are you sure it is not just shelling out to the xz command? Does 'tar xJf
file' work without xz executable in the path?

EDIT: never mind, apparently bstdar just links and uses liblzma5 directly. This is very nice, I should consider dropping gnu tar and use bsdtar myself.

Either way, this means that using xz will benefit most of the users of both linux and os x.

2015/03/06 6:52 "comex" notifications@github.com:

OS X does support xz by default in its 'tar' command (which is bsdtar -
not sure exactly when support was introduced, but I think it was in 10.9)
and Archive Utility (apparently newly in 10.10).

—
Reply to this email directly or view it on GitHub
#21724 (comment).

[abonander]

The 7-Zip Utility for Windows can decompress xz archives, according to Wikipedia. That's the only archiver I use on Windows. It's free and open source, but it is third-party.

However, isn't the installer the preferred way of getting Rust on Windows? That's what I use. I don't know how the installer does decompression but xz support will probably have to be implemented for it.

[ssokolow]

In addition to 7-Zip (which invented the underlying LZMA2 compression format) XZ is also supported by all the other major tools not already mentioned:

• WinRAR (If you're running Windows, technical enough to be interested in Rust, and not using 7-zip, it's probably because you prefer WinRAR)
• The Unarchiver (Serves a similar role to 7-zip as the "extract anything" utility of choice for OSX power users)
• PeaZip (As far as I can tell, what people generally fall back to when they want an open-source, GUI-based "do 99% of what I need" archive tool for Windows but don't like 7-Zip's UI. Of course, given that most people who don't like 7-zip just put up with WinRAR's shareware nags, it's not got much market share even as the next in line.)

Edit: I need to stop forgetting to double-check my memory of what download pages are offering before posting. I've trimmed out some irrelevant bits.

[pnkfelix]

Strong +1 to @alexcrichton 's suggestion that we simply provide both. It costs us relatively nothing to construct both artifacts; is there any serious cost to provide them both (e.g. are we worried about connection charges or storage space on our servers?)

[nagisa]

An update since the metadata reform. Now gzipped tarball is 107MB. xzipped is 78MB. Still an easy 30MB win.

EDIT: docless xz: 75MB, docless gzip: 100MB.

[steveklabnik]

Is this still an issue today?