Enable `rust-lld` on nightly `x86_64-unknown-linux-gnu` #124129

lqd · 2024-04-18T18:34:39Z

We believe we have done virtually all the internal work and tests we could to prepare for using lld as the default linker (at least on Linux). We're IMHO at a point where we'd need to expand testing and coverage in order to make progress on this effort.

Therefore, for further testing and gathering real-world feedback, unexpected issues and use-cases, this PR enables rust-lld as the default linker:

on nightly only (and dev channel)
on x86_64-unknown-linux-gnu only
when not using an external LLVM (except download-ci-llvm), so that distros are not impacted

as described in more detail in this zulip thread.

In case any issues happen to users, as e.g. lld is not bug-for-bug compatible with GNU ld, it's easy to disable with -Zlinker-features=-lld to revert to using the system's default linker.

I don't know who should review this kind of things, as it's somewhat of a crosscutting effort. Compiler contributor, compiler performance WG and infra member sounds perfect, so r? @Mark-Simulacrum.

The last crater run encountered a low number (44) of mainly avoidable issues, like small incompatibilities, user errors, and a difference between the two linkers about which default to use with --gc-sections. Here's the triage report, categorizing the issues, with some analyses and workarounds. I'd appreciate another set of eyes looking at these results.

The changes in this PR have been test-driven for CI changes, try builds with tests enabled, rustc-perf with bootstrapping, in PR #113382.

For infra, about the CI change: this PR forces rust.lld to false on vanilla LLVM builders, just to make sure we have coverage without rust-lld. Though to be clear, just using an external LLVM is already enough to keep rust.lld to false, in turn reverting everything to using the system's default linker.

cc @rust-lang/bootstrap for the bootstrap and config change
cc @petrochenkov for the small compiler change
cc @rust-lang/wg-compiler-performance

The blog post announcing the change, that we expect to merge around the same time as we merge this PR, is open on the blog repo.

Bootstrap change history: this PR changes the default of a config option on x86_64-unknown-linux-gnu. It's, however, not expected to cause issues, or require any changes to existing configurations. It's a big enough change that people should at least know about it, in case it causes unexpected problems. If that happens, set rust.lld = false in your config.toml (and open an issue).

lqd · 2024-04-18T18:53:43Z

@bors try @rust-timer queue

Enable `rust-lld` on nightly `x86_64-unknown-linux-gnu` r? ghost Draft until final preparations are done (making a bootstrap change entry requires a PR number)

bors · 2024-04-18T18:54:54Z

⌛ Trying commit f63b7c5 with merge 5a8c051...

rustbot · 2024-04-18T19:11:29Z

Some changes occurred in run-make tests.

cc @jieyouxu

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies config.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

These commits modify compiler targets.
(See the Target Tier Policy.)

bors · 2024-04-18T20:27:11Z

☀️ Try build successful - checks-actions
Build commit: 5a8c051 (5a8c051e2c9b7cb9c5bbbf9bc9cf44f12477e577)

rust-timer · 2024-04-19T06:04:58Z

Finished benchmarking commit (5a8c051): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-30.9%	[-72.0%, -0.5%]	31
Improvements ✅ (secondary)	-26.4%	[-71.3%, -1.5%]	73
All ❌✅ (primary)	-30.9%	[-72.0%, -0.5%]	31

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	17.7%	[12.3%, 23.1%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	17.7%	[12.3%, 23.1%]	2

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	7.7%	[6.1%, 9.2%]	4
Improvements ✅ (primary)	-30.7%	[-63.0%, -1.0%]	27
Improvements ✅ (secondary)	-24.0%	[-61.3%, -2.0%]	72
All ❌✅ (primary)	-30.7%	[-63.0%, -1.0%]	27

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.1%	[0.6%, 4.3%]	28
Regressions ❌ (secondary)	2.0%	[0.4%, 5.5%]	74
Improvements ✅ (primary)	-0.3%	[-0.3%, -0.2%]	4
Improvements ✅ (secondary)	-0.2%	[-0.3%, -0.1%]	2
All ❌✅ (primary)	1.8%	[-0.3%, 4.3%]	32

Bootstrap: 676.731s -> 669.514s (-1.07%)
Artifact size: 316.11 MiB -> 315.37 MiB (-0.23%)

albertlarsan68 · 2024-04-19T07:11:25Z

For the bootstrap change:
I would prefer another config, instead of overloading the already existing one.
Something like rust.allow_lld_default, which can only be true when rust.lld is true, seems better IMO.
Also, how does this interact with rust.use_lld?

Kobzol · 2024-04-19T07:16:53Z

The bootstrap change is "temporary", in the sense that once this change lands on beta, it will be enabled by default and bootstrap won't need to configure it, if I understand it correctly. So the bootstrap change doesn't really modify the meaning of rust.lld, it just enables the new default if LLD is built.

The annoying thing here is that we have many axes on what to configure - should LLD be built? (this should arguably be in the llvm section, not rust), should LLD be added to sysroot? should LLD be used when linking the toolchain? should it be used when linking Rust programs? should the self-contained or external LLD be used? And some of these are only relevant for rustc developers. We already have two LLD options that are quite inconsistent in their behavior, I'm not sure if adding a third option makes that situation better 😅 If we go this route, I would rather completely revamp the way we configure LLD (and linker usage in general), so that it makes more sense in bootstrap.

src/bootstrap/src/core/build_steps/compile.rs

src/ci/docker/host-x86_64/x86_64-gnu-llvm-18/Dockerfile

Mark-Simulacrum · 2024-04-20T14:02:40Z

The annoying thing here is that we have many axes on what to configure - should LLD be built? (this should arguably be in the llvm section, not rust), should LLD be added to sysroot? should LLD be used when linking the toolchain? should it be used when linking Rust programs? should the self-contained or external LLD be used? And some of these are only relevant for rustc developers. We already have two LLD options that are quite inconsistent in their behavior, I'm not sure if adding a third option makes that situation better 😅 If we go this route, I would rather completely revamp the way we configure LLD (and linker usage in general), so that it makes more sense in bootstrap.

I think this makes sense to me as the right direction to go in, rather than trying to add another option in this PR.

Overall this PR looks good to me at a high level, though I can't review the correctness of the compiler change (@petrochenkov should probably sign off on that), it seems OK to me. I think we should proceed with it. One thing that surprises me though is that this had a positive effect on the binary sizes for what we ship. Were we not already using LLD during that build? Perhaps we were picking up an old LLD from somewhere?

I agree that we should have a blog post notifying users of this change (similar to the parallel frontend one). Once that's ready and a few nits here are fixed, I'm happy to r+

lqd · 2024-04-20T17:24:12Z

Were we not already using LLD during that build? Perhaps we were picking up an old LLD from somewhere?

The artifact size reduction is also somewhat surprising to me. I believe that we are indeed already using LLD for that build, via rust.use-lld and that this should find and use the system's LLD 18.1.0 rather than an old one.

lqd · 2024-04-22T10:00:19Z

I wonder if these size reductions may be some kind of noise, they were not present in the previous runs of #113382 (other than the regular small fluctuations; the only difference with this PR is more builders are enabled there) including this one from yesterday.

The artifacts in the dist build are not linked with the self-contained linker (rust-lld/lld 18.1.4), but via --use-lld (CI's lld 18.1.0), and this can be verified in the .comment section. That is, this PR only makes x86_64-unknown-linux-gnu use rust-lld by default. Dist on the contrary specifies clang as the linker. When that happens, rustc needs to infer a linker flavor: clang is inferred to a C/C++ compiler not using lld, and the self-contained linker is not enabled. That's what already happens today.

Let's maybe see if another run has different results.

edit: done below, no significant change

rust-timer · 2024-05-17T05:31:26Z

Finished benchmarking commit (8af67ba): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-30.8%	[-71.8%, -0.5%]	31
Improvements ✅ (secondary)	-25.6%	[-70.9%, -0.5%]	75
All ❌✅ (primary)	-30.8%	[-71.8%, -0.5%]	31

Max RSS (memory usage)

Results (primary 16.6%, secondary 3.8%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	16.6%	[9.9%, 23.4%]	2
Regressions ❌ (secondary)	3.8%	[2.4%, 5.2%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	16.6%	[9.9%, 23.4%]	2

Cycles

Results (primary -30.6%, secondary -24.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-30.6%	[-62.4%, -1.0%]	27
Improvements ✅ (secondary)	-24.2%	[-61.6%, -2.3%]	71
All ❌✅ (primary)	-30.6%	[-62.4%, -1.0%]	27

Binary size

Results (primary 2.0%, secondary 1.8%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.2%	[0.6%, 4.2%]	28
Regressions ❌ (secondary)	1.9%	[0.5%, 5.5%]	74
Improvements ✅ (primary)	-0.3%	[-0.3%, -0.3%]	3
Improvements ✅ (secondary)	-0.6%	[-1.0%, -0.1%]	2
All ❌✅ (primary)	2.0%	[-0.3%, 4.2%]	31

Bootstrap: 678.599s -> 669.726s (-1.31%)
Artifact size: 316.13 MiB -> 316.16 MiB (0.01%)

Link is currently failing with rust-lld (rust-lang/rust#124129).

With rust-lang/rust#124129, x64 linux switches to use LLD by default, but this causes dependency issues on NixOS. Disable LLD for now. This is tracked by rust-lang/rust#125321.

Refs: dtolnay/linkme#88, rust-lang/rust#124129

Here are all the changes. I went through them one-by-one and confirmed that they should not be affecting us. In paritcular, we explicitly set rust.lld = false (because we want to use the lld that ships with clang), so the change in default does not affect us. There have been changes to x.py since you last updated: [INFO] New option `build.lldb` that will override the default lldb binary path used in debuginfo tests - PR Link rust-lang/rust#124501 [INFO] The compiler profile now defaults to rust.debuginfo-level = "line-tables-only" - PR Link rust-lang/rust#123337 [WARNING] `rust.lld` has a new default value of `true` on `x86_64-unknown-linux-gnu`. Starting at stage1, `rust-lld` will thus be this target's default linker. No config changes should be necessary. - PR Link rust-lang/rust#124129 [WARNING] Removed `dist.missing-tools` configuration as it was deprecated long time ago. - PR Link rust-lang/rust#125535 [WARNING] `llvm.lld` is enabled by default for the dist profile. If set to false, `lld` will not be included in the dist build. - PR Link rust-lang/rust#126701 [WARNING] `debug-logging` option has been removed from the default `tools` profile. - PR Link rust-lang/rust#127913 [INFO] the `wasm-component-ld` tool is now built as part of `build.extended` and can be a member of `build.tools` - PR Link rust-lang/rust#127866 [INFO] Removed android-ndk r25b support in favor of android-ndk r26d. - PR Link rust-lang/rust#120593 [WARNING] For tarball sources, default value for `rust.channel` will be taken from `src/ci/channel` file. - PR Link rust-lang/rust#125181 [INFO] New option `llvm.libzstd` to control whether llvm is built with zstd support. - PR Link rust-lang/rust#125642 [WARNING] ./x test --rustc-args was renamed to --compiletest-rustc-args as it only applies there. ./x miri --rustc-args was also removed. - PR Link rust-lang/rust#128841 [INFO] The `build.profiler` option now tries to use source code from `download-ci-llvm` if possible, instead of checking out the `src/llvm-project` submodule. - PR Link rust-lang/rust#129295 Original-Reviewed-on: https://fuchsia-review.googlesource.com/c/infra/recipes/+/1120078 Original-Revision: 27df37a30e50b14b9ffefc872b6997790f03d4ea GitOrigin-RevId: 341e222f002e36886b9960645b21faeaed633f90 Change-Id: Id1eb54a677a6f538bf7666d65b85d5fdba17ea42

Use lld by default on `x86_64-unknown-linux-gnu` stable This PR and stabilization report is joint work with `@Kobzol.` #### Use LLD on `x86_64-unknown-linux-gnu` by default, and stabilize `-Clinker-features=-lld` and `-Clink-self-contained=-linker` This PR proposes making LLD the default linker on the `x86_64-unknown-linux-gnu` target for the artifacts we distribute, and also stabilizing the `-Clinker-features=-lld` and `-Clink-self-contained=-linker` codegen options to make it possible to opt out. LLD has been used as the default linker on nightly and CI on this target since May 2024 ([PR](#124129), [blog post](https://blog.rust-lang.org/2024/05/17/enabling-rust-lld-on-linux.html)), and it seems like it is working fine, so we would like to propose stabilizing it. The main motivation for using LLD instead of the default BFD linker is improving [compilation times](https://perf.rust-lang.org/compare.html?start=b3e117044c7f707293edc040edb93e7ec5f7040a&end=baed03c51a68376c1789cc373581eea0daf89967&stat=instructions%3Au&tab=compile). For example, in the linked benchmark, it makes incremental recompilation of `ripgrep` in `debug` more than twice faster. Another benefit is that Rust compilation becomes more consistent and self-contained, because we will use a known version of the LLD linker, rather than "whatever GNU ld version is on the user's system". Due to the performance benefit being so huge, many people already opt into using LLD (or other fast linkers, such as mold) using various approaches ([1](https://github.com/search?type=code&q=%2Flinker-flavor%5B%3D+%5Dgnu-lld-cc%2F), [2](https://github.com/search?type=code&q=%2Flinker-features%5B%3D+%5D%5C%2Blld%2F), [3](https://github.com/search?type=code&q=language%3Atoml+%22-fuse-ld%3Dlld%22), [4](https://github.com/search?type=code&q=language%3Arust+%22-fuse-ld%3Dlld%22)). By making LLD the default linker on the `x86_64-unknown-linux-gnu` target, we will be able to speed up Rust compilation out of the box, without users having to opt in or know about it. > You can find an extended version of this stabilization report which includes analysis of crater results and more data [here](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view). ## What is being stabilized - `rust-lld` being used as the default linker on the `x86_64-unknown-linux-gnu` target. - Note that `rust-lld` is being enabled by default in the compiler artifacts distributed by our CI/rustup. It is still possible to use the system linker by default using `rust.lld = false` in `bootstrap.toml`, which can be useful e.g. for some Linux distros that might not want to use the LLD we distribute. - This is done by activating the LLD linker feature and using the self-contained linker on that target. Both of which are also usable on the CLI, if some opt outs are necessary, as described below. - `-Clinker-features=-lld` on the `x86_64-unknown-linux-gnu` target. This codegen option tells rustc to disable using the LLD linker. - Note that other options for this codegen flag (`cc`) remain unstable. - Note that only the opt-out is being stabilized, and only for `x86_64-unknown-linux-gnu`: opting in, or using the flag on other targets would still need to pass `-Zunstable-options`. - This flag is being stabilized so that users can opt out of LLD on stable, which would it turn also opt out of using the self-contained linker (since it's an LLD). - `-Clink-self-contained=-linker`. This codegen option tells rustc to use the self-contained linker. It's not particularly useful to turn it on by itself, but when enabled and combined with `-Clinker-features=+lld`, rustc will use the `rust-lld` linker wrapper shipped with the compiler toolchain, instead of some `lld` binary that the linker driver will find in the `PATH`. - Note that other options for this codegen flag (other than the previously stable `y/yes/n/no`). - Note that only the opt-out is being stabilized, and only for `x86_64-unknown-linux-gnu`: opting in, or using this flag on other targets would still need to pass `-Zunstable-options`. - This flag is being stabilized so that users can opt out of using self-contained linking on stable. Doing this would then fall back to using the system `lld`. To opt out of using LLD, `RUSTFLAGS="-Clinker-features=-lld"` would be used. To opt out of using `rust-lld`, falling back to the LLD installed on the system, `RUSTFLAGS="-Clink-self-contained=-linker"` would be used. ## Tests When enabling `rust-lld` on nightly, we also switched x64 linux to use it at stage >= 1, meaning that all tests have been running with lld since May 2024, on CI as well as contributors' machines. (Post opt-dist tests also had been using it when running their test subset earlier than that). There are also a few tests dedicated to the CLI behavior, or ensuring the default linker is indeed the one we expect: - [link-self-contained-consistency](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/link-self-contained-consistency.rs): Checks that `-Clink-self-contained` options are not inconsistent (i.e. that passing both `+linker` and `-linker` is an error). - [link-self-contained-unstable](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/link-self-contained-unstable.rs): Checks that only the `-linker` and `y/yes/n/no` options for `-Clink-self-contained` are stable. - [linker-features-unstable-cc](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/linker-features-unstable-cc.rs): Checks that only the non-lld options of `-Clinker-features` are unstable. - [linker-features-lld-disallowed](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/linker-features-lld-disallowed.rs): Checks that `-Clinker-features=-lld` is only stable on `x86_64-unknown-linux-gnu`. - [link-self-contained-linker-disallowed](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/link-self-contained-linker-disallowed.rs): Checks that `-Clink-self-contained=-linker` is only stable on `x86_64-unknown-linux-gnu`. - [no-gc-encapsulation-symbols](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/no-gc-encapsulation-symbols.rs): Checks that that linker encapsulation symbols are not garbage collected by LLD, so that crates like [linkme](https://github.com/dtolnay/linkme) still work. - [rust-lld](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/run-make/rust-lld): Checks that LLD is actually used when enabled with `-Clinker-features=+lld` and `-Clink-self-contained=+linker`. - [rust-lld-x86_64-unknown-linux-gnu](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/run-make/rust-lld-x86_64-unknown-linux-gnu): Checks that LLD is used by default on `x86_64-unknown-linux-gnu` when the bootstrap `rust.lld` config option is `true`. - [rust-lld-x86_64-unknown-linux-gnu-dist](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/run-make/rust-lld-x86_64-unknown-linux-gnu-dist): Dist test that checks that our distributed `x86_64-unknown-linux-gnu` archives actually use LLD by default. ## Ecosystem impact As already stated, LLD has been used as the default linker on x64 Linux on nightly for almost a year, and we haven't seen any blockers to stabilization in that time. There were a handful of issues reported, these are discussed later below. Furthermore, two crater runs ([November 2023](https://crater-reports.s3.amazonaws.com/pr-117684-2/index.html), [February 2025](https://crater-reports.s3.amazonaws.com/pr-137044-3/index.html)), were performed to test the impact of using LLD as the default linker. A triage of the earlier crater run was previously done [here](https://hackmd.io/OAJxlxc6Te6YUot9ftYSKQ), but all the important findings from both crater runs are reported below. Below is a list of compatibility differences between BFD and LLD that we have encountered. There is a more thorough list of differences in [this post](https://maskray.me/blog/2020-12-19-lld-and-gnu-linker-incompatibilities) from the current LLD maintainer. From that post, "99.9% pieces of software work with ld.lld without a change". --- ### `.ctors/.dtors` sections [#128286](#128286) reported an issue where LLD was unable to link certain CUDA library was using these sections that were using the `.ctors/.dtors` ELF sections. These were deprecated a long time ago (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770), replaced with a more modern `.init_array/.fini_array` sections. LLD doesn't (and won't) support these sections ([1](llvm/llvm-project#68071), [2](llvm/llvm-project#30572)), so if they appear in input object files, the linked artifact might produce incorrect behavior, because e.g. some global variables might not get initialized properly. However, the usage of `.ctors/.dtors` should be very rare in practice. We have performed a [crater run](#137044) to test this. It has identified only 8 crates where the `.ctors/.dtors` section is occurring in the final linked artifact. It was caused by a few crates using the `.ctors` link section manually, and by using a very (~6 year) old version of the [ctor](https://crates.io/crates/ctor) crate. [Crater run analysis](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view#ctorsdtors-sections) **Possible workaround** It is possible to [detect](e5e2316) if `.ctors/.dtors` section is present in the final linked artifact (LLD will keep it there, but it won't be populated), and warn users about it. This check is very cheap and doesn't even appear on [perf](#112049 (comment)). We have benchmarked the check on a 240 MiB Chrome binary, where it took 0.8ms with page cache flushed, and 0.06ms with page cache primed (which should be the common case, as the linked artifact is written to disk just before the check is performed). In theory, this could be also solved with a linker script that moves `.ctors` to `.init_array`. We think that these sections should be so rare that it is not worth it to implement any workarounds for now. --- ### Different garbage collection behavior [#130397](#130397) reported an issue where LLD prunes a local symbol, so it is missing in the linked artifact. However, BFD keeps the same symbol, so it is a regression. This is caused by a difference in linker garbage collection. Rust uses `--gc-sections` and puts each function into a separate linker section, which prunes unused code. There is some code (specifically the somewhat popular [linkme](https://github.com/dtolnay/linkme) crate) that (arguably ab-)uses so called linker encapsulation symbols to achieve distributed slices. BFD (2.37+) uses a conservative linking mode that works as intended with this behavior, but it might slightly increase binary size of the linked artifact. LLD does not use this workaround by default, which causes the sections to be eliminated, but it can be made to use the conservative mode using [`-z nostart-stop-gc`](https://lld.llvm.org/ELF/start-stop-gc.html#z-start-stop-gc). To avoid this issue, we told LLD to use the [conservative mode](#137685), which maintains backwards compatibility with BFD. We found that it has [no effect](#112049 (comment)) on compilation performance and binary size in our benchmark suite. With this change, `linkme` works. Since then, #140872 removed `linkme` distributed slice's dependence on conservative GC behavior, so this PR also removes that conservative mode: no transition period is necessary, as the PR immediately fixed the crate with no source changes. [Crater run analysis](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view#Different-garbage-collection-behavior) --- ### Various uncommon issues A small number of issues that only occurred in a handful of instances were found in crater, and it is unclear if LLD is at fault or if there is some other issue that was not detected with BFD. You can examine these [here](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view#Various-uncommon-issues). --- ### Missing jobserver support LLD doesn't support the jobserver protocol for limiting the number of threads used, it simply defaults to using all available cores, and is one of the reasons why it's faster than BFD. However, this should mostly be a non-issue, because most of the linking done during high parallelism sections of `cargo build` is linking of build scripts and proc macros, which are typically very fast to link (e.g. ~50ms), and a potential oversubscription of cores thus doesn't hurt that much. When the final artifact is linked (which typically takes the most time), there should be no other sources of parallelism conflicts from compiling other code, so LLD should be able to use all available threads. That being said, it is a difference of behavior, where previously a `-j` flag was generally not using more cpu than the specified limit. It can be impactful in some resource-constrained systems, but to be clear that is already the case today due to [cargo parallelism](rust-lang/cargo#9157). This could be one reason to opt out of using `rust-lld` on some systems. LLD has support for limiting the number of threads to use, so in theory rustc could try to get all the jobserver tokens available and use that as lld's thread limit. It'd still be suboptimal as new tokens would not be dynamically detected, and we could be using less threads than available. We did a benchmark on a real-world crate that shows that using multiple LLD threads for intermediate artifacts doesn't seem to have a performance effect. You can find it [here](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view#Missing-jobserver-support). --- #### Opting out of LLD in the ecosystem We have also examined repositories where people opted out of LLD on nightly, using [this GitHub query](https://github.com/search?q=%22linker-features%3D-lld%22&type=code). The summary can be found below: <details> <summary>Summary of LLD opt outs</summary> > This examination was performed on 2025-03-09. Here we briefly examine the most common reasons why people use `-Zlinker-features=-lld`, based on comments and git history. - Nix/NixOS ([1](https://github.com/rszyma/vscode-kanata/blob/59d703dff5a238b14ab3759cac27f73fb34bbcfe/flake.nix#L33), [2](https://github.com/sbernauer/breakwater/blob/3cc3449fc126c5c99d4a971733fd32be589884e0/.cargo/config.toml#L4), [3](https://github.com/tiiuae/ebpf-firewall/blame/32bdb17cedd1c9bea1ab3482623de458d95da7d0/.cargo/config.toml#L2), [4](https://github.com/jules-sommer/wavetheme-gen/blob/f5f657d014d4a30607625afb70f810c229c0294e/Cargo.toml#L4), [5](https://github.com/LayerTwo-Labs/zside-rust/blob/e4266f5c5571a1b180a9c70cf0939c7070e410c7/.cargo/config.toml#L10), [6](https://github.com/przyjacielpkp/zkml/blob/22a4aef24e9d2c77789229d7c634fc67e9eb1184/README.md?plain=1#L78), [7](https://github.com/LayerTwo-Labs/thunder-rust/blob/2222d53474c8d2d0428b4c56f8157095dced6d5a/.cargo/config.toml#L2), [8](https://github.com/enesoztrk/nixos-tc-aya-test/blob/b2ffa59d3eba8b60fd04b0a4c8bbe047400eb981/.cargo/config.toml#L4), [9](https://github.com/lowRISC/container-hotplug/blob/3ead4ef9c7f79c303392178c99677dbecff1aea6/.cargo/config.toml#L2), [10](https://github.com/Eliah-Lakhin/ad-astra/blob/ca6b8c8a5dba7bb5e894f3f1013f17876962a021/work/examples/lsp-client/src/extension.ts#L94)) - There was an [issue](NixOS/nixpkgs#312661) with LLD, which seems to have been fixed with NixOS/nixpkgs#314268. It's unclear whether that fixed all the Nix issues though. - Issues with linkme ([1](https://github.com/0xPolygonZero/zk_evm/blob/ef388619ffbd5305209519a3a5bc0396185d68ac/.cargo/config.toml#L4), [2](https://github.com/conjure-cp/conjure-oxide/blob/be0fc5827ff90e8486d416cc184b6ce24f73bf01/README.md?plain=1#L20), [3](https://github.com/clchiou/garage/blob/c5d8444d56bb6ee24ca95e5c6b9880ed996f4918/rust/.cargo/config.toml#L6), [4](https://github.com/PonasKovas/craftflow/blob/5b4cc1a5196e08a975368399fefda4b71f3a2f6f/.cargo/config.toml#L3), [5](https://github.com/kezhuw/zookeeper-client-rust/blob/4e27c7de2a7cc5e709af012b791c8fea9bb47f1f/.github/workflows/ci.yml#L82), [6](https://github.com/niklasdewally/conjure-oxide/blob/8fe60c12bca7011a2f9eded4b7c95ad0e77b6f44/.github/workflows/code-coverage.yml#L48), [7](https://github.com/kezhuw/spawns/blob/c8b468379805de9df3287c01b94b4ed3db6b61ed/.github/workflows/ci.yml#L74)) - These should be resolved with the conservative garbage collection ([#137685](#137685)). - Bazel ([1](https://github.com/google-parfait/confidential-federated-compute/blob/1823f69ed8f5f4f819f7bfa21da1ca629fdc826b/.bazelrc#L71)), WASM ([1](https://github.com/Eliah-Lakhin/ad-astra/blob/ca6b8c8a5dba7bb5e894f3f1013f17876962a021/work/examples/wasm-build.sh#L37), [2](https://github.com/yacineb/pgrx-wasi-test/blob/2bf99037ca1b650b2cbc35f1257a87fb6ead0920/build.sh#L21)), uncategorized ([2](https://github.com/nbdd0121/r2vm/blob/5118be6b9e757c6fef2f019385873f403c23c548/.cargo/config.toml#L3), [3](https://github.com/Wyvern/Img/blame/45020c7e1dc4926c8129647014c708db0c13f463/.cargo/config.toml#L209), [4](https://github.com/arnaudpoullet/leptos-i18n-compile-error/blob/042eb835f7ca0dc36be67cf7fe65b35b22b6059f/README.md?plain=1#L89), [5](https://github.com/JonLeeCon/numerical-rust-cpu/blob/fd0b3006768ed81c56147044dc05c92b11b7b6f0/exercises/.cargo/config.toml#L13), [6](https://github.com/PonasKovas/shallowclone/blob/be65f2ec923cac6ceedbc8db520c89969ebfce7c/.github/workflows/rust.yml#L20)) - Reason unclear. </details> ## History The idea to use a faster linker by default has been on the radar for quite some time ([#39915](#39915), [#71515](#71515)). There were [very early attempts](#29974) to use the gold linker by default, but these had to be [reverted](#30913) because of compatibility issues. Support for LLD was implemented back in [2017](#40018), but it has not been made default yet, except for some more niche targets, such as [WASM](#48125), [ARM Cortex](#53648) or [RISC-V](#53822). It took quite some time to figure out how should the interface for selecting the linker (and the way it is invoked) look like, as it differs a lot between different platforms, linkers and compiler drivers. During that time, LLD has matured and achieved [almost perfect compatibility](https://maskray.me/blog/2020-12-19-lld-and-gnu-linker-incompatibilities) with the default Linux linker (BFD). - [#56351](#56351) stabilized `-Clinker-flavor`, which is used to determine how to invoke the linker. It is especially useful on targets where selecting the linker directly with `-Clinker` is not possible or is impractical. - December 2018, author `@davidtwco,` reviewer `@nagisa` - [#76158](#76158) stabilized `-Clink-self-contained=[y|n]`, which allows overriding the compiler's heuristic for deciding whether it should use self-contained or external tools (linker, sanitizers, libc, etc.). It only allowed using the self-contained mode either for everything (`y`) or nothing (`n`), but did not allow granular choice. - September 2020, author `@mati864,` reviewer `@petrochenkov` - [#85961](#85961) implemented the `-Zgcc-ld` flag, which was a hacky way of opting into LLD usage. - June 2021, author `@sledgehammervampire,` reviewer `@petrochenkov` - [MCP 510](rust-lang/compiler-team#510) proposed stabilizing the behavior of `-Zgcc-ld` using more granular flags (`-Clink-self-contained=linker -Clinker-flavor=gcc-lld`). - Initially implemented in [#96827](#96827), but `@petrochenkov` [suggested](#96827 (comment)) a slightly different approach. - The PR was split into [#96884](#96884), where it was decided what will be the individual components of `-Clink-self-contained=linker`. - And [#96401](#96401), which implemented the `-Clinker-flavor` part. - The MCP was finally implemented in [#112910](#112910). - [#116514](#116514) then removed `-Zgcc-ld`, as it was replaced by `-Clinker-flavor=gnu-lld-cc` + `-Clink-self-contained=linker`. - April 2022 - October 2023, author `@lqd,` reviewer `@petrochenkov` - Various linker handling refactorings were performed in the meantime: [#97375](#97375), [#98212](#98212), [#100126](#100126), [#100552](#100552), [#102836](#102836), [#110807](#110807), [#101988](#101988), [#116515](#116515) - The implementation of linker flavors with LLD was causing a sort of a combinatorial explosion of various options. [#119906](#119906) suggested a different approach for linker flavors (described [here](#119906 (comment))), where the individual flavors could be enabled separately using `+/-` (e.g. `+lld`). - After some back and forth, this idea was moved to `-Clinker-features` (see [comment 1](#119906 (comment)) and [comment 2](#119906 (comment))), which was implemented in [#123656](#123656). - April 2024, author `@lqd,` reviewer `@petrochenkov` - [#124129](#124129) enabled LLD by default on nightly. - April 2024, author `@lqd,` reviewer `@petrochenkov` - [#137685](#137685), [#137926](#137926) enabled the conservative gargage collection mode (`-znostart-stop-gc`) to improve compatibility with BFD. - February 2025, author `@lqd,` reviewer `@petrochenkov` (implementation), author `@kobzol,` reviewer `@lqd` (test) - [#96025](#96025) (April 2022), [#117684](#117684) (November 2023), [#137044](#137044) (February 2025): crater runs. ## Unresolved questions/concerns - Is changing the linker considered a breaking change? In (hopefully very rare) cases, it might break some existing code. It should mostly only affect the final linked artifact, so it should be easy to opt out. - Similarly, is the single-threaded behavior of such tools encompassed in our stability guarantee: it can be observed via the `-j` job limit (though I believe we have/had some open issues on sometimes using more CPU resources than the job count limit implied). As mentioned above, LLD does not support the jobserver protocol. - A concern [was raised](#71515 (comment)) about increased memory usage of LLD. We should probably let users know about the possibly increased memory usage, and jobserver incompatibility: we did so when announcing this landing on nightly. - LLD seems to produce [slightly larger](https://perf.rust-lang.org/compare.html?start=b3e117044c7f707293edc040edb93e7ec5f7040a&end=baed03c51a68376c1789cc373581eea0daf89967&stat=size%3Alinked_artifact&tab=compile) binary artifacts. This can be partially clawed back using Identical Code Folding (`-Clink-args=-Wl,--icf=all`). - Should we detect the outdated `.ctors/.dtors` sections to provide a better error message, even if that should be rare in practice? --- ### Next steps After the FCP completes: - we should land this PR at the beginning of a beta cycle, to maximize time for testing - keep an eye on the beta crater run results for possible linker issues (or do a dedicated beta crater run with only this change) - release a blog post announcing the change, and asking for testing feedback of the appropriate beta - depending on feedback, or if a period of testing of 6 weeks is not long enough, we could keep this change on beta for another cycle --- Development, testing, try builds were done in #138645. r? `@petrochenkov` `@rustbot` label +needs-fcp +T-compiler try-job: aarch64-gnu try-job: i686-gnu-*

Use lld by default on `x86_64-unknown-linux-gnu` stable This PR and stabilization report is joint work with `@Kobzol.` #### Use LLD on `x86_64-unknown-linux-gnu` by default, and stabilize `-Clinker-features=-lld` and `-Clink-self-contained=-linker` This PR proposes making LLD the default linker on the `x86_64-unknown-linux-gnu` target for the artifacts we distribute, and also stabilizing the `-Clinker-features=-lld` and `-Clink-self-contained=-linker` codegen options to make it possible to opt out. LLD has been used as the default linker on nightly and CI on this target since May 2024 ([PR](#124129), [blog post](https://blog.rust-lang.org/2024/05/17/enabling-rust-lld-on-linux.html)), and it seems like it is working fine, so we would like to propose stabilizing it. The main motivation for using LLD instead of the default BFD linker is improving [compilation times](https://perf.rust-lang.org/compare.html?start=b3e117044c7f707293edc040edb93e7ec5f7040a&end=baed03c51a68376c1789cc373581eea0daf89967&stat=instructions%3Au&tab=compile). For example, in the linked benchmark, it makes incremental recompilation of `ripgrep` in `debug` more than twice faster. Another benefit is that Rust compilation becomes more consistent and self-contained, because we will use a known version of the LLD linker, rather than "whatever GNU ld version is on the user's system". Due to the performance benefit being so huge, many people already opt into using LLD (or other fast linkers, such as mold) using various approaches ([1](https://github.com/search?type=code&q=%2Flinker-flavor%5B%3D+%5Dgnu-lld-cc%2F), [2](https://github.com/search?type=code&q=%2Flinker-features%5B%3D+%5D%5C%2Blld%2F), [3](https://github.com/search?type=code&q=language%3Atoml+%22-fuse-ld%3Dlld%22), [4](https://github.com/search?type=code&q=language%3Arust+%22-fuse-ld%3Dlld%22)). By making LLD the default linker on the `x86_64-unknown-linux-gnu` target, we will be able to speed up Rust compilation out of the box, without users having to opt in or know about it. > You can find an extended version of this stabilization report which includes analysis of crater results and more data [here](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view). ## What is being stabilized - `rust-lld` being used as the default linker on the `x86_64-unknown-linux-gnu` target. - Note that `rust-lld` is being enabled by default in the compiler artifacts distributed by our CI/rustup. It is still possible to use the system linker by default using `rust.lld = false` in `bootstrap.toml`, which can be useful e.g. for some Linux distros that might not want to use the LLD we distribute. - This is done by activating the LLD linker feature and using the self-contained linker on that target. Both of which are also usable on the CLI, if some opt outs are necessary, as described below. - `-Clinker-features=-lld` on the `x86_64-unknown-linux-gnu` target. This codegen option tells rustc to disable using the LLD linker. - Note that other options for this codegen flag (`cc`) remain unstable. - Note that only the opt-out is being stabilized, and only for `x86_64-unknown-linux-gnu`: opting in, or using the flag on other targets would still need to pass `-Zunstable-options`. - This flag is being stabilized so that users can opt out of LLD on stable, which would it turn also opt out of using the self-contained linker (since it's an LLD). - `-Clink-self-contained=-linker`. This codegen option tells rustc to use the self-contained linker. It's not particularly useful to turn it on by itself, but when enabled and combined with `-Clinker-features=+lld`, rustc will use the `rust-lld` linker wrapper shipped with the compiler toolchain, instead of some `lld` binary that the linker driver will find in the `PATH`. - Note that other options for this codegen flag (other than the previously stable `y/yes/n/no`). - Note that only the opt-out is being stabilized, and only for `x86_64-unknown-linux-gnu`: opting in, or using this flag on other targets would still need to pass `-Zunstable-options`. - This flag is being stabilized so that users can opt out of using self-contained linking on stable. Doing this would then fall back to using the system `lld`. To opt out of using LLD, `RUSTFLAGS="-Clinker-features=-lld"` would be used. To opt out of using `rust-lld`, falling back to the LLD installed on the system, `RUSTFLAGS="-Clink-self-contained=-linker"` would be used. ## Tests When enabling `rust-lld` on nightly, we also switched x64 linux to use it at stage >= 1, meaning that all tests have been running with lld since May 2024, on CI as well as contributors' machines. (Post opt-dist tests also had been using it when running their test subset earlier than that). There are also a few tests dedicated to the CLI behavior, or ensuring the default linker is indeed the one we expect: - [link-self-contained-consistency](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/link-self-contained-consistency.rs): Checks that `-Clink-self-contained` options are not inconsistent (i.e. that passing both `+linker` and `-linker` is an error). - [link-self-contained-unstable](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/link-self-contained-unstable.rs): Checks that only the `-linker` and `y/yes/n/no` options for `-Clink-self-contained` are stable. - [linker-features-unstable-cc](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/linker-features-unstable-cc.rs): Checks that only the non-lld options of `-Clinker-features` are unstable. - [linker-features-lld-disallowed](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/linker-features-lld-disallowed.rs): Checks that `-Clinker-features=-lld` is only stable on `x86_64-unknown-linux-gnu`. - [link-self-contained-linker-disallowed](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/link-self-contained-linker-disallowed.rs): Checks that `-Clink-self-contained=-linker` is only stable on `x86_64-unknown-linux-gnu`. - [no-gc-encapsulation-symbols](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/ui/linking/no-gc-encapsulation-symbols.rs): Checks that that linker encapsulation symbols are not garbage collected by LLD, so that crates like [linkme](https://github.com/dtolnay/linkme) still work. - [rust-lld](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/run-make/rust-lld): Checks that LLD is actually used when enabled with `-Clinker-features=+lld` and `-Clink-self-contained=+linker`. - [rust-lld-x86_64-unknown-linux-gnu](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/run-make/rust-lld-x86_64-unknown-linux-gnu): Checks that LLD is used by default on `x86_64-unknown-linux-gnu` when the bootstrap `rust.lld` config option is `true`. - [rust-lld-x86_64-unknown-linux-gnu-dist](https://github.com/rust-lang/rust/blob/1117bc1e6ce049495b0044dfe756afafc817d2d7/tests/run-make/rust-lld-x86_64-unknown-linux-gnu-dist): Dist test that checks that our distributed `x86_64-unknown-linux-gnu` archives actually use LLD by default. ## Ecosystem impact As already stated, LLD has been used as the default linker on x64 Linux on nightly for almost a year, and we haven't seen any blockers to stabilization in that time. There were a handful of issues reported, these are discussed later below. Furthermore, two crater runs ([November 2023](https://crater-reports.s3.amazonaws.com/pr-117684-2/index.html), [February 2025](https://crater-reports.s3.amazonaws.com/pr-137044-3/index.html)), were performed to test the impact of using LLD as the default linker. A triage of the earlier crater run was previously done [here](https://hackmd.io/OAJxlxc6Te6YUot9ftYSKQ), but all the important findings from both crater runs are reported below. Below is a list of compatibility differences between BFD and LLD that we have encountered. There is a more thorough list of differences in [this post](https://maskray.me/blog/2020-12-19-lld-and-gnu-linker-incompatibilities) from the current LLD maintainer. From that post, "99.9% pieces of software work with ld.lld without a change". --- ### `.ctors/.dtors` sections [#128286](#128286) reported an issue where LLD was unable to link certain CUDA library was using these sections that were using the `.ctors/.dtors` ELF sections. These were deprecated a long time ago (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770), replaced with a more modern `.init_array/.fini_array` sections. LLD doesn't (and won't) support these sections ([1](llvm/llvm-project#68071), [2](llvm/llvm-project#30572)), so if they appear in input object files, the linked artifact might produce incorrect behavior, because e.g. some global variables might not get initialized properly. However, the usage of `.ctors/.dtors` should be very rare in practice. We have performed a [crater run](#137044) to test this. It has identified only 8 crates where the `.ctors/.dtors` section is occurring in the final linked artifact. It was caused by a few crates using the `.ctors` link section manually, and by using a very (~6 year) old version of the [ctor](https://crates.io/crates/ctor) crate. [Crater run analysis](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view#ctorsdtors-sections) **Possible workaround** It is possible to [detect](e5e2316) if `.ctors/.dtors` section is present in the final linked artifact (LLD will keep it there, but it won't be populated), and warn users about it. This check is very cheap and doesn't even appear on [perf](#112049 (comment)). We have benchmarked the check on a 240 MiB Chrome binary, where it took 0.8ms with page cache flushed, and 0.06ms with page cache primed (which should be the common case, as the linked artifact is written to disk just before the check is performed). In theory, this could be also solved with a linker script that moves `.ctors` to `.init_array`. We think that these sections should be so rare that it is not worth it to implement any workarounds for now. --- ### Different garbage collection behavior [#130397](#130397) reported an issue where LLD prunes a local symbol, so it is missing in the linked artifact. However, BFD keeps the same symbol, so it is a regression. This is caused by a difference in linker garbage collection. Rust uses `--gc-sections` and puts each function into a separate linker section, which prunes unused code. There is some code (specifically the somewhat popular [linkme](https://github.com/dtolnay/linkme) crate) that (arguably ab-)uses so called linker encapsulation symbols to achieve distributed slices. BFD (2.37+) uses a conservative linking mode that works as intended with this behavior, but it might slightly increase binary size of the linked artifact. LLD does not use this workaround by default, which causes the sections to be eliminated, but it can be made to use the conservative mode using [`-z nostart-stop-gc`](https://lld.llvm.org/ELF/start-stop-gc.html#z-start-stop-gc). To avoid this issue, we told LLD to use the [conservative mode](#137685), which maintains backwards compatibility with BFD. We found that it has [no effect](#112049 (comment)) on compilation performance and binary size in our benchmark suite. With this change, `linkme` works. Since then, #140872 removed `linkme` distributed slice's dependence on conservative GC behavior, so this PR also removes that conservative mode: no transition period is necessary, as the PR immediately fixed the crate with no source changes. [Crater run analysis](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view#Different-garbage-collection-behavior) --- ### Various uncommon issues A small number of issues that only occurred in a handful of instances were found in crater, and it is unclear if LLD is at fault or if there is some other issue that was not detected with BFD. You can examine these [here](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view#Various-uncommon-issues). --- ### Missing jobserver support LLD doesn't support the jobserver protocol for limiting the number of threads used, it simply defaults to using all available cores, and is one of the reasons why it's faster than BFD. However, this should mostly be a non-issue, because most of the linking done during high parallelism sections of `cargo build` is linking of build scripts and proc macros, which are typically very fast to link (e.g. ~50ms), and a potential oversubscription of cores thus doesn't hurt that much. When the final artifact is linked (which typically takes the most time), there should be no other sources of parallelism conflicts from compiling other code, so LLD should be able to use all available threads. That being said, it is a difference of behavior, where previously a `-j` flag was generally not using more cpu than the specified limit. It can be impactful in some resource-constrained systems, but to be clear that is already the case today due to [cargo parallelism](rust-lang/cargo#9157). This could be one reason to opt out of using `rust-lld` on some systems. LLD has support for limiting the number of threads to use, so in theory rustc could try to get all the jobserver tokens available and use that as lld's thread limit. It'd still be suboptimal as new tokens would not be dynamically detected, and we could be using less threads than available. We did a benchmark on a real-world crate that shows that using multiple LLD threads for intermediate artifacts doesn't seem to have a performance effect. You can find it [here](https://hackmd.io/tFDifkUcSLGoHPBRIl0z8w?view#Missing-jobserver-support). --- #### Opting out of LLD in the ecosystem We have also examined repositories where people opted out of LLD on nightly, using [this GitHub query](https://github.com/search?q=%22linker-features%3D-lld%22&type=code). The summary can be found below: <details> <summary>Summary of LLD opt outs</summary> > This examination was performed on 2025-03-09. Here we briefly examine the most common reasons why people use `-Zlinker-features=-lld`, based on comments and git history. - Nix/NixOS ([1](https://github.com/rszyma/vscode-kanata/blob/59d703dff5a238b14ab3759cac27f73fb34bbcfe/flake.nix#L33), [2](https://github.com/sbernauer/breakwater/blob/3cc3449fc126c5c99d4a971733fd32be589884e0/.cargo/config.toml#L4), [3](https://github.com/tiiuae/ebpf-firewall/blame/32bdb17cedd1c9bea1ab3482623de458d95da7d0/.cargo/config.toml#L2), [4](https://github.com/jules-sommer/wavetheme-gen/blob/f5f657d014d4a30607625afb70f810c229c0294e/Cargo.toml#L4), [5](https://github.com/LayerTwo-Labs/zside-rust/blob/e4266f5c5571a1b180a9c70cf0939c7070e410c7/.cargo/config.toml#L10), [6](https://github.com/przyjacielpkp/zkml/blob/22a4aef24e9d2c77789229d7c634fc67e9eb1184/README.md?plain=1#L78), [7](https://github.com/LayerTwo-Labs/thunder-rust/blob/2222d53474c8d2d0428b4c56f8157095dced6d5a/.cargo/config.toml#L2), [8](https://github.com/enesoztrk/nixos-tc-aya-test/blob/b2ffa59d3eba8b60fd04b0a4c8bbe047400eb981/.cargo/config.toml#L4), [9](https://github.com/lowRISC/container-hotplug/blob/3ead4ef9c7f79c303392178c99677dbecff1aea6/.cargo/config.toml#L2), [10](https://github.com/Eliah-Lakhin/ad-astra/blob/ca6b8c8a5dba7bb5e894f3f1013f17876962a021/work/examples/lsp-client/src/extension.ts#L94)) - There was an [issue](NixOS/nixpkgs#312661) with LLD, which seems to have been fixed with NixOS/nixpkgs#314268. It's unclear whether that fixed all the Nix issues though. - Issues with linkme ([1](https://github.com/0xPolygonZero/zk_evm/blob/ef388619ffbd5305209519a3a5bc0396185d68ac/.cargo/config.toml#L4), [2](https://github.com/conjure-cp/conjure-oxide/blob/be0fc5827ff90e8486d416cc184b6ce24f73bf01/README.md?plain=1#L20), [3](https://github.com/clchiou/garage/blob/c5d8444d56bb6ee24ca95e5c6b9880ed996f4918/rust/.cargo/config.toml#L6), [4](https://github.com/PonasKovas/craftflow/blob/5b4cc1a5196e08a975368399fefda4b71f3a2f6f/.cargo/config.toml#L3), [5](https://github.com/kezhuw/zookeeper-client-rust/blob/4e27c7de2a7cc5e709af012b791c8fea9bb47f1f/.github/workflows/ci.yml#L82), [6](https://github.com/niklasdewally/conjure-oxide/blob/8fe60c12bca7011a2f9eded4b7c95ad0e77b6f44/.github/workflows/code-coverage.yml#L48), [7](https://github.com/kezhuw/spawns/blob/c8b468379805de9df3287c01b94b4ed3db6b61ed/.github/workflows/ci.yml#L74)) - These should be resolved with the conservative garbage collection ([#137685](#137685)). - Bazel ([1](https://github.com/google-parfait/confidential-federated-compute/blob/1823f69ed8f5f4f819f7bfa21da1ca629fdc826b/.bazelrc#L71)), WASM ([1](https://github.com/Eliah-Lakhin/ad-astra/blob/ca6b8c8a5dba7bb5e894f3f1013f17876962a021/work/examples/wasm-build.sh#L37), [2](https://github.com/yacineb/pgrx-wasi-test/blob/2bf99037ca1b650b2cbc35f1257a87fb6ead0920/build.sh#L21)), uncategorized ([2](https://github.com/nbdd0121/r2vm/blob/5118be6b9e757c6fef2f019385873f403c23c548/.cargo/config.toml#L3), [3](https://github.com/Wyvern/Img/blame/45020c7e1dc4926c8129647014c708db0c13f463/.cargo/config.toml#L209), [4](https://github.com/arnaudpoullet/leptos-i18n-compile-error/blob/042eb835f7ca0dc36be67cf7fe65b35b22b6059f/README.md?plain=1#L89), [5](https://github.com/JonLeeCon/numerical-rust-cpu/blob/fd0b3006768ed81c56147044dc05c92b11b7b6f0/exercises/.cargo/config.toml#L13), [6](https://github.com/PonasKovas/shallowclone/blob/be65f2ec923cac6ceedbc8db520c89969ebfce7c/.github/workflows/rust.yml#L20)) - Reason unclear. </details> ## History The idea to use a faster linker by default has been on the radar for quite some time ([#39915](#39915), [#71515](#71515)). There were [very early attempts](#29974) to use the gold linker by default, but these had to be [reverted](#30913) because of compatibility issues. Support for LLD was implemented back in [2017](#40018), but it has not been made default yet, except for some more niche targets, such as [WASM](#48125), [ARM Cortex](#53648) or [RISC-V](#53822). It took quite some time to figure out how should the interface for selecting the linker (and the way it is invoked) look like, as it differs a lot between different platforms, linkers and compiler drivers. During that time, LLD has matured and achieved [almost perfect compatibility](https://maskray.me/blog/2020-12-19-lld-and-gnu-linker-incompatibilities) with the default Linux linker (BFD). - [#56351](#56351) stabilized `-Clinker-flavor`, which is used to determine how to invoke the linker. It is especially useful on targets where selecting the linker directly with `-Clinker` is not possible or is impractical. - December 2018, author `@davidtwco,` reviewer `@nagisa` - [#76158](#76158) stabilized `-Clink-self-contained=[y|n]`, which allows overriding the compiler's heuristic for deciding whether it should use self-contained or external tools (linker, sanitizers, libc, etc.). It only allowed using the self-contained mode either for everything (`y`) or nothing (`n`), but did not allow granular choice. - September 2020, author `@mati864,` reviewer `@petrochenkov` - [#85961](#85961) implemented the `-Zgcc-ld` flag, which was a hacky way of opting into LLD usage. - June 2021, author `@sledgehammervampire,` reviewer `@petrochenkov` - [MCP 510](rust-lang/compiler-team#510) proposed stabilizing the behavior of `-Zgcc-ld` using more granular flags (`-Clink-self-contained=linker -Clinker-flavor=gcc-lld`). - Initially implemented in [#96827](#96827), but `@petrochenkov` [suggested](#96827 (comment)) a slightly different approach. - The PR was split into [#96884](#96884), where it was decided what will be the individual components of `-Clink-self-contained=linker`. - And [#96401](#96401), which implemented the `-Clinker-flavor` part. - The MCP was finally implemented in [#112910](#112910). - [#116514](#116514) then removed `-Zgcc-ld`, as it was replaced by `-Clinker-flavor=gnu-lld-cc` + `-Clink-self-contained=linker`. - April 2022 - October 2023, author `@lqd,` reviewer `@petrochenkov` - Various linker handling refactorings were performed in the meantime: [#97375](#97375), [#98212](#98212), [#100126](#100126), [#100552](#100552), [#102836](#102836), [#110807](#110807), [#101988](#101988), [#116515](#116515) - The implementation of linker flavors with LLD was causing a sort of a combinatorial explosion of various options. [#119906](#119906) suggested a different approach for linker flavors (described [here](#119906 (comment))), where the individual flavors could be enabled separately using `+/-` (e.g. `+lld`). - After some back and forth, this idea was moved to `-Clinker-features` (see [comment 1](#119906 (comment)) and [comment 2](#119906 (comment))), which was implemented in [#123656](#123656). - April 2024, author `@lqd,` reviewer `@petrochenkov` - [#124129](#124129) enabled LLD by default on nightly. - April 2024, author `@lqd,` reviewer `@petrochenkov` - [#137685](#137685), [#137926](#137926) enabled the conservative gargage collection mode (`-znostart-stop-gc`) to improve compatibility with BFD. - February 2025, author `@lqd,` reviewer `@petrochenkov` (implementation), author `@kobzol,` reviewer `@lqd` (test) - [#96025](#96025) (April 2022), [#117684](#117684) (November 2023), [#137044](#137044) (February 2025): crater runs. ## Unresolved questions/concerns - Is changing the linker considered a breaking change? In (hopefully very rare) cases, it might break some existing code. It should mostly only affect the final linked artifact, so it should be easy to opt out. - Similarly, is the single-threaded behavior of such tools encompassed in our stability guarantee: it can be observed via the `-j` job limit (though I believe we have/had some open issues on sometimes using more CPU resources than the job count limit implied). As mentioned above, LLD does not support the jobserver protocol. - A concern [was raised](#71515 (comment)) about increased memory usage of LLD. We should probably let users know about the possibly increased memory usage, and jobserver incompatibility: we did so when announcing this landing on nightly. - LLD seems to produce [slightly larger](https://perf.rust-lang.org/compare.html?start=b3e117044c7f707293edc040edb93e7ec5f7040a&end=baed03c51a68376c1789cc373581eea0daf89967&stat=size%3Alinked_artifact&tab=compile) binary artifacts. This can be partially clawed back using Identical Code Folding (`-Clink-args=-Wl,--icf=all`). - Should we detect the outdated `.ctors/.dtors` sections to provide a better error message, even if that should be rare in practice? --- ### Next steps After the FCP completes: - we should land this PR at the beginning of a beta cycle, to maximize time for testing - keep an eye on the beta crater run results for possible linker issues (or do a dedicated beta crater run with only this change) - release a blog post announcing the change, and asking for testing feedback of the appropriate beta - depending on feedback, or if a period of testing of 6 weeks is not long enough, we could keep this change on beta for another cycle --- Development, testing, try builds were done in #138645. r? `@petrochenkov` `@rustbot` label +needs-fcp +T-compiler

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 18, 2024

rustbot assigned Mark-Simulacrum Apr 18, 2024

rustbot added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 18, 2024

lqd marked this pull request as ready for review April 18, 2024 19:11

lqd added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 18, 2024

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 19, 2024

Mark-Simulacrum reviewed Apr 20, 2024

View reviewed changes

src/bootstrap/src/core/build_steps/compile.rs Outdated Show resolved Hide resolved

src/ci/docker/host-x86_64/x86_64-gnu-llvm-18/Dockerfile Outdated Show resolved Hide resolved

lqd force-pushed the enable-lld branch 2 times, most recently from 1978bc2 to ded5bfb Compare April 20, 2024 16:52

This comment was marked as resolved.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 22, 2024

rustbot added this to the 1.80.0 milestone May 17, 2024

lqd deleted the enable-lld branch May 17, 2024 04:30

This was referenced May 17, 2024

Use lld by default on x64 Ubuntu 20.04 LTS #71515

Open

Linking with LLD #39915

Open

dtolnay mentioned this pull request May 18, 2024

Disable rust-lld in CI dtolnay/linkme#88

Merged

Emilgardis mentioned this pull request May 18, 2024

Nightly builds fail on many platforms now that nightly sets -fuse-ld=lld cross-rs/cross#1496

Closed

11 tasks

This was referenced May 18, 2024

Nightly CI failed: collect2: fatal error: cannot find 'ld' RalfJung/cargo-careful#31

Closed

rustc can't find the linker when --sysroot is set to a local build of the sysroot #125246

Closed

plusvic added a commit to VirusTotal/yara-x that referenced this pull request May 18, 2024

chore: disable rust-lld in nightly build.

7bc22bd

Link is currently failing with rust-lld (rust-lang/rust#124129).

pnkfelix mentioned this pull request May 20, 2024

lld doesn't work on NixOS #125321

Closed

nbdd0121 added a commit to Rust-for-Linux/klint that referenced this pull request May 20, 2024

Disable LLD

bd3ef2e

With rust-lang/rust#124129, x64 linux switches to use LLD by default, but this causes dependency issues on NixOS. Disable LLD for now. This is tracked by rust-lang/rust#125321.

torokati44 mentioned this pull request May 20, 2024

ci: Disable rust-lld on Rust nightly ruffle-rs/ruffle#16390

Merged

kezhuw added a commit to kezhuw/zookeeper-client-rust that referenced this pull request May 21, 2024

Disable rust-lld on nightly to fix ci failure

b2d6e58

Refs: dtolnay/linkme#88, rust-lang/rust#124129

kezhuw added a commit to kezhuw/spawns that referenced this pull request May 23, 2024

Disable rust-lld on nightly to fix coverage ci failure

4bae981

Refs: dtolnay/linkme#88, rust-lang/rust#124129

kezhuw mentioned this pull request May 23, 2024

Disable rust-lld on nightly to fix coverage ci failure kezhuw/spawns#5

Merged

kezhuw added a commit to kezhuw/spawns that referenced this pull request May 23, 2024

Disable rust-lld on nightly to fix coverage ci failure (#5)

c8b4683

Refs: dtolnay/linkme#88, rust-lang/rust#124129

This was referenced May 28, 2024

Support zstd-compressed ELF sections. rust-lang/backtrace-rs#626

Merged

Enable zstd for debug compression. #125642

Merged

nbdd0121 mentioned this pull request Aug 29, 2024

Rust build support doesn't use NIX_LDFLAGS in link flags NixOS/nixpkgs#326944

Closed

dianqk mentioned this pull request Oct 5, 2024

rust 1.81.0 buildable with LLVM 17.x? #131291

Closed

lqd mentioned this pull request Dec 3, 2024

Tracking issue for speeding up rustc via its build configuration #103595

Open

30 tasks

Hogeyama mentioned this pull request Mar 23, 2025

Add libssl to RPATH nix-community/fenix#191

Open

oxalica mentioned this pull request Apr 13, 2025

Nul terminate rust string literals #138504

Draft

lqd mentioned this pull request Apr 30, 2025

Use lld by default on x86_64-unknown-linux-gnu stable #140525

Merged

PeterWang-dev mentioned this pull request Jun 9, 2025

LLD linker does not have required nixpkgs wrapper script and flags NixOS/nixpkgs#24744

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable `rust-lld` on nightly `x86_64-unknown-linux-gnu` #124129

Enable `rust-lld` on nightly `x86_64-unknown-linux-gnu` #124129

Uh oh!

lqd commented Apr 18, 2024 •

edited

Loading

Uh oh!

lqd commented Apr 18, 2024

Uh oh!

This comment has been minimized.

bors commented Apr 18, 2024

Uh oh!

rustbot commented Apr 18, 2024

Uh oh!

bors commented Apr 18, 2024

Uh oh!

This comment has been minimized.

rust-timer commented Apr 19, 2024

Uh oh!

albertlarsan68 commented Apr 19, 2024

Uh oh!

Kobzol commented Apr 19, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Mark-Simulacrum commented Apr 20, 2024

Uh oh!

lqd commented Apr 20, 2024

Uh oh!

lqd commented Apr 22, 2024 •

edited

Loading

Uh oh!

This comment was marked as resolved.

rust-timer commented May 17, 2024

Uh oh!

Uh oh!

Enable rust-lld on nightly x86_64-unknown-linux-gnu #124129

Enable rust-lld on nightly x86_64-unknown-linux-gnu #124129

Uh oh!

Conversation

lqd commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lqd commented Apr 18, 2024

Uh oh!

This comment has been minimized.

bors commented Apr 18, 2024

Uh oh!

rustbot commented Apr 18, 2024

Uh oh!

bors commented Apr 18, 2024

Uh oh!

This comment has been minimized.

rust-timer commented Apr 19, 2024

Overall result: ✅ improvements - no action needed

Uh oh!

albertlarsan68 commented Apr 19, 2024

Uh oh!

Kobzol commented Apr 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mark-Simulacrum commented Apr 20, 2024

Uh oh!

lqd commented Apr 20, 2024

Uh oh!

lqd commented Apr 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

rust-timer commented May 17, 2024

Overall result: ✅ improvements - no action needed

Uh oh!

Uh oh!

Enable `rust-lld` on nightly `x86_64-unknown-linux-gnu` #124129

Enable `rust-lld` on nightly `x86_64-unknown-linux-gnu` #124129

lqd commented Apr 18, 2024 •

edited

Loading

Kobzol commented Apr 19, 2024 •

edited

Loading

lqd commented Apr 22, 2024 •

edited

Loading