`--fix` with `-Zthreads=8` on large workspace hangs

[CinchBlue]

Summary

I was compiling recently on my 1.3k crate + dep repo that is 1 giant Rust workspace, and it would hang on some of these crates during our build process:

   Checking quinn-proto v0.10.6
    Checking cast v0.3.0
    Checking flapigen v0.6.0
    Checking walkdir v2.5.0
    Checking num-bigint v0.4.4
    Checking num-iter v0.1.44
    Checking num-rational v0.3.2
    Checking num-rational v0.4.1
    Checking p256 v0.11.1
    Checking num-bigint v0.2.6
    Checking criterion-plot v0.5.0
    Checking chrono-tz v0.8.6
    Checking simba v0.8.1
    Checking nix v0.28.0
    Checking coreaudio-sys v0.2.15
    Checking plotters v0.3.5
    Checking tinytemplate v1.2.1
    Checking anes v0.1.6
    Checking bytesize v1.3.0
    Checking pathdiff v0.2.1
    Checking pkg-config v0.3.30
    Checking ciborium-ll v0.2.2

It would stop progressing the clippy command (but the rest of the cargo test and cargo build targets are okay -- only cargo clippy has the issue).

Reproducer

<cannot reproduce, is private code>

Version

rustc 1.79.0-nightly (385fa9d84 2024-04-04)
binary: rustc
commit-hash: 385fa9d845dd326c6bbfd58c22244215e431948a
commit-date: 2024-04-04
host: aarch64-apple-darwin
release: 1.79.0-nightly
LLVM version: 18.1.2

Additional Labels

No response

[blyxyas]

Could you check if cargo check --fix also has this issue? We rely on it and knowing this would reveal what part of the engine is failing.

[swlynch99]

I appear to have run into the same issue. Unfortunately this is also on a private codebase so I can't share a reproducer either. What I can do, though, is share some backtraces in the hope that they will be useful.

I am happy to provide more info as needed, since I can reproduce this hang reliably.

Version

$ rustc -vV
rustc 1.84.0-nightly (a93c1718c 2024-10-24)
binary: rustc
commit-hash: a93c1718c80b9f100056c8eec3fc37fbd6424134
commit-date: 2024-10-24
host: x86_64-unknown-linux-gnu
release: 1.84.0-nightly
LLVM version: 19.1.1

Running processes

This is mostly just from htop, but I have expanded it to include all the command line arguments for each process.

53330 swlynch ├─ cargo-clippy clippy --fix -j1
53331 swlynch │  └─ cargo fix -j1
53338 swlynch │     ├─ /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc --crate-name systemslab_util_serde --edition=2021 crates/util/serde/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=247 --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 --check-cfg cfg(docsrs) --check-cfg cfg(feature, values()) -C metadata=27ca523203e3056f -C extra-filename=-27ca523203e3056f --out-dir /home/swlynch/iop-systems/systemslab/target/debug/deps -C linker=clang -C incremental=/home/swlynch/iop-systems/systemslab/target/debug/incremental -L dependency=/home/swlynch/iop-systems/systemslab/target/debug/deps --extern chrono=/home/swlynch/iop-systems/systemslab/target/debug/deps/libchrono-1505fcf505f761ac.rmeta --extern schemars=/home/swlynch/iop-systems/systemslab/target/debug/deps/libschemars-dc4159c9465a5b3a.rmeta --extern serde=/home/swlynch/iop-systems/systemslab/target/debug/deps/libserde-84576a781cf12a83.rmeta --extern serde_with=/home/swlynch/iop-systems/systemslab/target/debug/deps/libserde_with-a7d80c0e7ced8f02.rmeta -Clink-arg=-fuse-ld=mold -Clink-arg=-Wl,--build-id=sha1 -Cforce-frame-pointers=yes -Zthreads=8
53340 swlynch │  └─ clippy-driver /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc crates/util/serde/src/lib.rs --crate-name systemslab_util_serde --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=247 --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 --check-cfg cfg(docsrs) --check-cfg cfg(feature, values()) -C metadata=27ca523203e3056f -C extra-filename=-27ca523203e3056f --out-dir /home/swlynch/iop-systems/systemslab/target/debug/deps -C linker=clang -C incremental=/home/swlynch/iop-systems/systemslab/target/debug/incremental -L dependency=/home/swlynch/iop-systems/systemslab/target/debug/deps --extern chrono=/home/swlynch/iop-systems/systemslab/target/debug/deps/libchrono-1505fcf505f761ac.rmeta --extern schemars=/home/swlynch/iop-systems/systemslab/target/debug/deps/libschemars-dc4159c9465a5b3a.rmeta --extern serde=/home/swlynch/iop-systems/systemslab/target/debug/deps/libserde-84576a781cf12a83.rmeta --extern serde_with=/home/swlynch/iop-systems/systemslab/target/debug/deps/libserde_with-a7d80c0e7ced8f02.rmeta -Clink-arg=-fuse-ld=mold -Clink-arg=-Wl,--build-id=sha1 -Cforce-frame-pointers=yes -Zthreads=8 --cap-lints=warn --edition 2021
53353 swlynch │     └─ /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc --crate-name systemslab_util_uuid --edition=2021 crates/util/uuid/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=247 --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 --check-cfg cfg(docsrs) --check-cfg cfg(feature, values()) -C metadata=19f0574ddc486574 -C extra-filename=-19f0574ddc486574 --out-dir /home/swlynch/iop-systems/systemslab/target/debug/deps -C linker=clang -C incremental=/home/swlynch/iop-systems/systemslab/target/debug/incremental -L dependency=/home/swlynch/iop-systems/systemslab/target/debug/deps --extern rand=/home/swlynch/iop-systems/systemslab/target/debug/deps/librand-fafbe45d537cf584.rmeta --extern thiserror=/home/swlynch/iop-systems/systemslab/target/debug/deps/libthiserror-8fdd09cce53e9fa3.rmeta --extern uuid=/home/swlynch/iop-systems/systemslab/target/debug/deps/libuuid-6093cb81efba3739.rmeta -Clink-arg=-fuse-ld=mold -Clink-arg=-Wl,--build-id=sha1 -Cforce-frame-pointers=yes -Zthreads=8

Backtraces

These are all taken by running thread apply all bt in gdb.

PID 53330 - cargo-clippy clippy --fix -j1

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f612aac13ea in __GI___wait4 (pid=53331, stat_loc=0x7ffdf858a750, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30

Thread 1 (Thread 0x7f6131f78b40 (LWP 53330) "cargo-clippy"):
#0  0x00007f612aac13ea in __GI___wait4 (pid=53331, stat_loc=0x7ffdf858a750, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x00007f61309bb69f in std::sys::pal::unix::process::process_inner::Process::wait () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#2  0x00007f612d921032 in std::process::Child::wait () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#3  0x000055b470107cee in cargo_clippy::main ()
#4  0x000055b470105573 in std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()> ()
#5  0x000055b470105689 in std::rt::lang_start::<()>::{closure#0} ()
#6  0x00007f61308828c1 in std::rt::lang_start_internal () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#7  0x000055b470108658 in main ()

PID 53331 - cargo fix -j1

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

Thread 7 (Thread 0x7fb3e3034640 (LWP 53352) "cargo"):
#0  0x00007fb3e8e69bcf in __GI___poll (fds=0x7fb3e3032f60, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055c26ecfee8e in cargo_util::read2::imp::read2 ()
#2  0x000055c26ecfc831 in <cargo_util::process_builder::ProcessBuilder>::exec_with_streaming ()
#3  0x000055c26e2b4f7b in <cargo::core::compiler::DefaultExecutor as cargo::core::compiler::Executor>::exec ()
#4  0x000055c26e2b8923 in cargo::core::compiler::rustc::{closure#3} ()
#5  0x000055c26e1138cd in <<cargo::core::compiler::job_queue::job::Work>::then::{closure#0} as core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::job_state::JobState,)>>::call_once::{shim:vtable#0} ()
#6  0x000055c26e1138cd in <<cargo::core::compiler::job_queue::job::Work>::then::{closure#0} as core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::job_state::JobState,)>>::call_once::{shim:vtable#0} ()
#7  0x000055c26e28c32e in <cargo::core::compiler::job_queue::job_state::JobState>::run_to_finish ()
#8  0x000055c26def2bcf in std::sys::backtrace::__rust_begin_short_backtrace::<<cargo::core::compiler::job_queue::DrainState>::run::{closure#1}, ()> ()
#9  0x000055c26e10dd03 in <<std::thread::Builder>::spawn_unchecked_<<cargo::core::compiler::job_queue::DrainState>::run::{closure#1}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} ()
#10 0x000055c26eece67b in alloc::boxed::{impl#48}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#11 alloc::boxed::{impl#48}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#12 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at std/src/sys/pal/unix/thread.rs:105
#13 0x00007fb3e8de5ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007fb3e8e77850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7fb3e3235640 (LWP 53339) "cargo"):
#0  __libc_recv (flags=<optimized out>, len=32, buf=0x7fb3e3234970, fd=6) at ../sysdeps/unix/sysv/linux/recv.c:28
#1  __libc_recv (fd=6, buf=0x7fb3e3234970, len=32, flags=0) at ../sysdeps/unix/sysv/linux/recv.c:23
#2  0x000055c26eec0602 in std::sys::pal::unix::net::Socket::recv_with_flags () at std/src/sys/pal/unix/net.rs:280
#3  std::sys::pal::unix::net::Socket::read () at std/src/sys/pal/unix/net.rs:295
#4  std::os::unix::net::datagram::UnixDatagram::recv () at std/src/os/unix/net/datagram.rs:389
#5  0x000055c26df889de in std::io::default_read_to_end::small_probe_read::<std::net::tcp::TcpStream> ()
#6  0x000055c26de7fbc5 in std::io::default_read_to_end::<std::net::tcp::TcpStream> ()
#7  0x000055c26def14ba in std::sys::backtrace::__rust_begin_short_backtrace::<<cargo::util::lockserver::LockServer>::run::{closure#0}, ()> ()
#8  0x000055c26e10d708 in <<std::thread::Builder>::spawn_unchecked_<<cargo::util::lockserver::LockServer>::run::{closure#0}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} ()
#9  0x000055c26eece67b in alloc::boxed::{impl#48}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#10 alloc::boxed::{impl#48}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#11 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at std/src/sys/pal/unix/thread.rs:105
#12 0x00007fb3e8de5ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007fb3e8e77850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7fb3e3436640 (LWP 53337) "cargo"):
#0  0x00007fb3e8e69bcf in __GI___poll (fds=0x7fb3e3434f60, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055c26ecfee8e in cargo_util::read2::imp::read2 ()
#2  0x000055c26ecfc831 in <cargo_util::process_builder::ProcessBuilder>::exec_with_streaming ()
#3  0x000055c26e2b4f7b in <cargo::core::compiler::DefaultExecutor as cargo::core::compiler::Executor>::exec ()
#4  0x000055c26e2b8923 in cargo::core::compiler::rustc::{closure#3} ()
#5  0x000055c26e1138cd in <<cargo::core::compiler::job_queue::job::Work>::then::{closure#0} as core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::job_state::JobState,)>>::call_once::{shim:vtable#0} ()
#6  0x000055c26e1138cd in <<cargo::core::compiler::job_queue::job::Work>::then::{closure#0} as core::ops::function::FnOnce<(&cargo::core::compiler::job_queue::job_state::JobState,)>>::call_once::{shim:vtable#0} ()
#7  0x000055c26e28c32e in <cargo::core::compiler::job_queue::job_state::JobState>::run_to_finish ()
#8  0x000055c26def2bcf in std::sys::backtrace::__rust_begin_short_backtrace::<<cargo::core::compiler::job_queue::DrainState>::run::{closure#1}, ()> ()
#9  0x000055c26e10dd03 in <<std::thread::Builder>::spawn_unchecked_<<cargo::core::compiler::job_queue::DrainState>::run::{closure#1}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} ()
#10 0x000055c26eece67b in alloc::boxed::{impl#48}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#11 alloc::boxed::{impl#48}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#12 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at std/src/sys/pal/unix/thread.rs:105
#13 0x00007fb3e8de5ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007fb3e8e77850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7fb3e3637640 (LWP 53336) "cargo"):
#0  0x00007fb3e8e78d30 in accept4 (fd=4, addr=..., addr_len=0x7fb3e36367b4, flags=524288) at ../sysdeps/unix/sysv/linux/accept4.c:31
#1  0x000055c26eec0461 in std::sys::pal::unix::net::{impl#0}::accept::{closure#0} () at std/src/sys/pal/unix/net.rs:260
#2  std::sys::pal::unix::cvt_r<i32, std::sys::pal::unix::net::{impl#0}::accept::{closure_env#0}> () at std/src/sys/pal/unix/mod.rs:324
#3  std::sys::pal::unix::net::Socket::accept () at std/src/sys/pal/unix/net.rs:260
#4  std::sys_common::net::TcpListener::accept () at std/src/sys_common/net.rs:455
#5  std::net::tcp::TcpListener::accept () at std/src/net/tcp.rs:837
#6  0x000055c26e52dbf3 in <cargo::util::diagnostic_server::RustfixDiagnosticServer>::run ()
#7  0x000055c26dee97a3 in std::sys::backtrace::__rust_begin_short_backtrace::<<cargo::util::diagnostic_server::RustfixDiagnosticServer>::start<<cargo::core::compiler::job_queue::JobQueue>::execute::{closure#1}::{closure#0}>::{closure#0}, ()> ()
#8  0x000055c26e10c47d in <<std::thread::Builder>::spawn_unchecked_<<cargo::util::diagnostic_server::RustfixDiagnosticServer>::start<<cargo::core::compiler::job_queue::JobQueue>::execute::{closure#1}::{closure#0}>::{closure#0}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} ()
#9  0x000055c26eece67b in alloc::boxed::{impl#48}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#10 alloc::boxed::{impl#48}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#11 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at std/src/sys/pal/unix/thread.rs:105
#12 0x00007fb3e8de5ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007fb3e8e77850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7fb3e3838640 (LWP 53335) "cargo"):
#0  __GI___libc_read (nbytes=1, buf=0x7fb3e3837a1f, fd=13) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=13, buf=0x7fb3e3837a1f, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x000055c26eec8b1d in std::sys::pal::unix::fd::FileDesc::read () at std/src/sys/pal/unix/fd.rs:92
#3  std::sys::pal::unix::fs::File::read () at std/src/sys/pal/unix/fs.rs:1264
#4  0x000055c26ed0a276 in <jobserver::imp::Client>::acquire_allow_interrupts ()
#5  0x000055c26ed09214 in std::sys::backtrace::__rust_begin_short_backtrace::<jobserver::imp::spawn_helper::{closure#1}, ()> ()
#6  0x000055c26ed0960b in <<std::thread::Builder>::spawn_unchecked_<jobserver::imp::spawn_helper::{closure#1}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} ()
#7  0x000055c26eece67b in alloc::boxed::{impl#48}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#8  alloc::boxed::{impl#48}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#9  std::sys::pal::unix::thread::{impl#2}::new::thread_start () at std/src/sys/pal/unix/thread.rs:105
#10 0x00007fb3e8de5ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fb3e8e77850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7fb3e8d4d640 (LWP 53332) "cargo"):
#0  0x00007fb3e8e78d30 in accept4 (fd=3, addr=..., addr_len=0x7fb3e8d4c844, flags=524288) at ../sysdeps/unix/sysv/linux/accept4.c:31
#1  0x000055c26eec0461 in std::sys::pal::unix::net::{impl#0}::accept::{closure#0} () at std/src/sys/pal/unix/net.rs:260
#2  std::sys::pal::unix::cvt_r<i32, std::sys::pal::unix::net::{impl#0}::accept::{closure_env#0}> () at std/src/sys/pal/unix/mod.rs:324
#3  std::sys::pal::unix::net::Socket::accept () at std/src/sys/pal/unix/net.rs:260
#4  std::sys_common::net::TcpListener::accept () at std/src/sys_common/net.rs:455
#5  std::net::tcp::TcpListener::accept () at std/src/net/tcp.rs:837
#6  0x000055c26def171d in std::sys::backtrace::__rust_begin_short_backtrace::<<cargo::util::lockserver::LockServer>::start::{closure#0}, ()> ()
#7  0x000055c26e10d9f5 in <<std::thread::Builder>::spawn_unchecked_<<cargo::util::lockserver::LockServer>::start::{closure#0}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} ()
#8  0x000055c26eece67b in alloc::boxed::{impl#48}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#9  alloc::boxed::{impl#48}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at alloc/src/boxed.rs:2454
#10 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at std/src/sys/pal/unix/thread.rs:105
#11 0x00007fb3e8de5ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007fb3e8e77850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7fb3e8d4e9c0 (LWP 53331) "cargo"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055c26eed29eb in std::sys::pal::unix::futex::futex_wait () at std/src/sys/pal/unix/futex.rs:72
#2  std::sys::sync::condvar::futex::Condvar::wait_optional_timeout () at std/src/sys/sync/condvar/futex.rs:49
#3  std::sys::sync::condvar::futex::Condvar::wait_timeout () at std/src/sys/sync/condvar/futex.rs:37
#4  0x000055c26e299da5 in <cargo::core::compiler::job_queue::DrainState>::drain_the_queue ()
#5  0x000055c26e5866c1 in <cargo::core::compiler::job_queue::JobQueue>::execute ()
#6  0x000055c26e57b0d8 in <cargo::core::compiler::build_runner::BuildRunner>::compile ()
#7  0x000055c26e5c52cc in cargo::ops::cargo_compile::compile_ws ()
#8  0x000055c26e37e48b in cargo::ops::cargo_compile::compile ()
#9  0x000055c26e3f8c36 in cargo::ops::fix::fix ()
#10 0x000055c26dd6ba80 in cargo::commands::fix::exec ()
#11 0x000055c26dd94ac5 in <cargo::cli::Exec>::exec ()
#12 0x000055c26dd8a5b0 in cargo::main ()
#13 0x000055c26dcfc623 in std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()> ()
#14 0x000055c26dd0ea49 in std::rt::lang_start::<()>::{closure#0} ()
#15 0x000055c26eeb4367 in core::ops::function::impls::{impl#2}::call_once<(), (dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at core/src/ops/function.rs:284
#16 std::panicking::try::do_call<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panicking.rs:557
#17 std::panicking::try<i32, &(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at std/src/panicking.rs:520
#18 std::panic::catch_unwind<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panic.rs:358
#19 std::rt::lang_start_internal::{closure#1} () at std/src/rt.rs:174
#20 std::panicking::try::do_call<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panicking.rs:557
#21 std::panicking::try<isize, std::rt::lang_start_internal::{closure_env#1}> () at std/src/panicking.rs:520
#22 std::panic::catch_unwind<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panic.rs:358
#23 std::rt::lang_start_internal () at std/src/rt.rs:174
#24 0x000055c26dd98155 in main ()

PID 53338 - cargo invocation with clippy-driver as a child

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f6f8e337b97 in __GI___poll (fds=0x7ffe9c5abcec, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29

Thread 1 (Thread 0x7f6f8e21c9c0 (LWP 53338) "cargo"):
#0  0x00007f6f8e337b97 in __GI___poll (fds=0x7ffe9c5abcec, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055f43c197959 in std::sys::pal::unix::pipe::read2::{closure#0} () at std/src/sys/pal/unix/pipe.rs:118
#2  std::sys::pal::unix::cvt_r<i32, std::sys::pal::unix::pipe::read2::{closure_env#0}> () at std/src/sys/pal/unix/mod.rs:324
#3  std::sys::pal::unix::pipe::read2 () at std/src/sys/pal/unix/pipe.rs:118
#4  0x000055f43c19009f in std::process::Child::wait_with_output () at std/src/process.rs:2247
#5  0x000055f43bfc7d18 in <cargo_util::process_builder::ProcessBuilder>::output ()
#6  0x000055f43b6cd811 in cargo::ops::fix::rustfix_crate ()
#7  0x000055f43b6caa52 in cargo::ops::fix::fix_exec_rustc ()
#8  0x000055f43b053d6d in cargo::main ()
#9  0x000055f43afc8623 in std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()> ()
#10 0x000055f43afdaa49 in std::rt::lang_start::<()>::{closure#0} ()
#11 0x000055f43c180367 in core::ops::function::impls::{impl#2}::call_once<(), (dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at core/src/ops/function.rs:284
#12 std::panicking::try::do_call<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panicking.rs:557
#13 std::panicking::try<i32, &(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at std/src/panicking.rs:520
#14 std::panic::catch_unwind<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panic.rs:358
#15 std::rt::lang_start_internal::{closure#1} () at std/src/rt.rs:174
#16 std::panicking::try::do_call<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panicking.rs:557
#17 std::panicking::try<isize, std::rt::lang_start_internal::{closure_env#1}> () at std/src/panicking.rs:520
#18 std::panic::catch_unwind<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panic.rs:358
#19 std::rt::lang_start_internal () at std/src/rt.rs:174
#20 0x000055f43b064155 in main ()

PID 53340 - clippy-driver

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

Thread 3 (Thread 0x7f10a27fc640 (LWP 53348) "rustc"):
#0  __GI___libc_read (nbytes=1, buf=0x7f10a27fb44f, fd=5) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=5, buf=0x7f10a27fb44f, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007f10c028484d in <jobserver::imp::Client>::acquire_allow_interrupts () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#3  0x00007f10c077e660 in <jobserver::Client>::acquire_raw () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#4  0x00007f10bdcebc4d in rustc_data_structures::jobserver::acquire_thread () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#5  0x00007f10bd7f128b in <rayon_core::registry::WorkerThread>::wait_until_cold () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#6  0x00007f10bd7ee9f9 in <rayon_core::registry::ThreadBuilder>::run () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#7  0x00007f10bdd18bc7 in <<crossbeam_utils::thread::ScopedThreadBuilder>::spawn<<rayon_core::ThreadPoolBuilder>::build_scoped<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#3}::{closure#0}::{closure#0}, rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#3}::{closure#0}::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}::{closure#0}::{closure#0}, ()>::{closure#0} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#8  0x00007f10bdd0cb5e in std::sys::backtrace::__rust_begin_short_backtrace::<alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output = ()> + core::marker::Send>, ()> () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#9  0x00007f10bdd186f2 in <<std::thread::Builder>::spawn_unchecked_<alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output = ()> + core::marker::Send>, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#10 0x00007f10c0283d2b in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#11 0x00007f10ba46bac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007f10ba4fd850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f10b1bfe640 (LWP 53342) "rustc"):
#0  __GI___libc_read (nbytes=1, buf=0x7f10b1bfd4ff, fd=5) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=5, buf=0x7f10b1bfd4ff, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007f10c028484d in <jobserver::imp::Client>::acquire_allow_interrupts () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#3  0x00007f10c077e660 in <jobserver::Client>::acquire_raw () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#4  0x00007f10bdcebc4d in rustc_data_structures::jobserver::acquire_thread () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#5  0x00007f10bd7ee9d8 in <rayon_core::registry::ThreadBuilder>::run () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#6  0x00007f10bdd18bc7 in <<crossbeam_utils::thread::ScopedThreadBuilder>::spawn<<rayon_core::ThreadPoolBuilder>::build_scoped<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#3}::{closure#0}::{closure#0}, rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#3}::{closure#0}::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}::{closure#0}::{closure#0}, ()>::{closure#0} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#7  0x00007f10bdd0cb5e in std::sys::backtrace::__rust_begin_short_backtrace::<alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output = ()> + core::marker::Send>, ()> () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#8  0x00007f10bdd186f2 in <<std::thread::Builder>::spawn_unchecked_<alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output = ()> + core::marker::Send>, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#9  0x00007f10c0283d2b in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#10 0x00007f10ba46bac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007f10ba4fd850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f10c191eb40 (LWP 53340) "clippy-driver"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f10beca2415 in std::sys::sync::condvar::futex::Condvar::wait () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#2  0x00007f10bd7e42b9 in <std::sync::condvar::Condvar>::wait::<bool> () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#3  0x00007f10bd7ee779 in <rayon_core::latch::LockLatch>::wait () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#4  0x00007f10bd7f22d8 in <rayon_core::thread_pool::ThreadPool>::wait_until_stopped () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#5  0x00007f10bdd0bb19 in crossbeam_utils::thread::scope::<<rayon_core::ThreadPoolBuilder>::build_scoped<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#3}::{closure#0}::{closure#0}, rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#3}::{closure#0}::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}, core::result::Result<core::result::Result<(), rustc_span::ErrorGuaranteed>, rayon_core::ThreadPoolBuildError>> () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#6  0x00007f10c0201ade in rustc_driver_impl::run_compiler () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#7  0x00007f10bdd356b4 in <rustc_driver_impl::RunCompiler>::run () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#8  0x0000556cc5b31a79 in clippy_driver::main ()
#9  0x0000556cc5b2d793 in std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()> ()
#10 0x0000556cc5b2d919 in std::rt::lang_start::<()>::{closure#0} ()
#11 0x00007f10c02828c1 in std::rt::lang_start_internal () from /home/swlynch/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-5396912e8af1f65d.so
#12 0x0000556cc5b328b8 in main ()

PID 53353 - the other cargo subprocess

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f31b04b8670 in __libc_recv (fd=5, buf=0x7ffcbf127d08, len=1, flags=0) at ../sysdeps/unix/sysv/linux/recv.c:28

Thread 1 (Thread 0x7f31b038e9c0 (LWP 53353) "cargo"):
#0  0x00007f31b04b8670 in __libc_recv (fd=5, buf=0x7ffcbf127d08, len=1, flags=0) at ../sysdeps/unix/sysv/linux/recv.c:28
#1  0x000055655e062602 in std::sys::pal::unix::net::Socket::recv_with_flags () at std/src/sys/pal/unix/net.rs:280
#2  std::sys::pal::unix::net::Socket::read () at std/src/sys/pal/unix/net.rs:295
#3  std::os::unix::net::datagram::UnixDatagram::recv () at std/src/os/unix/net/datagram.rs:389
#4  0x000055655d5a2c79 in cargo::ops::fix::rustfix_crate ()
#5  0x000055655d5a0a52 in cargo::ops::fix::fix_exec_rustc ()
#6  0x000055655cf29d6d in cargo::main ()
#7  0x000055655ce9e623 in std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()> ()
#8  0x000055655ceb0a49 in std::rt::lang_start::<()>::{closure#0} ()
#9  0x000055655e056367 in core::ops::function::impls::{impl#2}::call_once<(), (dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at core/src/ops/function.rs:284
#10 std::panicking::try::do_call<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panicking.rs:557
#11 std::panicking::try<i32, &(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at std/src/panicking.rs:520
#12 std::panic::catch_unwind<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panic.rs:358
#13 std::rt::lang_start_internal::{closure#1} () at std/src/rt.rs:174
#14 std::panicking::try::do_call<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panicking.rs:557
#15 std::panicking::try<isize, std::rt::lang_start_internal::{closure_env#1}> () at std/src/panicking.rs:520
#16 std::panic::catch_unwind<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panic.rs:358
#17 std::rt::lang_start_internal () at std/src/rt.rs:174
#18 0x000055655cf3a155 in main ()

cargo fix

cargo check --fix doesn't seem to be a thing any more. However, it looks like it has turned into cargo fix and I can confirm that

cargo fix -j1

hangs on my workspace.

[swlynch99]

It is also worth noting that you can run

cargo clippy --fix -j1000

as a workaround provided that you have compiled all the dependencies in advance (and have enough memory available).

[BugenZhao]

I've also reproduced this issue, this time in a public repository.

After disabling parallel front-end by removing -Zthreads=8, cargo clippy --fix no long hangs.

[matthiaskrgr]

might be fixed once rust-lang/rust#140145 lands

[samueltardieu]

@BugenZhao Would you be able to check if you can still reproduce this with the nightly version of the compiler (which includes the jobserver fix)?

[swlynch99]

I was able to test this out locally with the current nightly (2025-05-07) and cargo clippy --fix no longer hangs on my codebase.

[samueltardieu]

Since this was expected, I'll close this issue, don't hesitate anyone to reopen it if the problem persists for you.