Why did syn-1.0.89 opt full suddenly get noisy?

[pnkfelix]

Spawned off of rust-lang/rust#96000 (comment)

It seems like syn-1.0.89 opt full suddenly became much noisier around 2022-04-17, and I do not know why:

I'm mostly filing this ticket because I'm uncomfortable saying "oh well that's life" while letting this benchmark continue to sit here and potentially flummox future performance triage efforts.

[nnethercote]

Something relating to codegen unit boundaries, maybe?

[Mark-Simulacrum]

Looks like indeed some major variability in LLVM query times -- so that's probably the right diagnosis. Unclear what actually caused that, though rust-lang/rust#96016 is the first bump on the graph -- it might be that preserving spans is making the symbol hash or something similar to it vary over time for something and as a result causing variability; I think we've seen something similar before.

[rylev]

Obviously the best thing to do here is to find the source of the noise, but should we consider not running this benchmark in the meantime (at least full builds)? This has shown up in many PRs as the only source of a perf regression and it causes confusion and makes folks trust the perf suite less.

[nnethercote]

Shouldn't the confidence testing identify that this benchmark is noisy, and stop reporting it unless the changes are large enough?

[rylev]

Eventually, this might happen if the benchmark consistently bounces back and forth. However, it does take time for that noise to distribute through the historical data. I'm not sure when that will happen, but you might be right that we can afford to just wait.