Increased request latency discovered in 0.106.1 (> 0.105.0)

[arielvalentin]

Describe the bug

This issue was originally reported in CNCF slack by Srinivas Bevara and confirmed by @codeboten who asked me to open this issue

After Upgrade to v0.106.1 from v0.104.0 we have run into issues where we see an increase in request latency being reported by both the collector's internal telemetry and our ingress proxies.

Memory remains unaffected however pprof is reporting garbage collection seems to have increased due internal telemetry tracing related allocations:

This is unexpected since tracing telemetry is disabled in our deployments.

Steps to reproduce

Run a load test comparing v0.104.0 and v0.106.1

What did you expect to see?

• p99 Request latency and throughput should be comparable between both versions ~74ms
• Internal telemetry must not generate recording spans when it is disabled

What did you see instead?

• An increase in p99 latency to > 1s
• Profiling reveals increased objected allocations related to internal telemetry span data
• Profiling reveals an increase in time spent in GC

What version did you use?

v0.106.1

What config did you use?

Our configuration has many transformations and connector pipelines. The bare minimum you may likely need is the one you use for benchmarking.

Environment

Ubuntu 20.04.1

Additional context

Screen shots of increased p99 latency and CPU and steady memory

[codeboten]

After some discussion amongst maintainers, we want to provide a quick fix for this issue via the feature gate service.noopTracerProvider, and further investigate what's the cause of this problem before deciding if there should be a permanent configuration option to set a no-op tracer provider.

Returning a noop tracer provider if no tracing is configured in the telemetry settings would break the current behaviour, which the zpages extension relies on to provide telemetry about traces.

[arielvalentin]

Was there a recent change in zpages results in an increase in the number spans being allocated?

[bvsvas]

Thanks, @arielvalentin, for taking this up and raising a GitHub issue.

[arielvalentin]

@codeboten did this feature flag make it into v0.107.0?

Yes it did https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.107.0

[arielvalentin]

I recently upgraded to v0.107.0 from v0.104.0. During off peak times performance is about on-par but during our high peak times our p99 skyrockets and memory spikes forcing the memory limiter to trigger hard limits responding with 503s.

I have enabled the feature gate to ensure we are using the noop-tracer but there still seems to be an increase in memory allocations and latency.

Our heavy usage of OTTL has not changed much between versions and as far as I can tell that is what appears the most in the profiles along with a high number of context objects being allocated by the noop-tracer.

profiles-20240823.tar.gz

[codeboten]

Re-opening to continue the investigation

[TylerHelmuth]

@arielvalentin Did the OTTL performance degrade, or are you saying it remained the majority?