[exporter/otlp] Review telemetry

[mx-psi]

As part of the stabilization of the OTLP exporter, we want to ensure that it has the core self telemetry needed to understand the component and debug common issues. This issue tracks reviewing existing telemetry and defining any new telemetry needed to make the component observable. It should be rare that we add new telemetry as per the 1.0 tenet that "The Collector is already used in production at scale and has been tested in a variety of environments".

Blockers

• [exporterhelper] Use span links when tracing across queue #12212 (only the part not related to persistent queues)

[jade-guiton-dd]

exporter/otlp and exporter/otlphttp telemetry review

Because most of the internal telemetry in these two components is emitted by exporterhelper, I will make a joint review of both of them here.

Review

This review is based on the observability requirements for stable components.

⚠️ Like the receiver, a lot of this telemetry is not emitted for profiles, only traces/metrics/logs. These components should NOT be considered stable for profiles.

1. How much data the component receives.

✅ This should be covered by pipeline instrumentation, once implemented.

2. How much data the component outputs.

✅ The otelcol_exporter_sent_(log_records / metric_points / spans) metric should suffice.

3. How much data is dropped because of errors.

✅ When queueing and batching are disabled, this should be covered by pipeline instrumentation, although this is probably a rare case.

✅ Otherwise, the otelcol_exporter_enqueue_failed / _send_failed / _sent metrics do a good job of counting errors of different types.

⚠️ The only caveat is that _send_failed includes connection errors and errors caused by the Collector shutting down. The former can reasonably be considered downstream errors, and the latter should be rare and easy to distinguish, so I don't think this necessarily needs modification.

4. Details for error conditions.

✅ Error messages are variously available as part of emitted spans, span events, and logs.

5. Other possible discrepancies between input and output, if any.
   • How much data is currently held by the component, and how much can be held if there is a fixed capacity.

✅ This is handled by the otelcol_exporter_queue_size and otelcol_exporter_queue_capacity metrics (both gauges).

6. Processing performance.
   As an application of this, components which hold items in a queue should allow differentiating between time spent processing a batch of data and time where the batch is simply waiting in the queue.
   If multiple spans are emitted for a given batch (before and after a queue for example), they should either belong to the same trace, or have span links between them, so that they can be correlated.

❌ When the queue is enabled (true by default), the exporter span for a batch is not correlated with the previous component's span (different trace, no span link). Moreover, there is no span for the initial Consume operation that queued the data. This means we can only identify the post-queueing element of latency, and there is no way to know how long a batch stayed in the queue.

All internal telemetry emitted by a component should have attributes identifying the specific component instance that it originates from. This should follow the same conventions as the pipeline universal telemetry.

⚠️ The exporter attribute on the exporterhelper spans and metrics is enough to disambiguate the exporter configuration, and the span/metric names + signal span attribute allow disambiguating between signals, which is enough to identify the component instance. However, this does not follow pipeline telemetry conventions.

Conclusion and actions required

There are two blockers for stabilization on traces/metrics/logs:

• Emit a span for the batch enqueue operation, and link it to the related processing span(s)
  • Issue: [exporterhelper] Use span links when tracing across queue #12212
• Follow pipeline telemetry conventions for identifying components
  • Issue: [exporterhelper] Implement new component-identifying attributes #12213

[mx-psi]

This looks reasonable and I think we can go ahead with opening the pending issues. I have two small comments/questions:

⚠️ The only caveat is that _send_failed includes connection errors and errors caused by the Collector shutting down. The former can reasonably be considered downstream errors, and the latter should be rare and easy to distinguish, so I don't think this necessarily needs modification.

I don't understand the 'Collector shutting down' part. Why should this not be a 'send failed' error?

Other improvements we may want to do before stabilization but which aren't blockers [...]

I think following the conventions should be a blocker

[jade-guiton-dd]

I don't understand the 'Collector shutting down' part. Why should this not be a 'send failed' error?

In that paragraph, I claim that _send_failed mostly encompasses downstream errors, and the different metrics can thus be used to distinguish those from internal errors. Errors caused by a Context being cancelled by the Collector shutting down are an exception to that, clearly being internal errors.

I think following the conventions should be a blocker

Right, we discussed this afterward. I'll change that.

I'll go ahead and open the related issues.