Breaking change in v0.128.0: telemetry periodic exporter otlp.headers

[joschi]

Component(s)

service

What happened?

Describe the bug

The upgrade to opentelemetry-go-contrib v1.35.0/v0.60.0/v0.29.0/v0.15.0/v0.10.0/v0.8.0/v0.7.0 in #12575 broke backward-compatibility for the headers field in the OTLPMetric type.

Using OpenTelemetry Collector 0.127.0, this was a valid configuration but breaks with OpenTelemetry Collector 0.128.0:

service:
  telemetry:
    metrics:
      address: "..."
      readers:
      - periodic:
          exporter:
            otlp:
              headers:
                "Authorization": "Basic abcdef"
                "My-Header": "My-Value"

Before the upgrade, telemetry.metrics.readers.periodic.exporter.otlp.headers was a map, after the update it's a list of name/value pairs.

Additionally, the telemetry.metrics.address key is not valid anymore but I am not sure what the reason is for that and what the replacement might be. Hints appreciated. 😉

Old type:

• https://github.com/open-telemetry/opentelemetry-go-contrib/blob/otelconf/v0.16.0/otelconf/v0.2.0/generated_config.go#L205-L206
• https://github.com/open-telemetry/opentelemetry-go-contrib/blob/otelconf/v0.16.0/otelconf/v0.2.0/generated_config.go#L112

New type:

• https://github.com/open-telemetry/opentelemetry-go-contrib/blob/otelconf/v0.16.0/otelconf/v0.3.0/generated_config.go#L308-L309
• https://github.com/open-telemetry/opentelemetry-go-contrib/blob/otelconf/v0.16.0/otelconf/v0.3.0/generated_config.go#L248-L254

Working configuration with OTel Collector v0.128.0:

service:
  telemetry:
    metrics:
      readers:
      - periodic:
          exporter:
            otlp:
              headers:
                - name: "Authorization"
                  value: "Basic abcdef"
                - name: "My-Header"
                  value: "My-Value"

What did you expect to see?

Successful startup of OpenTelemetry Collector with the configuration which was working with OpenTelemetry Collector 0.127.0.

What did you see instead?

opentelemetry-collector Error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):
opentelemetry-collector
opentelemetry-collector error decoding 'service.telemetry.metrics': decoding failed due to the following error(s):
opentelemetry-collector
opentelemetry-collector 'readers[0].periodic.exporter.otlp.headers': source data must be an array or slice, got map
opentelemetry-collector 'readers[1] periodic.exporter.otlp headers': source data must be an array or slice, got map
opentelemetry-collector '' has invalid keys: address
opentelemetry-collector 2025/06/24 10:40:00 collector server run finished with error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):
opentelemetry-collector
opentelemetry-collector error decoding 'service.telemetry metrics': decoding failed due to the following error(s):
opentelemetry-collector
opentelemetry-collector 'readers[0].periodic.exporter.otlp.headers': source data must be an array or slice,
opentelemetry-collector 'readers[1].periodic.exporter.otlp.headers': source data must be an array or slice, got map
opentelemetry-collector '' has invalid keys: address

Collector version

v0.128.0

Environment information

Environment

OS: Linux (Docker container)

OpenTelemetry Collector configuration

service:
  telemetry:
    metrics:
      address: "..."
      readers:
      - periodic:
          exporter:
            otlp:
              headers:
                "Authorization": "Basic abcdef"
                "My-Header": "My-Value"

Log output

'readers[0].periodic.exporter.otlp.headers': source data must be an array or slice, got map
'' has invalid keys: address

Additional context

No response

[joschi]

Something similar happened in #12332.

Maybe we should add some smoke tests for configuration changes, especially if they can be introduced by dependencies (instead of code in this repository).

[joschi]

Additionally, the telemetry.metrics.address key is not valid anymore but I am not sure what the reason is for that and what the replacement might be. Hints appreciated. 😉

Using a pull reader is the replacement, see open-telemetry/opentelemetry-helm-charts#1564. 😄

[codeboten]

The release notes for https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.123.0 call out the change in the usage of the address field. As mentioned in the previous comment from @joschi, the field was moved in favour of the reader configuration.

[codeboten]

@joschi can you confirm whether removing the address field allows you to use the previous headers format? I tested it locally and it worked on my end.

[joschi]

The release notes for https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.123.0 call out the change in the usage of the address field.

Yes, the address field was deprecated and ignored:

service/telemetry: Mark telemetry.disableAddressFieldForInternalTelemetry as beta, usage of deprecated service::telemetry::address are ignored

I'd argue making the startup fail when a property exists is a different thing than ignoring a property. 😅

FWIW, we only set address to an empty string because we didn't want the listener to come up and this worked until OTel Collector v0.127.0, so at least for us the changed behavior in OTel Collector v0.128.0 was a breaking change (while just deprecating and ignoring the address field in v0.123.0 was not).

[joschi]

can you confirm whether removing the address field allows you to use the previous headers format?

Yes, I can confirm that this is working. Thanks for the hint! 😄

Complete configuration used to verify:

receivers:
  nop:
exporters:
  debug: {}
service:
  pipelines:
    logs:
      receivers: [nop]
      exporters: [debug]
  telemetry:
    metrics:
      readers:
      - periodic:
          exporter:
            otlp:
              endpoint: http://localhost:4317
              insecure: true
              protocol: grpc
              headers:
                "My-Header": "My-Value"
          interval: 5000