Exporter does not respect tls/insecure_skip_verify settings

[bugzyt]

Component(s)

exporter/x

What happened?

Describe the bug
Unable to upgrade the Opentelemetry Collector helm chart 0.105v to the latest version due to a TLS issue. None of the TLS settings including insecure_skip_verify are working. I've tried different versions from 0.109v - 0.120v but same issue persist.

Doc reference: https://docs.splunk.com/observability/en/gdi/opentelemetry/common-config/collector-common-config-tls.html

Steps to reproduce
My collector config:

exporters:
  splunk_hec:
    index: "main"
    endpoint: https://splunk-hec.com/services/collector/event
    tls:
      insecure: false
      insecure_skip_verify: true
service:
  pipelines:
    logs:
      exporters: [splunk_hec]

What did you expect to see?
I expected the collector to skip TLS verification as documented.

What did you see instead?
The collector is still attempting to verify the server TLS certificate.

Collector version

0.109.0v - 0.120.0v

Environment information

Environment

OS: Red Hat Enterprise Linux CoreOS 414.92
Kubernetes: 1.27

OpenTelemetry Collector configuration

exporters:
  splunk_hec:
    index: "main"
    endpoint: https://splunk-hec.com/services/collector/event
    tls:
      insecure: false
      insecure_skip_verify: true
service:
  pipelines:
    logs:
      exporters: [splunk_hec]

Log output

2025-03-04T18:46:54.343Z info    exporterhelper/queued_retry.go:215      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "splunk_hec", "error": "Post \"https://splunk-hec.com/services/collector/event": remote error: tls: handshake failure", "interval": "16.969110576s"}

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/x: @mx-psi @dmathieu

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmathieu]

This looks like a spunk exporter issue. This isssue would then have to be moved to collector-contrib.

[github-actions]

Pinging code owners for exporter/splunkhec: @atoulme @dmitryax. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[technocrattobe]

I am able to reproduce it. v105 doesnt have this issue, it even works with ca cert or insecure_skip_verify. I checked v117 and v120 , they have this bug

[atoulme]

You are pointing that the behavior worked with 0.105.0, but errors out with 0.109.0 forward. We moved to go 1.22 in 0.108.0. This might be the problem: the TLS handshake might fail because the ciphers do not match. Please check the ciphers used.

Beyond this, please offer a reproduction path. You might need to work with Splunk support to help us reproduce the issue.

[atoulme]

See also https://tip.golang.org/doc/go1.22#minor_library_changes

By default, the minimum version offered by crypto/tls servers is now TLS 1.2 if not specified with [config.MinimumVersion](https://tip.golang.org/pkg/crypto/tls#Config.MinimumVersion), matching the behavior of crypto/tls clients. This change can be reverted with the tls10server=1 GODEBUG setting.

By default, cipher suites without ECDHE support are no longer offered by either clients or servers during pre-TLS 1.3 handshakes. This change can be reverted with the tlsrsakex=1 GODEBUG setting.

[technocrattobe]

Should the cipher change on the splunk hec endpoint?
The container usually doesnt let us use any additional ciphers

[bugzyt]

Thanks for your response @atoulme . It works when we force golang to use older ciphers in otel-collector but where does the cipher and tls version update happen in splunk?

[atoulme]

Please ask Splunk support for help there. I do not know.