syncTargetAllocator in prometheus metrics receiver doesnt detect regex changes in metrics relabel config

[rashmichandrashekar]

Component(s)

receiver/prometheus

What happened?

Description

When prometheus scrape config is updated for metric relabel config with just regex change, the prometheus metrics receiver doesn't update the hash and hence doesnt pick up the metrics relabel config with the new regex.

Steps to Reproduce

With oteloperator's targetallcoator component enabled, update just the regex field in the scrape job to a different value. The prometheus receiver doesnt get the updated regex in the metric relabel config.

Expected Result

New config should have updated regex for metric relabeling.

Actual Result

The new regex is not picked up by the prometheus metrics receiver.

Collector version

v0.85.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration - Configuration applied to the otel collector's operator config

scrape_configs:
  - job_name: kube-proxy
    scrape_interval: 30s
    label_limit: 63
    label_name_length_limit: 511
    label_value_length_limit: 1023
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: keep
      source_labels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_pod_name
      separator: "/"
      regex: kube-system/kube-proxy.+
    - source_labels:
      - __address__
      action: replace
      target_label: __address__
      regex: "regex1"
      replacement: "$$1:10249"

scrape_configs:
  - job_name: kube-proxy
    scrape_interval: 30s
    label_limit: 63
    label_name_length_limit: 511
    label_value_length_limit: 1023
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: keep
      source_labels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_pod_name
      separator: "/"
      regex: kube-system/kube-proxy.+
    - source_labels:
      - __address__
      action: replace
      target_label: __address__
      regex: "regex2"
      replacement: "$$1:10249"

No response

Log output

No response

Additional context

This is related to this issue. The same fix needs to be made here

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Can you reproduce this with the collector configuration? It is hard to tell if this is an issue with the collector or operator

[rashmichandrashekar]

Can you reproduce this with the collector configuration? It is hard to tell if this is an issue with the collector or operator

Thanks @dashpole. This is only related to operator since the method syncTargetAllocator is only relevant to the operator's TargetAllocator component.

[swiatekm]

@dashpole the problem here is that we calculate the hash for scrape configs from the target allocator (which we use to check if we need to reload the scrape manager) using hashstructure, which ignores private struct fields. However, the regular expression fields in a scrape config struct are private, so they're ignored for the purpose of calculating this hash. As a result, if only the regular expression changes in the scrape config, we don't apply the change.

We fixed a similar issue in the larget allocator by simply hashing the serialized scrape configs instead.

[dashpole]

got it. Thanks @swiatekm-sumo for the clarification. Can you link to the similar issue?

[swiatekm]

Operator issue: open-telemetry/opentelemetry-operator#1926
Fix: open-telemetry/opentelemetry-operator#2171