[receiver/hostmetrics/process] "error reading username for process ... error reading parent pid for process ... (pid 1): invalid pid 0"

[MarioAlexis]

What happened?

Description

Installing latest otel-collector version (0.60.0) receiver/hostmetrics/process complains about "invalid pid 0" on a CNF
environment. My lack of knowledge on GO does not allow me to determine if this call "parentPid, err := parentPid(handle, pid)"
block metrics exposure "per process" as documentation set out. Even if we set the parameter "mute_process_name_error" at
true.

Does the receiver/hostmetrics/process expose really metrics "per process"? I only can see

# HELP process_memory_physical_usage The amount of physical memory in use.
# TYPE process_memory_physical_usage gauge
process_memory_physical_usage 5.197824e+07
# HELP process_memory_virtual_usage Virtual memory size.
# TYPE process_memory_virtual_usage gauge
process_memory_virtual_usage 8.02689024e+08

Steps to Reproduce

docker run -v $(pwd)/config.yaml:/etc/otelcol/config.yaml otel/opentelemetry-collector:0.60.0

Expected Result

No Error output

Actual Result

        error   scraperhelper/scrapercontroller.go:197  Error scraping metrics  {"kind": "receiver", "name": "hostmetrics", "pipeline": "metrics", "error": "error reading username for process \"otelcol\" (pid 1): open /etc/passwd: no such file or directory; error reading parent pid for process \"otelcol\" (pid 1): invalid pid 0", "scraper": "process"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
        go.opentelemetry.io/collector@v0.60.0/receiver/scraperhelper/scrapercontroller.go:197
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
        go.opentelemetry.io/collector@v0.60.0/receiver/scraperhelper/scrapercontroller.go:172

Collector version

v0.60.0

OpenTelemetry Collector configuration

receivers:
  hostmetrics:
    collection_interval: 5s
    scrapers:
      filesystem:
      network:
      processes:
      process:
        mute_process_name_error: true
exporters:
  prometheus:
    endpoint: 0.0.0.0:5656
    resource_to_telemetry_conversion:
      enabled: false
service:
  pipelines:
    metrics:
      receivers:
      - hostmetrics
      exporters:
      - prometheus
  telemetry:
    logs:
      level: WARN
    metrics:
      level: detailed

Log output

        error   scraperhelper/scrapercontroller.go:197  Error scraping metrics  {"kind": "receiver", "name": "hostmetrics", "pipeline": "metrics", "error": "error reading username for process \"otelcol\" (pid 1): open /etc/passwd: no such file or directory; error reading parent pid for process \"otelcol\" (pid 1): invalid pid 0", "scraper": "process"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
        go.opentelemetry.io/collector@v0.60.0/receiver/scraperhelper/scrapercontroller.go:197
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
        go.opentelemetry.io/collector@v0.60.0/receiver/scraperhelper/scrapercontroller.go:172

Additional context

No response

[github-actions]

Pinging code owners: @dmitryax. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[MarioAlexis]

Hi

The fact that metrics per processes are missing when using receiver/hostmetrics/process, are we talking about a possible regression on that feature ?

Here an example of metrics that we are currently scraping.

process_memory_virtual_usage{k8s_pod_name="otel-server-6588f888c4-kgkjj",k8s_pod_uid="2551c638-13d6-4e2a-b70d-e86037f2bd57",orchestrator_namespace="platform-dev",process_command="/usr/bin/otelcol",process_command_line="/usr/bin/otelcol --config /opt/myapp/myapp-telemetry/etc/myapp-telemetry-collector-config.yaml",process_executable_name="otelcol",process_executable_path="/usr/bin/otelcontribcol",process_owner="root",process_pid="43"} 1.262133248e+09

process_memory_physical_usage{k8s_pod_name="otel-server-6588f888c4-kgkjj",k8s_pod_uid="2551c638-13d6-4e2a-b70d-e86037f2bd57",orchestrator_namespace="platform-dev",process_command="/usr/bin/otelcol",process_command_line="/usr/bin/otelcol --config /opt/myapp/myapp-telemetry/etc/myapp-telemetry-collector-config.yaml",process_executable_name="otelcol",process_executable_path="/usr/bin/otelcontribcol",process_owner="root",process_pid="43"} 3.43281664e+08

process_cpu_time{k8s_pod_name="otel-server-6588f888c4-kgkjj",k8s_pod_uid="2551c638-13d6-4e2a-b70d-e86037f2bd57",orchestrator_namespace="platform-dev",process_command="/usr/bin/otelcol",process_command_line="/usr/bin/otelcol --config /opt/myapp/myapp-telemetry/etc/myapp-telemetry-collector-config.yaml",process_executable_name="otelcol",process_executable_path="/usr/bin/otelcontribcol",process_owner="root",process_pid="43",state="system"} 566.98

process_cpu_time{k8s_pod_name="otel-server-6588f888c4-kgkjj",k8s_pod_uid="2551c638-13d6-4e2a-b70d-e86037f2bd57",orchestrator_namespace="platform-dev",process_command="/usr/bin/otelcol",process_command_line="/usr/bin/otelcol --config /opt/myapp/myapp-telemetry/etc/myapp-telemetry-collector-config.yaml",process_executable_name="otelcol",process_executable_path="/usr/bin/otelcontribcol",process_owner="root",process_pid="43",state="user"} 834.15

This is not possible anymore on recent versions.
The above metrics are coming from Otel-collector-contrib version 6a1c247 or 0.42.0

$> /usr/bin/otelcontribcol --version
otelcontribcol version 6a1c247

Thank you for your time.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/hostmetrics: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

Does the receiver/hostmetrics/process expose really metrics "per process"? I only can see

Yes, it does. The process scraper collects metrics per process running on the host, but, since you run the collector in a container, you have only one process.

@MarioAlexis I'm trying to see what the problem is, and why you don't see metrics anymore. I see that you report different errors:

• error reading parent pid for process in the issue title
• error reading username for process in the log output snippets.

Any of them should not cause missing metrics, only resource attributes. Do you have any other reported errors by a chance?

[MarioAlexis]

Hi @dmitryax

You're right. I must have discarded a portion of the output while formating the text. I have updated the description of the issue with, this time, the good output. Sorry about that.

I will update the issue title by adding error reading username for process, since there's 2 errors on a single output.

This is still reproducible on version 0.66.0

Any of them should not cause missing metrics

I was thinking that maybe the error prompt by otelcol-contrib was causing the missing hostmetrics/process metrics.
My real problem on why I was not able to get metrics per process it's because of this param inside prometheus exporter:

    resource_to_telemetry_conversion:
      enabled: false

Setting that to true I was able to see all metrics for each process.

But otelcol-contrib still logging this error

        error   scraperhelper/scrapercontroller.go:197  Error scraping metrics  {"kind": "receiver", "name": "hostmetrics", "pipeline": "metrics", "error": "error reading username for process \"otelcol\" (pid 1): open /etc/passwd: no such file or directory; error reading parent pid for process \"otelcol\" (pid 1): invalid pid 0", "scraper": "process"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
        go.opentelemetry.io/collector@v0.60.0/receiver/scraperhelper/scrapercontroller.go:197
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
        go.opentelemetry.io/collector@v0.60.0/receiver/scraperhelper/scrapercontroller.go:172

Even by having mute_process_name_error param set to true

[dmitryax]

The logged error is expected because you run the collector in a container that has only one process without any parent process or username. This is not a typical usage of this scraper, this scraper is expected to be run on a Linux host collecting metrics from all other processes.

What's your use case? Do you need to get CPU/memory metrics of the collector? In that case, you can use hostmetrics memory and cpu scrapers since your host = one collector process container. Also, the collector exposes its own metrics that also include cpu/memory AFAIRm that can be configured under service::telemetry.

[MarioAlexis]

Thank you to clarify the purpose of this scraper.
We are currently using Otelcol-contrib on a container(k8s environement) to scrape static endpoints and exposing local metrics per process.

This is not a typical usage of this scraper, this scraper is expected to be run on a Linux host collecting metrics from all other processes.

Is there another scrapper more suitable for container environment that will expose metrics per process ?
Even if hostmetrics/process is expected to run on a Linux host, it's actually working pretty fine on a container.
I was wondering if the mute_process_name_error param should also mute that kind of error.

We don't want Otelcol to log an error when isn't really an issue. Could be a warning log.