Agent reports HEALTHY status despite connection and authentication failures

[AndersonQ]

An Elastic Agent (v9.1.0-SNAPSHOT) that is unable to connect to Fleet or Elasticsearch due to version incompatibility and authentication errors is still reporting a HEALTHY status for all of its components.

The agent was initially enrolled in a 9.1 stack, which was then replaced by an 8.17 stack. This scenario leads to expected connection failures, but the agent's health status does not reflect these failures, which is misleading. The expectation is that the agent should report an UNHEALTHY status.

For confirmed bugs, please report:

• Version:
  • Elastic Agent: 9.1.0-SNAPSHOT
• Operating System:
  • Ubuntu / all - should not be OS specific
• Discuss Forum URL:
  • N/A
• Steps to Reproduce:
  1. Enroll a 9.1.0-SNAPSHOT Elastic Agent to a Fleet Server on a 9.1 stack.
  2. Take down the 9.1 stack.
  3. Bring up a new 8.17 stack using the same hostnames for Fleet and Elasticsearch.
  4. Run elastic-agent status --output full on the agent machine.
  5. Observe that the agent and all its components report a (HEALTHY) status.
  6. Inspect the agent logs and confirm the presence of connection and authentication errors.

Agent Status Output

The following output shows all components as HEALTHY.

elastic-agent status --output full  
┌─ fleet  
│  └─ status: (HEALTHY) Connected  
└─ elastic-agent  
   ├─ status: (HEALTHY) Running  
   ├─ info  
   │  ├─ id: 15fa4650-efee-4bbb-8930-d1b6a2e57748  
   │  ├─ version: 9.1.0  
   │  └─ commit: aa2d9005350c4152f95fb5413f5d15b538921410  
   ├─ beat/metrics-monitoring  
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '731448'  
   │  ├─ beat/metrics-monitoring  
   │  │  ├─ status: (HEALTHY) Healthy  
   │  │  └─ type: OUTPUT  
   │  └─ beat/metrics-monitoring-metrics-monitoring-beats  
   │     ├─ status: (HEALTHY) Healthy  
   │     └─ type: INPUT  
   ├─ cel-default  
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '804848'  
   │  ├─ cel-default  
   │  │  ├─ status: (HEALTHY) Healthy  
   │  │  └─ type: OUTPUT  
   │  └─ cel-default-cel-cel-74fd389b-6f40-4e56-84f6-251e26c1f503  
   │     ├─ status: (HEALTHY) Healthy  
   │     └─ type: INPUT  
   ├─ filestream-default  
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '752079'  
   │  ├─ filestream-default  
   │  │  ├─ status: (HEALTHY) Healthy  
   │  │  └─ type: OUTPUT  
   │  └─ filestream-default-filestream-filestream-de7d836a-b124-489b-8fcd-b384baa68e17  
   │     ├─ status: (HEALTHY) Healthy  
   │     └─ type: INPUT  
   ├─ filestream-monitoring  
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '731373'  
   │  ├─ filestream-monitoring  
   │  │  ├─ status: (HEALTHY) Healthy  
   │  │  └─ type: OUTPUT  
   │  └─ filestream-monitoring-filestream-monitoring-agent  
   │     ├─ status: (HEALTHY) Healthy  
   │     └─ type: INPUT  
   ├─ http/metrics-monitoring  
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '731497'  
   │  ├─ http/metrics-monitoring  
   │  │  ├─ status: (HEALTHY) Healthy  
   │  │  └─ type: OUTPUT  
   │  └─ http/metrics-monitoring-metrics-monitoring-agent  
   │     ├─ status: (HEALTHY) Healthy  
   │     └─ type: INPUT  
   ├─ httpjson-default  
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '817362'  
   │  ├─ httpjson-default  
   │  │  ├─ status: (HEALTHY) Healthy  
   │  │  └─ type: OUTPUT  
   │  └─ httpjson-default-httpjson-generic-f5a31217-8cdf-4a30-b7a0-42ea1cd66e34  
   │     ├─ status: (HEALTHY) Healthy  
   │     └─ type: INPUT  
   ├─ log-default  
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '731261'  
   │  ├─ log-default  
   │  │  ├─ status: (HEALTHY) Healthy  
   │  │  └─ type: OUTPUT  
   │  └─ log-default-logfile-system-72ea0eb3-8312-4bf1-a251-a92e19135825  
   │     ├─ status: (HEALTHY) Healthy  
   │     └─ type: INPUT  
   └─ system/metrics-default  
     ├─ status: (HEALTHY) Healthy: communicating with pid '731309'  
     ├─ system/metrics-default  
     │  ├─ status: (HEALTHY) Healthy  
     │  └─ type: OUTPUT  
     └─ system/metrics-default-system/metrics-system-72ea0eb3-8312-4bf1-a251-a92e19135825  
         ├─ status: (HEALTHY) Healthy  
         └─ type: INPUT

Error Logs

Agent connection to Fleet errors:

The agent logs show repeated warnings about an invalid API key.  
{"log.level":"warn","@timestamp":"2025-06-17T07:41:33.565Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/gateway/fleet.(\*FleetGateway).execute","file.name":"fleet/fleet\_gateway.go","file.line":374},"message":"retrieved an invalid api key error '11' times. will use long scheduler","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}  
{"log.level":"warn","@timestamp":"2025-06-17T08:41:34.181Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/gateway/fleet.(\*FleetGateway).execute","file.name":"fleet/fleet\_gateway.go","file.line":374},"message":"retrieved an invalid api key error '12' times. will use long scheduler","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}  
{"log.level":"warn","@timestamp":"2025-06-17T09:41:34.683Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/gateway/fleet.(\*FleetGateway).execute","file.name":"fleet/fleet\_gateway.go","file.line":374},"message":"retrieved an invalid api key error '13' times. will use long scheduler","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

Beats errors trying to connect to Elasticsearch:

The beat logs show 401 Unauthorized errors when trying to connect to Elasticsearch.

{"log.level":"error","@timestamp":"2025-06-17T10:15:19.774Z","message":"Failed to connect to backoff(elasticsearch(https://elasticsearch:9200)): 401 Unauthorized: {\\"error\\":{\\"root\_cause\\":\[{\\"type\\":\\"security\_exception\\",\\"reason\\":\\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\\",\\"additional\_unsuccessful\_credentials\\":\\"API key: unable to find apikey with id 6X9uaJcBq-btR7Wu3ovF\\",\\"header\\":{\\"WWW-Authenticate\\":\[\\"Basic realm=\\\\\\"security\\\\\\", charset=\\\\\\"UTF-8\\\\\\"\\",\\"Bearer realm=\\\\\\"security\\\\\\"\\",\\"ApiKey\\"\]}}\],\\"type\\":\\"security\_exception\\",\\"reason\\":\\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\\",\\"additional\_unsuccessful\_credentials\\":\\"API key: unable to find apikey with id 6X9uaJcBq-btR7Wu3ovF\\",\\"header\\":{\\"WWW-Authenticate\\":\[\\"Basic realm=\\\\\\"security\\\\\\", charset=\\\\\\"UTF-8\\\\\\"\\",\\"Bearer realm=\\\\\\"security\\\\\\"\\",\\"ApiKey\\"\]}},\\"status\\":401}","component":{"binary":"filebeat","dataset":"elastic\_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher\_pipeline\_output","log.origin":{"file.line":149,"file.name":"pipeline/client\_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(\*netClientWorker).run"},"ecs.version":"1.6.0"}  
{"log.level":"error","@timestamp":"2025-06-17T10:15:20.992Z","message":"Failed to connect to backoff(elasticsearch(https://elasticsearch:9200)): 401 Unauthorized: {\\"error\\":{\\"root\_cause\\":\[{\\"type\\":\\"security\_exception\\",\\"reason\\":\\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\\",\\"additional\_unsuccessful\_credentials\\":\\"API key: unable to find apikey with id 6X9uaJcBq-btR7Wu3ovF\\",\\"header\\":{\\"WWW-Authenticate\\":\[\\"Basic realm=\\\\\\"security\\\\\\", charset=\\\\\\"UTF-8\\\\\\"\\",\\"Bearer realm=\\\\\\"security\\\\\\"\\",\\"ApiKey\\"\]}}\],\\"type\\":\\"security\_exception\\",\\"reason\\":\\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\\",\\"additional\_unsuccessful\_credentials\\":\\"API key: unable to find apikey with id 6X9uaJcBq-btR7Wu3ovF\\",\\"header\\":{\\"WWW-Authenticate\\":\[\\"Basic realm=\\\\\\"security\\\\\\", charset=\\\\\\"UTF-8\\\\\\"\\",\\"Bearer realm=\\\\\\"security\\\\\\"\\",\\"ApiKey\\"\]}},\\"status\\":401}","component":{"binary":"metricbeat","dataset":"elastic\_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":149,"file.name":"pipeline/client\_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(\*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher\_pipeline\_output","ecs.version":"1.6.0"}

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Your agent is in this block:

elastic-agent/internal/pkg/agent/application/gateway/fleet/fleet_gateway.go

Lines 370 to 381 in e728b39

	resp, took, err := cmd.Execute(ctx, req)
	if isUnauth(err) {
	f.unauthCounter++
	if f.shouldUseLongSched() {
	f.log.Warnf("retrieved an invalid api key error '%d' times. will use long scheduler", f.unauthCounter)
	f.scheduler.SetDuration(defaultGatewaySettings.ErrConsecutiveUnauthDuration)
	return &fleetapi.CheckinResponse{}, took, nil
	}
	return nil, took, err
	}

We used to automatically un-enroll when this happened, which we stopped doing because it was too destructive. We never needed to update the local agent Fleet state because it was going to be unenrolled. When we stopped unenrolling we missed updating the local Fleet connection state.

The remote state of the agent in the Fleet UI would still correctly be offline in this case though, it's only misleading if you are locally looking at the agent status and not the agent logs.