Mongodb Atlas receiver not able to pull logs issue

[ammarasheikh]

Component(s)

receiver/mongodbatlas

Describe the issue you're reporting

Hi,

Sorry for my lack of knowledge as I am using otelemetry mongodb atlas reciever for the first time. I am using the example yaml configuration as below: but I am keep receiving this warning and not able to retrieve any logs. I have the API keys having or owner permission. Not quite sure why I am not seeing any logs. Please help. Thanks

2023-11-01T18:11:20.383Z warn mongodbatlasreceiver@v0.81.0/logs.go:229 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos.gz"} ││ 2023-11-01T18:11:20.687Z warn mongodbatlasreceiver@v0.81.0/logs.go:259 Attempted to retrieve audit logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos-aud │

Here's my config:
receivers:
mongodbatlas:
# existing metrics settings
public_key: ${MONGODB_ATLAS_PUBLIC_KEY}
private_key: ${MONGODB_ATLAS_PRIVATE_KEY}
granularity: PT1M
retry_on_failure:
enabled: true
# new logging settings
logs:
enabled: true #default true
projects: # List of projects to monitor logs from
- name: project_1 # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true# The user needs to enable this in Atlas so it needs to be enabled here

[github-actions]

Pinging code owners:

• receiver/mongodbatlas: @djaglowski @schmikei

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ammarasheikh]

hi @djaglowski & @schmikei, can you please help?

[schmikei]

@ammarasheikh I apologize for the delay, my email has been swamped as of late!

Just confiriming, have you configured audit on the mongodbatlas side?

https://www.mongodb.com/docs/atlas/database-auditing/

and have the procedure steps been followed on that page?

[ammarasheikh]

hi @schmikei, Thanks for getting back. Yes I had enabled audit logs on the mongdodb atlas cluster.

[schmikei]

hi @schmikei, Thanks for getting back. Yes I had enabled audit logs on the mongdodb atlas cluster.

🤔 just confirming then are you able to use the atlas CLI to grab the audit logs you're looking for? And if you want to share more cluster info that may be helpful in trying to assist you!

https://www.mongodb.com/docs/atlas/cli/stable/command/atlas-logs-download/#atlas-logs-download

[ammarasheikh]

@schmikei , I was able to download the logs using Atlas CLI earlier but today I am getting an error "undefined response type". I am trying to resolve that issue. Just to provide additional details for the cluster it is a 3-node replica set, please see below screenshot. Another interesting thing which I see in the activity logs is logs continue to download successfully but I am not able to see them. The only message I see is the warning.

[ammarasheikh]

I am able to download the logs using atlas cli:

[schmikei]

Is it possible that some of your replica hosts (maybe some secondary) have 0 byte length for the interval of logs(I believe 5 minute windows from start of collection with a sliding window)? I.e. we downloaded a gzipped response of 0 bytes within the receiver? Are you not getting any process or audit logs from any host?

I think I was able to replicate the logged behavior using a low traffic cluster but primarily was getting these for the secondary hosts of the replica set.

I do think some improvement could be a logging enhancement to list the host and time range that the warn logging message is getting generated from.

[ammarasheikh]

Hi @schmikei, I have tried downloading the logs from secondary hosts and none of them has 0 byte . I know the cluster I am using is low traffic but still I can see logs generated from each replica node. Does the MongoDB Atlas receiver retrieve the logs from each replica host or just the primary as in the config we only provide project name whereas in the atlas cli we provide the individual host name to download the .gz logs?

[schmikei]

I have tried downloading the logs from secondary hosts and none of them has 0 byte

This depends mostly on the window the logs are requested, the default for atlas logs download goes back 1 days worth of logs whereas the receiver chunks its processing looking at the last 5 minutes of collection to keep the window of logs to process efficient I think is the intention.

https://www.mongodb.com/docs/atlas/cli/v1.9/command/atlas-logs-download/

Thats why I think at minimal we could improve the log that is emitted to print out the date range that is being requested from each host. I'm sort of strapped for time at the moment with some other projects but should be a small change if anybody wants to take a stab at it.

Does the MongoDB Atlas receiver retrieve the logs from each replica host or just the primary as in the config we only provide project name whereas in the atlas cli we provide the individual host name to download the .gz logs?

So right now audit logs are collected on a per-project basis by looking at each cluster per project scope and collects all hosts listed under that cluster scope. So that means each member of your replica set is being collected from for these logs when the receiver requests them.

I don't think I was involved on that initial decision, so I can't add context into why that is the current direction but is just the current state of the receiver and changing it might be considered a breaking change for some users.

[ammarasheikh]

Hi @schmikei, I am still not getting anywhere, I have added multiple projects to my config but still I am getting the same warning. I should at least see some logs as I know some of the clusters have activity so receiver should be able to retrieve logs. Is there a debugging option that could at least show which project the receiver has tried downloading the logs? Here's my new config with multiple projects, does it look good to you?

receivers:
mongodbatlas:
# existing metrics settings
public_key: xxxxxxx
private_key: xxxxxxxxxxxxxxxxxxxxxxxxxxx
granularity: PT1M
retry_on_failure:
enabled: true
#new logging settings
logs:
enabled: true #default true
projects: # List of projects to monitor logs from
- name: TRIPX_NONPROD # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: Hoth # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: Rahul_Mongo_db # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: aks-yoda-app-scheduler # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: alert-poc-prj # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: dependency-leanix # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
exporters:
logging:
file:
path: /usr/src/app/log/mongologs.json
rotation:
max_megabytes: 10
max_days: 3
max_backups: 3
localtime: true
service:
pipelines:
logs:
receivers: [mongodbatlas]
exporters: [file]