[pkg/stanze] The known files in the filelog have not been updated correctly

[xieyuguang]

Component(s)

pkg/stanza, receiver/filelog

What happened?

Description

During the initial run, the progress of scanned files cannot be saved. The source code[pkg/stanza/fileconsumer/file.go] shows that during the first poll, the initial value of m.previousPollFiles is empty, so after executing m.closePreviousFiles(), it will not be added to m.knownFiles. After the second poll, m.previousPollFiles will be removed in readers := m.makeReaders(paths), so it will still not be added to m.knownFiles in subsequent executions of m.closePreviousFiles().

Steps to Reproduce

Use filelogreceiver and enable storage configuration with file_storage, first delete the local storage file, and then start otelcol. After running, observe the local storage file and find that storage does not record the matched files.

Expected Result

m.knownFiles should record the currently matched files.

Actual Result

m.knownFiles is empty.

Collector version

v0.89.0

Environment information

No response

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski
• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

I have not been able to replicate the problem.

I agree that we are not pushing checkpoints to storage until they move to knownFiles. However, the behavior of the Stop function results in all files being closed and checkpointed.

After running, observe the local storage file and find that storage does not record the matched files.

Are you observing the contents of the storage while running, or after you've stopped it?

[xieyuguang]

I observed it while running. I think the correct behavior should be consistent while running and after the stop, as there may be scenarios such as the host shutting down directly.