Add both bytes and items sizes to the persistent metadata

[bogdandrutu]

No changelog since this is not released yet.

[codecov]

Codecov Report

Attention: Patch coverage is 88.23529% with 14 lines in your changes missing coverage. Please review.

Project coverage is 91.59%. Comparing base (6408887) to head (1d561b3).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
.../exporterhelper/internal/queue/persistent_queue.go	91.37%	4 Missing and 1 partial ⚠️
exporter/exporterhelper/internal/queue/queue.go	70.58%	3 Missing and 2 partials ⚠️
.../exporterhelper/internal/queuebatch/queue_batch.go	75.00%	2 Missing and 2 partials ⚠️

❌ Your patch status has failed because the patch coverage (88.23%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13262      +/-   ##
==========================================
- Coverage   91.63%   91.59%   -0.04%     
==========================================
  Files         522      521       -1     
  Lines       29168    29240      +72     
==========================================
+ Hits        26727    26783      +56     
- Misses       1923     1938      +15     
- Partials      518      519       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[malus2077]

The current implementation looks clear and solid—really appreciate the thoughtful design.

One remaining question: After a user updates the sizer configuration, how should the queue reliably determine the correct sizer to use for mixed entries? The current logic doesn’t seem to fully address this scenario.

Do you have any suggestions or guidance on how we might handle this cleanly? Would greatly appreciate your insights here. Thanks!

[malus2077]

I’d lean toward keeping request_size here — once a request is Read() into currentDispatchItems but hasn’t hit Done() yet, queue_size is roughly writeIndex - readIndex + len(currentDispatchItems).

If we don’t stash request_size at that point, we’d need to do extra work during that in-between state. Holding onto it keeps things simple and lines up with how the other two sizer types are handled.

[bogdandrutu]

One remaining question: After a user updates the sizer configuration, how should the queue reliably determine the correct sizer to use for mixed entries? The current logic doesn’t seem to fully address this scenario.

With this proposal we have all possible sizes recorded in the metadata. It does not matter what sizer user used before or they will use now, since we have all correct sizes.

[malus2077]

With this proposal we have all possible sizes recorded in the metadata. It does not matter what sizer user used before or they will use now, since we have all correct sizes.

I’ve completed the remaining implementation based on this PR—see #13274 for details. Please pay special attention to how legacy data sizes are handled in that PR. The current implementation is primarily a quick validation to align ideas, and doesn’t yet cover all finer details.

[malus2077]

I'd love your input on a couple of things.

• Does the full implementation in Record all possible sizes and handle legacy data migration #13274 align with the intended direction of the current PR?
• Specifically regarding legacy data, does my approach feel appropriate? If not, what would be a better approach to handling legacy formats?

I look forward to your thoughts!

[bogdandrutu]

@malus2077 I want to land #13043 first, because we don't need to care about previous saved size, since we only allowed request sizer the backup size is useless and only complicates logic.

[malus2077]

@malus2077 I want to land #13043 first, because we don't need to care about previous saved size, since we only allowed request sizer the backup size is useless and only complicates logic.

I’m concerned about a potential runtime issue after loading old data. Consider this scenario:

1. An old item from the persistent queue is read and processed, but it lacks accurate itemsSize or bytesSize information.
2. When processing completes, onDone() is called with itemsSize=0 and bytesSize=0.
3. As a result, the queue’s internal metadata.ItemsSize and metadata.BytesSize counters are decremented incorrectly, which could eventually lead to an inconsistent state (such as a negative queue size).

What do you think about this potential issue?

[bogdandrutu]

@malus2077 here is the complete implementation, still not saving the sizes, but you have a PR to do the conversion from old to new and save the new format.

[dmitryax]

This approach LGTM. The only concern is that it might be costly to calculate the bytes size for every request even if it's not being used. But I don't have any better solution in mind

[bogdandrutu]

This approach LGTM. The only concern is that it might be costly to calculate the bytes size for every request even if it's not being used. But I don't have any better solution in mind

I have an optimization in mind, if items sizer is used to not call the bytes sizer, but to use len(byte[]) returned since we anyway serialize the value. Will do that in a followup.