-
Notifications
You must be signed in to change notification settings - Fork 283
Open
Description
Problem Statement
The current Usage
class in strands/types/event_loop.py
only tracks basic token metrics (inputTokens
, outputTokens
, totalTokens
). However, Amazon Bedrock's converse API now returns additional cached token metrics (cacheReadInputTokens
and cacheWriteInputTokens
) that provide valuable insights into:
- Caching efficiency and performance optimization
- Cost savings from prompt/tool caching
- Cache hit rates for better observability
Currently, these metrics are lost and users cannot track the effectiveness of their caching strategies.
Proposed Solution
Extend the Usage
TypedDict to include optional cached token fields while maintaining 100% backward compatibility:
class Usage(TypedDict, total=False):
inputTokens: Required[int]
outputTokens: Required[int]
totalTokens: Required[int]
cacheReadInputTokens: Optional[int] # NEW - tokens read from cache
cacheWriteInputTokens: Optional[int] # NEW - tokens written to cache
This follows the existing project pattern used in AnthropicConfig with Required/Optional annotations.
Use Case
- Cost Optimization: Users can measure actual cost savings from caching
- Performance Monitoring: Track cache hit rates to optimize prompt/tool design
- Debugging: Understanding why caching isn't working as expected
- Observability: EventLoopMetrics can display cache efficiency in summaries
Example usage:
agent = Agent(model=BedrockModel(cache_prompt="default"))
response = agent("Complex prompt...")
# Now users can see: "Cache: read=150 tokens, write=50 tokens, efficiency=75%"
Alternatives Solutions
- Create separate CachedUsage class - but this breaks consistency
- Add to Metrics class - but Usage is the logical place for token counts
- Model-specific extensions - but Usage should be provider-agnostic
Additional Context
- Amazon Bedrock documentation: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
- Bedrock API already returns these fields in converse response
- Implementation is backward compatible - no breaking changes
- Follows existing TypedDict patterns in the codebase
- Ready to implement following contribution guidelines
cliren
Metadata
Metadata
Assignees
Labels
No labels