Skip to content

[FEATURE] Add cached token metrics support to Usage class #529

@oaltagar-aws

Description

@oaltagar-aws

Problem Statement

The current Usage class in strands/types/event_loop.py only tracks basic token metrics (inputTokens, outputTokens, totalTokens). However, Amazon Bedrock's converse API now returns additional cached token metrics (cacheReadInputTokens and cacheWriteInputTokens) that provide valuable insights into:

  • Caching efficiency and performance optimization
  • Cost savings from prompt/tool caching
  • Cache hit rates for better observability

Currently, these metrics are lost and users cannot track the effectiveness of their caching strategies.

Proposed Solution

Extend the Usage TypedDict to include optional cached token fields while maintaining 100% backward compatibility:

class Usage(TypedDict, total=False):
    inputTokens: Required[int]
    outputTokens: Required[int] 
    totalTokens: Required[int]
    cacheReadInputTokens: Optional[int]  # NEW - tokens read from cache
    cacheWriteInputTokens: Optional[int] # NEW - tokens written to cache

This follows the existing project pattern used in AnthropicConfig with Required/Optional annotations.

Use Case

  1. Cost Optimization: Users can measure actual cost savings from caching
  2. Performance Monitoring: Track cache hit rates to optimize prompt/tool design
  3. Debugging: Understanding why caching isn't working as expected
  4. Observability: EventLoopMetrics can display cache efficiency in summaries

Example usage:

agent = Agent(model=BedrockModel(cache_prompt="default"))
response = agent("Complex prompt...")
# Now users can see: "Cache: read=150 tokens, write=50 tokens, efficiency=75%"

Alternatives Solutions

  1. Create separate CachedUsage class - but this breaks consistency
  2. Add to Metrics class - but Usage is the logical place for token counts
  3. Model-specific extensions - but Usage should be provider-agnostic

Additional Context

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions