[BUG] Regression: ValidationException due to orphaned tool_use blocks after removing message processor

[cagataycali]

Description

After PR #417 (refactor: Remove message processor), we're experiencing validation errors when tool results are not completed by Bedrock and the agent receives a message end event, followed by a customer asking another question.

Root Cause

The removal of clean_orphaned_empty_tool_uses(agent.messages) from the event loop cycle has eliminated the cleanup mechanism for orphaned empty tool uses. This function was responsible for handling cases where Bedrock doesn't complete tool results.

Error Details

❌ Error: An error occurred (ValidationException) when calling the ConverseStream operation: The model returned the following 
errors: messages.8: `tool_use` ids were found without `tool_result` blocks immediately after: tooluse_p5c8yrxwSXOO-FOx6CBC_A. 
Each `tool_use` block must have a corresponding `tool_result` block in the next message.

Steps to Reproduce

1. Start an agent conversation
2. Have Bedrock not complete tool results (this happens often)
3. Agent receives message end event from Bedrock
4. Customer asks another question
5. ValidationException occurs due to orphaned tool_use blocks

Expected Behavior

Orphaned empty tool uses should be cleaned up automatically to prevent validation errors, allowing the conversation to continue smoothly.

Actual Behavior

ValidationException is thrown, breaking the conversation flow.

Proposed Solution

Reintroduce the clean_orphaned_empty_tool_uses function or implement an equivalent mechanism to handle orphaned tool_use blocks that don't have corresponding tool_result blocks.

Related PR

• refactor: Remove message processor #417 - refactor: Remove message processor (where the cleanup function was removed)

Impact

• Severity: High - Breaks conversation flow
• Frequency: Often (happens when Bedrock doesn't complete tool results)
• User Experience: Poor - Users encounter validation errors mid-conversation

Additional Context

This is a regression introduced by the message processor removal. The cleanup mechanism was essential for handling Bedrock's inconsistent tool result completion behavior.

[zastrowm]

IIRC this was removed because:

• Post-1.0 we want to support resuming from any point in the lifecycle, so did this before 1.0 so that it wouldn't be a breaking change later
• It wasn't clear that this wasn't left over from when the SDK was geared more towards Phoenix/CLI and thus was needed for something that wasn't necessary for the SDK and having this in may have been masking bugs in the SDK

[zastrowm]

Pinged offline:

Often (happens when Bedrock doesn't complete tool results)
Do we know the cause of this and/or why this happens? Perhaps this should be in the bedrock model provider itself?

Response:

When bedrock doesn't finish the tool inputs and just sends messageEnd event (due to bedrock timeout) tool_use blocks have no tool_results.

[zastrowm]

Suggested way to reproduce:

1. Set boto3 read timeout to a very low value
2. Configure/prompt the agent to write a very large value (like an overly large value).
3. We should then hit the validation error

[zastrowm]

Both myself and cagataycali have attempted to reproduce directly using the SDK but have been unable to as of yet. Doing the above ^ results in a timeout, but not a partial message or a toolUse message without a toolResult

[tpastor]

I can reproduce as well when there is a call to the file_write tool that ends up in max_token. (anything called right after that will end up with the same error). Reverting the mentioned PR indeed solved the issue.

[zastrowm]

Thanks @tpastor, that helped me reproduce:

from strands import Agent
from strands.handlers import CompositeCallbackHandler, PrintingCallbackHandler
from strands.models import BedrockModel

from strands_tools import file_read, file_write

bedrock_model = BedrockModel(
    model_id="us.anthropic.claude-3-7-sonnet-20250219-v1:0",
    temperature=0.3,
    top_p=0.8,
    max_tokens=300,
)

def callback(**kwargs):
    if "event" in kwargs and "contentBlockDelta" in kwargs["event"] and "delta" in kwargs["event"]["contentBlockDelta"] and "toolUse" in kwargs["event"]["contentBlockDelta"]["delta"]:
        print(kwargs["event"]["contentBlockDelta"]["delta"]["toolUse"])

agent = Agent(
    model=bedrock_model,
    tools=[file_read, file_write],
    callback_handler=CompositeCallbackHandler(PrintingCallbackHandler(), callback)
)

result = agent("Write a temporary file that contains a story about rabbits universe.  Make sure that it's a very very long story - longer than Game of thrones")
print(f"Messages: {agent.messages}")
print(f"StopReason: {result.stop_reason}")

Which results in the following output:

Tool #1: file_write
{'input': '{"pat'}
{'input': 'h"'}
{'input': ': "/tmp/'}
{'input': 'rabbits_un'}
{'input': 'iv'}
{'input': 'erse_'}
{'input': 'epic.txt"'}
Messages: [{'role': 'user', 'content': [{'text': "Write a temporary file that contains a story about rabbits universe.  Make sure that it's a very very long story - longer than Game of thrones"}]}, {'role': 'assistant', 'content': [{'text': "I'll help you write a very long story about a rabbit universe to a temporary file. This will be an extensive narrative, even longer than Game of Thrones (which spans millions of words across multiple books).\n\nLet me create this temporary file for you:"}, {'toolUse': {'toolUseId': 'tooluse_6lgmhUrZTEemDaVWGrprMA', 'name': 'file_write', 'input': {}}}]}]
StopReason: max_tokens

which shows the orphaned tool_use message

[zastrowm]

Will have to consult with the team on what to do here - we specifically want to support resuming from tool_use messages and I'm not sure we should be removing these messages automatically as wanting to remove the message seems like an application-level decision.

[zastrowm]

If you want to restore the old behavior where tool uses are automatically removed, you can use a hook:

logger = logging.getLogger(__name__)

class RemoveOrphanedToolUses(HookProvider):
    def register_hooks(self, registry: "HookRegistry", **kwargs: Any) -> None:
        registry.add_callback(BeforeInvocationEvent, self._remove_tool_uses)

    def _remove_tool_uses(self, event: BeforeInvocationEvent):
        messages = event.agent.messages

        if not messages:
            return

        # Dictionary to track empty toolUse entries: {tool_id: (msg_index, content_index, tool_name)}
        empty_tool_uses: dict[str, tuple[int, int, str]] = {}

        # Set to track toolResults that have been seen
        tool_results: set[str] = set()

        # Identify empty toolUse entries
        for i, msg in enumerate(messages):
            if msg.get("role") != "assistant":
                continue

            for j, content in enumerate(msg.get("content", [])):
                if isinstance(content, dict) and "toolUse" in content:
                    tool_use = content.get("toolUse", {})
                    tool_id = tool_use.get("toolUseId")
                    tool_input = tool_use.get("input", {})
                    tool_name = tool_use.get("name", "unknown tool")

                    # Check if this is an empty toolUse
                    if tool_id and (not tool_input or tool_input == {}):
                        empty_tool_uses[tool_id] = (i, j, tool_name)

        # Identify toolResults
        for msg in messages:
            if msg.get("role") != "user":
                continue

            for content in msg.get("content", []):
                if isinstance(content, dict) and "toolResult" in content:
                    tool_result = content.get("toolResult", {})
                    tool_id = tool_result.get("toolUseId")
                    if tool_id:
                        tool_results.add(tool_id)

        # Filter for orphaned empty toolUses (no corresponding toolResult)
        orphaned_tool_uses = {tool_id: info for tool_id, info in empty_tool_uses.items() if tool_id not in tool_results}

        # Apply fixes in reverse order of occurrence (to avoid index shifting)
        if not orphaned_tool_uses:
            return

        # Sort by message index and content index in reverse order
        sorted_orphaned = sorted(orphaned_tool_uses.items(), key=lambda x: (x[1][0], x[1][1]), reverse=True)

        # Apply fixes
        for tool_id, (msg_idx, content_idx, tool_name) in sorted_orphaned:
            logger.debug(
                "tool_name=<%s>, tool_id=<%s>, message_index=<%s>, content_index=<%s> "
                "fixing orphaned empty tool use at message index",
                tool_name,
                tool_id,
                msg_idx,
                content_idx,
            )
            try:
                # Check if this is the sole content in the message
                if len(messages[msg_idx]["content"]) == 1:
                    # Replace with a message indicating the attempted tool
                    messages[msg_idx]["content"] = [{"text": f"[Attempted to use {tool_name}, but operation was canceled]"}]
                    logger.debug("message_index=<%s> | replaced content with context message", msg_idx)
                else:
                    # Simply remove the orphaned toolUse entry
                    messages[msg_idx]["content"].pop(content_idx)
                    logger.debug(
                        "message_index=<%s>, content_index=<%s> | removed content item from message", msg_idx, content_idx
                    )
            except Exception as e:
                logger.warning("failed to fix orphaned tool use | %s", e)

        return

agent = Agent(
    model=bedrock_model,
    tools=[file_read, file_write],
    hooks=[RemoveOrphanedToolUses()],
)

result = agent("Write a temporary file that contains a story about rabbits universe.  Make sure that it's a very very long story - longer than Game of thrones")
agent("Did it work?")

which results in no exception, but makes the agent not understand what happened:

I'll write a temporary file with a very long story about a rabbit universe. Since you want it to be longer than Game of Thrones (which is several million words across multiple books), I'll create a substantial story for you.
Tool #1: file_write
I apologize for the confusion. I didn't actually write the file yet. Let me create that temporary file with a very long story about a rabbit universe for you now.
Tool #2: file_write

[zastrowm]

Team has had some discussions around this. The general thoughts are:

• We know we want some change in behavior here:

  • A. We don't think silently removing the failed ToolUse message is the right default
  • B. We're not sure that from within the agent-loop that we can apply some general purpose behavior that is appropriate to cover all use case. Specifically we think different customers will want different behaviors.
  • C. We also don't think that simply returning stop_reason=max_tokens makes it clear to customers what is going on (for example [FEATURE] use_aws - Expose stopReason to the client #520)

• Due to B & C we want to make surface this more clearly to callers. We're leaning towards making this an exception that callers should handle. However we'd also like to have a mechanism where you can specify a strategy to use here that allows the agent to continue running autonomously but not via a hook (the hook above is a workaround, not an intended solution).

It's also worth calling that while we're currently focused on the stop_reason=max_token use case here, there are also other use-cases that may be similar; e.g. read-timeouts even when max_tokens=65k.

[dbschmigelski]

A discussion post has been raised to request community input on their desired solution #541

[dbschmigelski]

Strands is going to move forward with the following solution:

We will be consistent with other failure types and fail hard by default with a MaxTokensReachedException.

To allow agents to recover without terminating the event loop a new Hook event will be added. This event will be triggered when max tokens is reached where the result of the hook will determine if we proceed or of if the exception is re-thrown.

Finally, to aid customers, an implementation of the HookProvider will be added to the SDK to solve this use case in a manner which we believe will work for the majority of use cases. The implementation will do the following:

1. Purge the invalid ToolUse message from the messages array
2. Insert a new message indicating that the previous call failed with max_tokens
   The idea here is that on the next iteration of the cycle, the max_tokens message will bias the agent to use or not avoid certain problematic tools.