Bug: Duplicate observations for same tool_call_id after crash recovery

[csmith49]

Summary

When a server crashes during tool execution and restarts, crash recovery emits an AgentErrorEvent. However, if run() is called again, the action is re-executed, resulting in both an AgentErrorEvent and an ObservationEvent for the same tool_call_id.

Root Cause

get_unmatched_actions() in state.py:454 only checks for ObservationEvent and UserRejectObservation:

if isinstance(event, (ObservationEvent, UserRejectObservation)):
    observed_action_ids.add(event.action_id)

It does NOT check for AgentErrorEvent, so the action remains "unmatched" even after crash recovery emits an error for it.

Complete Control Flow

1. ActionEvent created (tool_call_id=X)
2. Server crashes during tool execution
3. On restart, event_service.start() detects execution_status==RUNNING
4. Crash recovery emits AgentErrorEvent (tool_call_id=X) - event_service.py:479-488
5. Crash recovery sets execution_status=ERROR
6. User calls run() again
7. run() allows ERROR status to proceed - local_conversation.py:549-554:

   if self._state.execution_status in [IDLE, PAUSED, ERROR]:
       self._state.execution_status = RUNNING

8. agent.step() calls get_unmatched_actions() which returns the action (because AgentErrorEvent is not checked)
9. agent.step() calls _execute_actions() on the "pending" action - agent.py:264-271
10. Tool executes and emits ObservationEvent (tool_call_id=X)
11. Result: BOTH AgentErrorEvent AND ObservationEvent for same tool_call_id

Potential Fixes

1. Add AgentErrorEvent to get_unmatched_actions() - But AgentErrorEvent does not have action_id, only tool_call_id. Would need to match by tool_call_id instead, or add action_id to AgentErrorEvent.

2. Change crash recovery to NOT allow re-execution - Either use a different status that blocks run(), or do not emit AgentErrorEvent at all.

3. Make get_unmatched_actions() also check AgentErrorEvent by tool_call_id - This would be a behavior change but might be the cleanest fix.

Related Code

• openhands-agent-server/openhands/agent_server/event_service.py:470-488 - Crash recovery
• openhands-sdk/openhands/sdk/conversation/state.py:450-462 - get_unmatched_actions()
• openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py:549-554 - ERROR status handling
• openhands-sdk/openhands/sdk/agent/agent.py:264-271 - Pending action execution

[csmith49]

See #1782, #2256, and #2204. This issue is just to get the failing control flow into the conversation along with some potential fixes (that might be easier than what I've already linked).

[csmith49]

@OpenHands Open a PR that implements the third fix. Make sure you add tests to capture the failing control flow before implementing the fix, and verify that it indeed does resolve the issue.

[openhands-ai]

I'm on it! csmith49 can track my progress at all-hands.dev

[openhands-ai]

Final Summary

I have successfully completed the task to fix issue #2298: "Bug: Duplicate observations for same tool_call_id after crash recovery".

Checklist

✅ Tests written first to capture the bug - Created 9 tests in tests/sdk/conversation/test_get_unmatched_actions.py that initially failed to demonstrate the bug exists

✅ Fix implemented - Modified get_unmatched_actions() in openhands-sdk/openhands/sdk/conversation/state.py to also check for AgentErrorEvent by tool_call_id

✅ Tests pass after fix - All 9 new tests pass, plus all existing related tests (confirmation mode and event service tests)

✅ Pre-commit hooks pass - Ruff, pyright, and other linters pass on all modified files

✅ Branch created with proper prefix - openhands/fix-duplicate-observations-after-crash-recovery

✅ Changes committed with Co-authored-by - Commit includes proper attribution

✅ Branch pushed to GitHub - Changes are on the remote

✅ PR created - #2300

• PR mentions "Fixes Bug: Duplicate observations for same tool_call_id after crash recovery #2298"
• PR includes detailed description of the problem, root cause, and solution

Changes Made

2 files changed:

1. openhands-sdk/openhands/sdk/conversation/state.py - Added AgentErrorEvent handling to get_unmatched_actions()
2. tests/sdk/conversation/test_get_unmatched_actions.py - New test file with 9 tests covering the fix

The changes are concise and directly address the issue without any extraneous modifications.