[STG-1561] Correctly remap ax-psuedo-nodes to nearest real element before attempting to using them

[pirate]

why

Sometimes the model can try to select a "pseudo" element that only exists in the a11y tree but has no direct DOM element. This throws a confusing error for users:

[
  {
    "method": "not-supported",
    "selector": "not-supported",
    "arguments": [],
    "description": "an element inside a shadow DOM"
  }
]

This error is actualy incorrect for this case, and we can do better by auto-resolving to the correct element.

what changed

1. Build the AX tree as today.
2. For any AX node without backendDOMNodeId, resolve a surrogate target node.
3. Assign that node’s encodedId to the unresolved AX node before formatting the tree.
4. Let the rest of observe() continue unchanged.

That means the model still only ever sees ids like 0-123, and packages/core/lib/v3/handlers/observeHandler.ts can keep doing the same combinedXpathMap lookup.

How to choose the surrogate target
Use the closest intended DOM-backed node in the AX tree, with a strict preference order:

• Exact node if it already has backendDOMNodeId
• Closest DOM-backed descendant
• Closest DOM-backed ancestor

Within the same distance, prefer:

• Actionable roles: button, link, checkbox, radio, textbox, combobox, menuitem, tab, option
• Matching accessible name
• Matching role
• Same subtree branch, not siblings

That handles the two common cases:

• Virtual wrapper/container AX node around a real control: choose descendant
• Virtual text/label AX node inside a real control: choose ancestor

test plan

• packages/core/node_modules/.bin/vitest run packages/core/tests/unit/snapshot-a11y-tree-utils.test.ts --environment node
• pnpm -w exec tsc -p packages/core/tsconfig.json --noEmit

---

Summary by cubic

Remaps AX pseudo/virtual nodes to the nearest DOM-backed element so observe/act always target a real element. Fixes the incorrect "not-supported" error and keeps encodedId semantics, addressing STG-1561.

• Bug Fixes

  • Maps AX nodes without backendDOMNodeId to the closest DOM-backed descendant, else ancestor; prefers actionable roles and elements with accessible names.
  • Assigns the surrogate’s encodedId to unresolved nodes to preserve the observe flow.
  • Adds unit tests for descendant/ancestor proxy cases and updates test helpers to allow undefined backendDOMNodeId.

• Refactors

  • Linting and minor formatting only.

^{Written for commit 739b81e. Summary will update on new commits. Review in cubic}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 739b81e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

No issues found across 2 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Model as AI Model
    participant Core as Stagehand Core
    participant CDP as Browser (CDP)
    participant A11y as a11yTree Logic

    Note over Model, A11y: Execution Flow for observe() / act()

    Core->>CDP: Fetch raw AXTree nodes
    CDP-->>Core: Array of AXNode (backendDOMNodeId might be null)

    Core->>A11y: decorateRoles(nodes)
    
    loop For each AXNode
        alt Has backendDOMNodeId
            A11y->>A11y: Generate standard encodedId
        else NEW: Missing backendDOMNodeId (Pseudo-node)
            A11y->>A11y: resolveMissingEncodedIds()
            
            Note over A11y: Search Strategy: Descendants first, then Ancestors
            
            opt Search Descendants (BFS)
                A11y->>A11y: Find DOM-backed nodes in subtree
                alt Multiple candidates found
                    A11y->>A11y: NEW: Score based on role (button, link, etc.) and name
                end
            end
            
            alt No descendant found
                A11y->>A11y: NEW: Traverse up parentId chain for nearest DOM-backed ancestor
            end

            A11y->>A11y: CHANGED: Map surrogate encodedId to this node
        end
    end

    A11y-->>Core: Decorated nodes (All nodes now have valid encodedId)
    Core->>Core: Build Hierarchical Tree
    Core-->>Model: Return A11y Tree Snapshot

    Note over Model, Core: Action Resolution
    Model->>Core: act(target_id)
    Note right of Core: target_id (surrogate) now maps correctly<br/>to a real DOM element in the XpathMap.

Loading

[seanmcguire12]

@pirate i did a bit of testing on some HTML snippets with pseudo AX nodes (AX nodes that dont have a backendDomNodeId), and found out that their AX node ID seems to always be a negative number. nodes with negative IDs already get filtered out, so in theory we should not run into the scenario where we attempt to look up an xpath for a pure AX node.

i think we can close this one. if you are worried about running into this scenario, i think a better approach would be just remove the fallback logic which allows AX node IDs to get into the stringified tree. that way, the LLM will never be able to choose a node that we don't have a computed xpath for