Browser Automation for AI Agents: Accessibility Trees Beat Pixels

If you've wired an LLM agent to a browser by taking a screenshot, sending it to a vision model, and asking "where do I click," you've felt the pain: it's slow, it's expensive on tokens, and it's fragile the moment a page re-layouts. Vercel's new agent-browser CLI (a Rust-native tool that shipped as a Claude Code / Cursor / Codex skill in September 2026) and tools like browser-use have converged on a different answer — drive the browser through the accessibility tree, not the pixels. This post explains why that matters and how to build the pattern yourself, even if you never touch agent-browser directly.

The problem: screenshots don't scale

A typical "vision agent" loop looks like this:

Take a full-page screenshot.
Send it to a multimodal model with the task ("click the checkout button").
Model returns pixel coordinates.
Click, screenshot again, repeat.

This works for demos. It breaks down in production for three reasons:

Cost. A 1920×1080 screenshot is 1,000–1,500 tokens depending on the model's tiling scheme. A 20-step task burns 20-30K tokens just on images, before you count the text history you're replaying each turn.
Latency. Vision inference is slower than text-only inference, and you're doing it on every single step.
Precision. Pixel-coordinate clicking fails silently when the page scales, a modal shifts layout by 4px, or the viewport size differs from what the model assumed. The model can "see" a button and still miss it by a few pixels.

Accessibility-tree automation sidesteps all three. Every browser already builds an accessibility tree — the same structure screen readers use — that describes every interactive element as a typed node (button, link, textbox) with a name, role, and state. That tree is:

Text, not pixels — a full page's interactive surface is usually 500-2,000 tokens, not 1,000+ per screenshot.
Stable — a button's accessible name doesn't shift when a stylesheet does.
Directly actionable — the model reasons over "button 'Add to cart', ref e14" and you click e14, no coordinate math.

How it actually works under the hood

The pipeline looks like this:

Connect via CDP (Chrome DevTools Protocol) to a real or headless Chromium instance.
Snapshot the accessibility tree using the Accessibility.getFullAXTree CDP domain, or Playwright's page.accessibility.snapshot() wrapper around it.
Compact and label the tree — strip presentational nodes, collapse whitespace-only text, and assign each interactive node a short reference id (e1, e2, ... — this is what agent-browser calls "@eN refs").
Hand the compacted tree to the LLM as text, alongside the task.
Model responds with an action referencing a node id: click(e14), type(e7, "user@example.com").
Resolve the ref back to a real DOM element via its backend node id and dispatch the action through CDP.
Re-snapshot only what changed where possible (diffing against the previous tree) instead of re-sending the whole page.

Step 7 is the detail that separates a toy implementation from a production one — without diffing, a long multi-step flow re-sends the same 1,500-token tree every turn for no reason.

Minimal implementation with Playwright (TypeScript)

import { chromium, type Page } from "playwright";

interface ActionableNode {
  ref: string;
  role: string;
  name: string;
}

async function snapshotInteractive(page: Page): Promise<ActionableNode[]> {
  const tree = await page.accessibility.snapshot({ interestingOnly: true });
  const nodes: ActionableNode[] = [];
  let counter = 0;

  function walk(node: any) {
    if (!node) return;
    const clickableRoles = ["button", "link", "textbox", "checkbox", "combobox"];
    if (clickableRoles.includes(node.role) && node.name) {
      nodes.push({ ref: `e${counter++}`, role: node.role, name: node.name });
    }
    (node.children || []).forEach(walk);
  }
  walk(tree);
  return nodes;
}

async function clickByName(page: Page, name: string) {
  // Playwright's accessible-name locator does the ref -> element
  // resolution for you; agent-browser does this via backend node ids over CDP.
  await page.getByRole("button", { name }).click();
}

async function run() {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto("https://example-shop.test/cart");

  const nodes = await snapshotInteractive(page);
  console.log(JSON.stringify(nodes, null, 2));
  // -> feed `nodes` as text into your LLM's context alongside the task

  await clickByName(page, "Checkout");
  await browser.close();
}

run();

This is roughly 40 lines and gets you 80% of what a dedicated tool gives you. The remaining 20% — diffing between snapshots, ref stability across re-renders, handling iframes and shadow DOM, retry/backoff on flaky navigation — is exactly the work that agent-browser, browser-use, and Playwright's own MCP server have already done, which is why most teams should reach for one of those instead of hand-rolling this.

Python with Playwright (for agents built on LangGraph, etc.)

from playwright.sync_api import sync_playwright

def get_actionable_nodes(page):
    tree = page.accessibility.snapshot(interesting_only=True)
    nodes = []
    counter = 0

    def walk(node):
        nonlocal counter
        if node is None:
            return
        if node.get("role") in {"button", "link", "textbox", "checkbox"} and node.get("name"):
            nodes.append({"ref": f"e{counter}", "role": node["role"], "name": node["name"]})
            counter += 1
        for child in node.get("children", []) or []:
            walk(child)

    walk(tree)
    return nodes

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example-shop.test/cart")
    nodes = get_actionable_nodes(page)
    print(nodes)
    page.get_by_role("button", name="Checkout").click()
    browser.close()

Both snippets do the same thing: turn the DOM into a compact, LLM-readable list of interactive elements instead of an image.

When you still need vision

Accessibility trees aren't a universal replacement. Fall back to screenshots (or a hybrid: tree first, screenshot only when the tree lookup fails) when:

The target is a <canvas>-based UI (design tools, games, some dashboards) with no accessible structure.
You need to verify visual correctness itself — "does this render broken" is a vision question by definition.
The site actively breaks accessibility semantics (unlabeled div soup with click handlers and no ARIA roles) — sadly common on older enterprise software.

A robust agent tries the tree first and escalates to a screenshot + vision call only on failure. That keeps the common case cheap and fast while keeping a fallback for the messy 10%.

Tool comparison

Tool	Approach	Language	Best for
Playwright (raw)	CDP accessibility snapshot, manual glue	TS / Python / Java	Full control, embedding in an existing test suite or agent framework
`agent-browser` (Vercel)	Rust-native CLI, CDP + compact @eN refs, real Chrome profile support	CLI, any agent	Coding agents (Claude Code, Cursor, Codex) automating a real logged-in browser
`browser-use`	Python library, DOM+AX hybrid extraction, built-in agent loop	Python	Python-first agent stacks, quick prototyping
Puppeteer	CDP, low-level, no built-in AX compaction	JS/TS	Teams already invested in Puppeteer tooling
Computer-use (vision-based)	Screenshot + pixel coordinates	Model-dependent	Canvas UIs, native apps, visual verification

Common pitfalls

Stale refs across re-renders. If you cache e14 and the page re-renders (SPA route change, list re-sort), that ref may now point to a different element or nothing. Re-snapshot after any action that can mutate the DOM, don't reuse refs across turns.
Iframes and shadow DOM silently dropped. Naive accessibility-tree walks stop at frame boundaries. Confirm your tool (or your own walker) recurses into same-origin iframes, and treat cross-origin iframes as a known blind spot.
Treating the tree as ground truth for visual state. A checkbox's accessible checked state can lag a custom-styled toggle's actual visual state if the site's ARIA wiring is buggy. Spot-check against a screenshot when a flow is safety-critical (payments, account deletion).
No rate limiting on retries. Agents love to retry a failed click in a tight loop. Cap retries and back off — sites with bot detection will lock you out fast, and that's a feature, not a bug, of the site.
Running against production without a plan for irreversible actions. Automation that can click "Delete" or "Submit Payment" needs an explicit allowlist of actions that require human confirmation, enforced in code — not left to the model's judgment.

Next steps

If you're building an agent that needs to operate a real website — filling forms, scraping behind auth, running end-to-end checks — start with an existing accessibility-tree tool (agent-browser, browser-use, or Playwright's MCP server) rather than the snippets above; they've already solved ref stability and iframe handling. Reserve vision models for the visual-verification and canvas-UI cases where text truly can't do the job.

If your team is building agents that need to safely operate real interfaces — not just chat — this is exactly the kind of production pattern we cover hands-on in Joinloop's AI engineering cohort.