> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myrmagent.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Browser Automation

> Let your agent browse the web and automate browser tasks.

# Browser Automation

Myrm agents can autonomously browse the web, fill forms, extract data, and perform actions.

## Browser Engine

**Dual stealth engines (Patchright + Camoufox)** — When a site blocks the default Chromium path, Myrm can hot-swap to a Firefox-based stealth engine **without losing your login cookies**, so authenticated workflows do not bounce back to a sign-in page.

Three-layer interaction stack:

1. **Headless Chrome / Camoufox** — Full browser rendering with anti-bot fallback
2. **DOM Parser** — Structured content extraction (self-healing locators, Shadow DOM)
3. **Vision** — Screenshot-based interaction for complex UIs

### 12-Layer Anti-Detection Stealth

Myrm injects a comprehensive anti-detection system that makes automated browsing indistinguishable from real users. Verified against 5 major anti-bot systems (BrowserScan, Fingerprint.com, CreepJS, Cloudflare, DataDome) and production-tested on 10+ high-risk platforms including WeChat, Weibo, Zhihu, Xiaohongshu, Douyin, Twitter/X, Instagram, and Facebook.

* **CDP leak patches** — Patchright eliminates `Runtime.enable`, `Console.enable`, and `--enable-automation` detection vectors at the protocol level
* **13 JS stealth scripts** — `navigator.webdriver`, `window.chrome` stub, plugin/language emulation, automation artifact cleanup, `toString()` disguise via WeakMap, anti-debugger neutralization, Performance API cleanup, and more
* **Humanized interaction** — Random typing delay (30–100ms/char) and click delay (50–150ms) with dynamic timeout adaptation
* **Proxy rotation & self-healing** — Automatic bad-proxy isolation, context hot-swap for fresh IPs, and stateful session migration (Cookie + localStorage preserved across engine switches)

### Why This Matters for Users

| Without anti-detection                 | With Myrm's Stealth              |
| -------------------------------------- | -------------------------------- |
| Blocked on first page load             | Passes Cloudflare and DataDome   |
| Login sessions revoked by platform     | Stays logged in across sessions  |
| Must use expensive residential proxies | Works with standard proxies      |
| Manual browser profile management      | Automatic fingerprint management |

## Page Understanding — Dual-Path ARIA Snapshot

Myrm uses a **semantic accessibility tree** instead of raw DOM for page understanding. This means your agent sees the page the way a screen reader does — structured, noise-free, and focused on interactive elements.

### How It Works

| Path            | When Used          | Speed   | Method                                                 |
| --------------- | ------------------ | ------- | ------------------------------------------------------ |
| **Fast Path**   | 90% of pages       | 5-10ms  | Playwright native `ariaSnapshot()` — zero JS injection |
| **Custom Path** | Complex edge cases | 20-50ms | JavaScript traversal with depth control                |

### What the Agent Sees

Instead of thousands of DOM nodes, the agent gets a clean semantic tree:

```yaml theme={null}
- heading "Sign In" [level=1]
- textbox "Email" [ref=1]
- textbox "Password" [ref=2]
- button "Log In" [ref=3]
- link "Forgot password?" [ref=4]
```

Every interactive element gets a stable **ref ID** that the agent uses for precise targeting — no fragile CSS selectors.

### Built-in Resilience

* **Shadow DOM traversal** — Sees through Web Components automatically
* **iframe extraction** — Cross-origin iframes included in the tree
* **Scope filtering** — `full`, `interactive`, or `focused` views for token efficiency
* **Incremental diff** — Only shows what changed between snapshots

### Why Not Occlusion Detection?

Some tools (like OpenCLI) scan every element with `elementFromPoint` to prune occluded elements. We deliberately don't do this because:

1. It adds 100-500ms per snapshot (reflow for every element)
2. It can accidentally prune valid elements (dropdown menus, tooltips)
3. Playwright's `click()` already refuses to click covered elements
4. If a click fails, our **Self-Healing Locator** fixes it in O(1) time

Net result: 5-10ms snapshots that are 60-80% smaller than DOM-based approaches, with zero accuracy trade-offs.

## Capabilities

* Navigate to URLs
* Click elements and fill forms
* Extract text and structured data
* Take screenshots
* Handle authentication flows
* Handle browser dialogs intelligently

## Dialog Policy (Handling Popups Automatically)

JavaScript dialogs (`alert`, `confirm`, `prompt`, `beforeunload`) can block all page interaction if not handled. Myrm provides **four configurable policies** per agent:

| Policy              | Behavior                                        | Best for                   |
| ------------------- | ----------------------------------------------- | -------------------------- |
| **Smart** (default) | Accept alerts/confirms, dismiss prompts         | General browsing           |
| **Auto Accept**     | Accept all dialogs                              | Bulk automation, scraping  |
| **Auto Dismiss**    | Dismiss all dialogs                             | Data extraction, read-only |
| **Wait for Agent**  | Pause and let AI decide (with timeout fallback) | Critical confirmations     |

### Configuration

Set the dialog policy per agent in **Settings → Agents → Capability Config → Dialog Handling**.

The agent can also switch policy at runtime using the `dialog_policy` action in `browser_manage`.

### How It Works

* In **Smart** mode (default), your agent never gets stuck on cookie banners or info alerts.
* In **Wait for Agent** mode, the dialog content appears in the next browser snapshot, giving the AI full context to decide. If the agent doesn't respond within the timeout, a safe fallback fires automatically.
* Dialog history is always visible to the agent — it knows what popups were handled and how.

## Credential Vault (Login Without Leaking Passwords)

For sites and apps that require login, configure **Form Credential Vault** entries in **Settings → Credentials**:

1. Add a **label** (e.g. `company-admin`), password, and optional TOTP seed.
2. Tell the agent to sign in — it references the label only.
3. Myrm injects credentials at the browser DOM or desktop input layer; values never appear in the conversation.

**Why it matters:** Tool arguments and chat history are durable. Passing `"hunter2"` through `type` or `fill` copies the password across logs, retries, and context. Label-based injection keeps secrets in the vault boundary — the same principle used by research-grade browser vault designs, extended here to desktop Computer Use and product GUI.

## Visual Verification (Auto QA)

Every browser action can be automatically verified — no manual inspection needed.

Pass a `verify_goal` parameter when clicking or navigating, and Myrm's **3-layer verification funnel** kicks in:

1. **dHash Check** (\~2ms) — Perceptual hash detects if the screen changed at all. Skips the LLM call if nothing changed, saving tokens.
2. **Pixel-Level Diff** — Canvas API comparison with YIQ color space and anti-aliasing detection for precise change localization.
3. **Vision LLM Scoring** — Multimodal AI scores the result 1–5 with reasoning, confirming if your goal was met.

### Example

Ask the agent: *"Click 'Add to Cart' and verify the cart count shows 2"*

The agent uses `verify_goal="Cart count shows 2"`. After clicking, the system automatically takes a screenshot and verifies the result. If the score is below 4, the agent gets feedback to retry.

### Privacy

Password fields are **automatically masked** in screenshots — the Vision LLM never sees sensitive credentials.

## Search Integration

7-intent search system understands different search needs:

* Factual queries
* Navigation requests
* Research tasks
* Price comparison
* News and updates
* Image search
* Local search

## Computer Use

Desktop automation via Computer Use protocol for native application control across macOS, Windows, and Linux.

## 8-Layer Visual Tracing System

When Agent operates browsers or desktop apps, you never have to guess what it's doing — an 8-layer visual system shows you everything in real time.

### What You See

| Layer               | What It Does                                     | What You Get                                                                                                            |
| ------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| **BrowserLiveView** | Draggable side panel with live screenshots       | See exactly what the browser page looks like, with 14-role BBox element overlay you can click to select                 |
| **DesktopLiveView** | Mirror panel for desktop apps                    | Same real-time preview for native application operations                                                                |
| **ElementOverlay**  | Interactive BBox highlights                      | Buttons, links, inputs, checkboxes, dropdowns — all highlighted and selectable with role+name labels                    |
| **OS Overlay**      | System-level red highlight frame (Tauri desktop) | A red border appears on your **real display** — visible even when the chat window is covered                            |
| **ProgressSteps**   | Tree-shaped step tracker                         | Tool icons, Agent badges, duration, progress bar, error diagnostics, recovery buttons, and live terminal output         |
| **HITL Approval**   | Approve/Deny with payload editing                | Review operations before they execute, edit parameters, compare code diffs in Monaco Editor                             |
| **VNC Takeover**    | Remote browser control                           | When Agent gets stuck, VNC opens automatically so you can take over and fix the issue                                   |
| **SSE Events**      | 10+ real-time event types                        | browser\_view\_update, desktop\_view\_update, locator\_self\_healed, captcha states, tool lifecycle — all streamed live |

### How It Compares

No competing product (OpenClaw, CoPaw, PilotDeck, Hermes, Claude Computer Use, Codex App) offers any of these visual tracing capabilities. Most competitors provide text-only logs or simple progress bars.

### Test Coverage

196 tests across Rust (Tauri OS overlay), Python (harness events, takeover, CAPTCHA), and TypeScript (frontend ProgressSteps, SSE handlers) — all passing.

## Schema-Driven Structured Extraction

Instead of dumping raw page text into the conversation (which wastes context tokens), you can pass a **JSON Schema** to the extraction tool and get back validated, structured JSON.

### How it works

1. Agent calls `browser_extract` with an `extraction_schema` parameter (a JSON Schema string).
2. The tool extracts the page text, then uses an LLM to produce structured output matching your schema.
3. You receive clean, validated JSON — not thousands of tokens of raw HTML.

### Example

Ask the agent: *"Extract all product names and prices from this page"*

The agent uses a schema like:

```json theme={null}
{
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "price": { "type": "string" }
    },
    "required": ["name", "price"]
  }
}
```

Result: a clean JSON array with just the data you need.

### Key Features

* **Object and Array schemas** — Extract a single object or a list of items.
* **Pagination deduplication** — Pass `already_collected` to avoid duplicates across pages.
* **Dual-strategy reliability** — Preferred `structured_output` with raw JSON fallback for model compatibility.
* **Token savings** — 70–97% reduction compared to raw text extraction.
* **Schema complexity protection** — Limits on property count and nesting depth prevent misuse.

## Ephemeral Session Isolation

Every browser context is **ephemeral by default** — created via `browser.new_context()` with no persistent profile. Cookies, localStorage, and cache are automatically destroyed when the session ends. The agent never has access to your real browser profile, eliminating session leakage risk entirely.

## Persistent Session Vault (Cross-Session Login)

While sessions are ephemeral by default, you can configure **auto-restore domains** for each agent to preserve login state across conversations.

### How It Works

1. **Configure domains** — In the agent settings WebUI, add domains like `twitter.com`, `github.com` to the agent's `auto_restore_domains` list.
2. **Sessions are encrypted** — Cookie and storage data saved via AES-256-GCM encryption in the Session Vault.
3. **Auto-restore on startup** — Every time the agent launches a new browser session, configured domains have their login state automatically restored.
4. **Agent can manage explicitly** — The agent can also call `browser_manage(action="save_session")` / `restore_session` / `list_sessions` / `delete_session` for fine-grained control.

### Architecture

* **AES-256-GCM encryption** — Session data is encrypted at rest, never stored in plaintext.
* **O(1) LRU memory cache** — Hot sessions served from memory; no disk I/O on repeated access.
* **Singleflight dedup** — Concurrent requests for the same domain are coalesced into one decrypt operation.
* **TTL auto-expiry** — Stale sessions are garbage-collected automatically.
* **Full execution coverage** — Works across Web, Channel, Cron, and Eval execution contexts.

### When to Use

| Scenario                               | Recommendation                                          |
| -------------------------------------- | ------------------------------------------------------- |
| One-off browsing                       | Default (ephemeral) — no config needed                  |
| Repeated access to authenticated sites | Add to `auto_restore_domains`                           |
| Agent workflow needs login mid-task    | Agent calls `save_session` / `restore_session`          |
| Shared credentials across agents       | Each agent configures its own domain list independently |

### Session–Memory Bridge (Zero-Cost Awareness)

When the memory system is enabled, saved browser sessions are **automatically visible** in the agent's context — no extra tool call needed.

* **How it works:** Every time a session is saved or deleted, a `SessionMemoryBridge` updates the agent's `active_browser_sessions` profile attribute. This attribute is injected into every LLM turn via the memory context middleware.
* **Result:** When a user says "post a tweet", the agent already knows `twitter.com` is available — it calls `restore_session("twitter.com")` directly, **skipping the `list_sessions` step** and saving one LLM inference round.
* **Prompt cache safe:** The profile attribute only changes on save/delete (very infrequent), so prompt prefix caching is unaffected.

| Without Bridge                                                                                  | With Bridge                                                                                 |
| ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| User: "Post a tweet" → Agent calls `list_sessions` → sees twitter.com → calls `restore_session` | User: "Post a tweet" → Agent sees twitter.com in context → calls `restore_session` directly |
| 2 tool calls + 2 LLM inferences                                                                 | 1 tool call + 1 LLM inference                                                               |

## CAPTCHA Detection & Human Takeover

When the agent encounters a CAPTCHA or situation requiring human intervention, it handles it seamlessly:

### Automatic CAPTCHA Detection

10 CAPTCHA types are detected automatically (Cloudflare Turnstile, reCAPTCHA, hCaptcha, etc.). When detected:

1. Agent pauses execution via LangGraph `interrupt()`
2. The VNC panel auto-opens showing the browser
3. A notification appears with the reason for the pause
4. You solve the CAPTCHA manually
5. Agent resumes automatically and continues the task

### Agent-Triggered Human Takeover

For scenarios beyond CAPTCHAs — 2FA prompts, payment gateways, proprietary forms — the agent can proactively request your help:

1. Agent recognizes it cannot proceed (e.g., "this requires SMS verification")
2. Calls `browser_ask_human` with a clear explanation
3. VNC panel opens with the reason displayed
4. You complete the action directly in the browser
5. Click "Done" — agent captures the new page state and continues

### Why This Matters

| Without Myrm                    | With Myrm                                 |
| ------------------------------- | ----------------------------------------- |
| Agent loops forever on CAPTCHA  | Auto-detects, pauses, notifies            |
| Agent fails silently on 2FA     | Proactively asks for help                 |
| User must monitor constantly    | Only interrupts when truly needed         |
| Lost context after intervention | Captures post-human page state and adapts |

## Browser Extension Bridge

For scenarios where you need the agent to operate within your **real browser** — with your existing login sessions, extensions, and browsing context intact — Myrm supports a Chrome Extension bridge.

### When to Use

| Scenario                                           | Best Approach                       |
| -------------------------------------------------- | ----------------------------------- |
| Fresh automated browsing                           | Default (ephemeral engine)          |
| Re-use specific login sessions                     | SessionVault `auto_restore_domains` |
| Need full browser fingerprint consistency          | **Extension Bridge**                |
| Agent must see your real tabs/bookmarks/extensions | **Extension Bridge**                |
| SaaS mode controlling a remote user's browser      | **Extension Bridge**                |

### How It Works

1. **Install the Myrm Extension** — A lightweight Chrome MV3 extension that maintains a WebSocket connection to the Myrm server.
2. **Authorize domains** — In Settings → Integration → Extension Bridge, add domains you want the agent to control (e.g., `github.com`, `*.google.com`). Wildcard patterns are supported.
3. **Agent connects** — When a task requires browser access to an authorized domain, the agent connects to your real browser via CDP (Chrome DevTools Protocol) proxy.

### Security Model

* **Domain-level authorization** — Only explicitly authorized domains can be controlled. The agent cannot access any tab outside your allow-list.
* **Token authentication** — The extension authenticates to the server with a secret token, preventing unauthorized connections.
* **Service Worker keepalive** — The MV3 extension uses `chrome.alarms` for persistent connection with exponential backoff reconnection.
* **Domain sync on reconnect** — When the extension reconnects, it immediately synchronizes authorized domains with the server.

### Frontend Management

The Extension Bridge settings panel (Settings → Integration) provides:

* **One-click copyable WebSocket URL** — The correct connection URL for your deployment mode, with a Copy button and toast confirmation
* **Auth token status** — Shows whether `EXTENSION_AUTH_TOKEN` is configured on the server (without exposing the token itself)
* **Copyable extension path** — Shows `~/.myrm/myrm-agent/myrm-agent-extension` with one-click copy for Chrome's "Load unpacked" dialog
* **Step-by-step setup guide** — Appears when extension is not connected: Load unpacked extension → Copy URL → Connect
* **Real-time connection status** — Connected/disconnected indicator with extension version and browser name
* **Domain authorization management** — Add, remove, and wildcard pattern support
* **Available tabs list** — Filtered by authorized domains, shows active tab indicator

### Architecture

The Extension Bridge spans 5 layers:

1. **Harness Protocol** (`ExtensionBridge`) — Defines the contract at the framework level
2. **Server Service** (`ExtensionBridgeService`) — Manages WebSocket lifecycle, heartbeat, and CDP proxy
3. **API Router** — WebSocket + REST endpoints for extension and frontend
4. **Chrome Extension** — MV3 Service Worker with CDP attachment capability
5. **Frontend Panel** — React settings component for status and domain management

## Per-Agent Browser Source Configuration

Each agent can independently choose how it accesses the browser. This is configured in the agent settings (both during agent creation and in the per-chat configuration panel).

### Available Modes

| Mode               | Behavior                                        | Best For                                             |
| ------------------ | ----------------------------------------------- | ---------------------------------------------------- |
| **Auto** (default) | System decides based on availability            | General use — no manual setup needed                 |
| **Extension**      | Uses your real browser via the Extension Bridge | Shopping, social media, any task needing login state |
| **Launch**         | Spins up a fresh managed browser instance       | Research, data extraction, testing                   |

### How to Configure

1. **Agent Creation**: Settings → Agents → Create/Edit → Browser Source
2. **Per-Chat Override**: Click the agent config icon in the chat window → Browser Source

### Pool Isolation

When agents use different browser sources, the browser pool maintains strict isolation:

* Extension-sourced browsers (unmanaged) are never shared with launch-mode agents
* Launch-sourced browsers (managed) are never shared with extension-mode agents
* This prevents session cross-contamination between agents with different trust levels

## Zero-Config Browser Setup

When you first use browser tools on a fresh Myrm desktop install, the browser engine (Chromium) is automatically downloaded and installed — **no terminal commands required**.

### How It Works

1. Agent requests a browser page for a task
2. If Chromium is not installed, Myrm detects the missing executable
3. Automatic download and installation runs in the background
4. The browser task continues seamlessly after installation

If automatic installation fails (no network, insufficient disk space, etc.), Myrm provides clear diagnostic messages with specific fix instructions.

### Smart Failure Handling

* **Cooldown protection** — After a failed install, Myrm waits 30 minutes before retrying to avoid blocking your workflow
* **Concurrency safe** — Multiple tasks requesting browsers simultaneously are serialized to prevent conflicts
* **Diagnostic messages** — Shows exactly what went wrong (disk space, network, permissions) and how to fix it
* **Doctor auto-fix** — Run `myrm doctor` with auto-fix to diagnose and repair browser issues

### Why This Matters

| Other Agent tools                                  | Myrm                                    |
| -------------------------------------------------- | --------------------------------------- |
| Must run `playwright install chromium` in terminal | Automatic — just use the app            |
| Windows dependency errors confuse non-dev users    | Clear diagnostics with fix instructions |
| Browser features broken until manual setup         | Works on first use                      |