> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myrmagent.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Search & Fetch

> Built-in web_search and web_fetch — multi-engine search, 3-tier fetch, and content filtering that keeps junk out of your Agent context.

# Web Search & Fetch

Myrm includes **web\_search** and **web\_fetch** as built-in tools. They work together: search finds sources, fetch reads pages — both filter content **before** it reaches the LLM.

## Why It Matters

Most agents pass raw search snippets or full HTML into the model. That wastes tokens and hurts answer quality. Myrm runs a local filter pipeline:

```
Query → multi-engine search → BM25/Reranker → relevant snippets
URL   → 3-tier fetch → DOM prune → (optional) BM25+vector+Reranker → clean text
```

**Result:** \~40–50% fewer tokens on web-heavy tasks, \$0/month local operation.

:::note Honest comparison
Hermes, OpenClaw, OpenCode, and Claude Code all have basic web search/fetch tools. Myrm's advantage is the **full local filter pipeline** — not merely having the tools, but cleaning content before it hits the LLM.
:::

## Web Search

### Engines

Supports **7 providers** (configure in Settings → Search):

| Provider              | Best for                                        |
| --------------------- | ----------------------------------------------- |
| SearxNG (self-hosted) | Privacy, 70+ aggregated engines including Baidu |
| Tavily                | General research                                |
| Exa                   | Semantic/neural search                          |
| Perplexity            | Q\&A style                                      |
| Google PSE            | Custom site search                              |
| DataForSEO            | SEO/data tasks                                  |
| Firecrawl             | Search-as-a-service fallback                    |

### X (Twitter) Live Search

Myrm offers dedicated **X/Twitter real-time search** via xAI's Responses API — retrieve posts, threads, and trending discussions with inline citations.

**Two authentication paths:**

| Method          | Setup                                                  | Best for                                           |
| --------------- | ------------------------------------------------------ | -------------------------------------------------- |
| API Key         | Paste xAI API key in Settings → Models & Providers     | Developers with paid xAI API                       |
| SuperGrok OAuth | One-click device-code login in Settings → Integrations | SuperGrok / X Premium+ subscribers (no extra cost) |

**Capabilities:** date range filters, handle inclusion/exclusion (max 10), image/video understanding in matching posts, inline citations with source URLs, degraded-result detection (warns when filters yield no sourced data).

OAuth tokens are AES-256-GCM encrypted at rest, auto-refreshed before expiry, and reused across 5 xAI services (X Search, Image Gen, Video Gen, TTS, STT) — one login unlocks the full xAI multimodal stack.

### Intent Detection (Zero LLM Cost)

Myrm auto-detects **7 intent types** and adjusts engine parameters:

* Code, News, Academic, Finance, Security, Social, General

Say *"latest AI security CVEs"* — no mode picker needed.

### 5-Layer Intelligent Routing

Unlike competitors that use a single retrieval path, Myrm employs 5 layers of routing intelligence — all without a separate Planner LLM call:

| Layer | Mechanism                 | Benefit                                                                                                                         |
| ----- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| 1     | LLM natural selection     | Agent decides which tools to call per step                                                                                      |
| 2     | Tool description guidance | Each tool states when it should be invoked                                                                                      |
| 3     | Intent optimizer          | Zero-cost keyword routing to engine categories                                                                                  |
| 4     | Capability discovery      | `discover_capability_tool` progressive loading for MCP skills (all builtin tools Turn1 eager; no mid-thread bind\_tools mutate) |
| 5     | RSG post-verification     | Evaluates quality and guides follow-up searches                                                                                 |

### Web Fetch vs Browser (Smart Routing)

When both **web\_fetch** and **browser** tools are enabled, Myrm guides the Agent without coupling to System Prompt:

| Task                                   | Preferred tool  | Why                                                   |
| -------------------------------------- | --------------- | ----------------------------------------------------- |
| Read article / docs / blog (no clicks) | **web\_fetch**  | Faster, cheaper; JS rendering handled internally      |
| Login, forms, clicks, infinite scroll  | **browser**     | Full interaction + session reuse via Extension Bridge |
| Find URLs first                        | **web\_search** | Discovery before fetch or browse                      |

**How it works:** Tool descriptions + Dynamic Hints (injected only when the paired tool is available) + symmetric Loop Guard suggestions + separate browser pool isolation (CRAWL vs AGENT). Hermes only hints browser→web\_extract; OpenClaw has no cross-tool hints.

This means: no extra latency, no prompt cache break, and more flexible than a static pre-planner.

### Retrieval Modes

| Mode                | When           | Pipeline                                               |
| ------------------- | -------------- | ------------------------------------------------------ |
| **Basic** (default) | Most queries   | BM25 + RRF multi-query fusion → smart truncation       |
| **Precision**       | Long documents | Chunk → BM25 top-50 → Reranker top-20 → merge adjacent |

Precision mode uses semantic reranking (score threshold **0.6**) — low-relevance chunks are dropped.

### Domain Diversity Sort

Multi-query results are automatically reordered to prevent any single domain from dominating. Each subsequent result from the same domain is penalized by an exponential decay factor (default **0.8**):

* 1st result from `example.com` — full score
* 2nd result from `example.com` — score × 0.8
* 3rd result from `example.com` — score × 0.64

This ensures the Agent sees diverse sources rather than multiple pages from one website. No configuration needed — enabled by default in `combine_search_results_unified`.

### Visual references (no dedicated image search tool)

When reference images are needed, the agent uses **web\_search → web\_fetch / browser** to obtain image URLs from pages, then **image\_tool** for generation — same pattern as OpenClaw / Hermes; no separate DuckDuckGo image-search dependency.

### Resilience

* **Engine fallback** — primary fails → automatic switch
* **15-minute result cache** — repeat queries cost nothing
* **30s health probe** — detects unavailable engines early

## Web Fetch

### 3-Tier Architecture

| Tier       | Speed   | Use case                   |
| ---------- | ------- | -------------------------- |
| L1 HTTP    | \~100ms | Static pages, APIs         |
| L2 Browser | \~1–3s  | JavaScript-rendered pages  |
| L3 Stealth | \~3–5s  | Cloudflare, anti-bot sites |

**AdaptiveRouter** learns per-domain costs and picks the optimal tier automatically.

### Optional L4 Remote Fallback (Default OFF)

When L1–L3 all fail, you can **opt in** to a remote reader fallback (Settings → Search → **Web Fetch Remote Escalation**):

| Provider     | Role                                                                                         |
| ------------ | -------------------------------------------------------------------------------------------- |
| Jina Reader  | First remote attempt (optional API key; free tier when available)                            |
| Firecrawl v2 | Second attempt — **Keyless (1,000 req/mo free)** or bring your own API key for higher quotas |

**Firecrawl v2 highlights:**

* **Zero-config start** — works without an API key; Firecrawl's free tier gives you 1,000 requests/month
* **Custom API Base** — point to your own self-hosted Firecrawl instance for enterprise privacy or higher throughput
* **One-click verify** — the Settings UI tests connectivity against your configured endpoint (including self-hosted)

**Design guarantees (vs competitors that always hit cloud APIs):**

* **Default OFF** — no extra Turn1 tools, prompt cache unchanged
* **Per-session cap** (1–50) — prevents runaway remote billing
* **SSRF guard** before any remote call
* **No built-in bulk crawl** — use search+fetch+browser locally, or Firecrawl MCP for site-wide indexing
* **Plane deny switch** — `MYRM_WEB_FETCH_ESCALATION=denied` for enterprise hard blocks

OpenClaw/Hermes often depend on Firecrawl for hard pages and always require an API key; Myrm keeps **local-first**, supports keyless free-tier out of the box, and treats cloud as an explicit, capped last resort.

### Privacy Protection

**Explicit Opt-In Design:** Search only activates when you manually configure and enable a provider in Settings. Unlike some alternatives that silently send queries to free engines (e.g., DuckDuckGo) behind your back, Myrm never makes network calls without your explicit configuration. No API key configured = no search at all.

**Transparent Fallback:** When a primary provider fails and falls back to your configured backup, you receive a real-time notification in the chat — never a silent switch.

When a proxy is configured, L2/L3 automatically enable **DNS over HTTPS (DoH)** — DNS queries are encrypted so ISPs and network administrators cannot see which sites you visit. Zero configuration, zero performance overhead.

### Ad/Tracker Domain Blocking

L2 and L3 automatically block **3500+ known advertising and tracking domains** at the network level (Peter Lowe's curated list). Ad scripts never load — pages render faster, DOM stays clean, and token consumption drops. Zero configuration, enabled by default in standard mode.

### fetch\_and\_extract (Smart Extraction)

When reranker + embedding are configured, `web_fetch` supports **fetch\_and\_extract**:

1. 3-tier crawl (HTTP → Browser → Stealth)
2. Chunk page content
3. **BM25 + vector hybrid retrieval** (Qdrant embeddings)
4. Reranker re-ranking → top relevant passages only

This replaces cloud LLM summarization (e.g. Hermes `web_extract` + Gemini) with **zero-LLM local filtering**.

### Content Cleaning

Before text enters the Agent context:

1. **DOM pruning** — removes nav, ads, footer, sidebar
2. **HTML → Markdown** — structured, LLM-friendly
3. **Smart truncation** — `max_chars` with `was_truncated` flag
4. **Binary routing** — PDFs parsed separately (no garbled HTML)
5. **YouTube fast-path** — auto-detects YouTube URLs, extracts timestamped subtitles directly via API; falls back to HTML fetcher if transcript unavailable
6. **Bilibili fast-path** — auto-detects Bilibili video URLs, extracts timestamped subtitles via public API (view + player); supports AI-generated subtitles with SessionVault cookies; falls back to browser crawl if unavailable

### Caching

* Request coalescing (concurrent same-URL = one fetch)
* Stale-While-Revalidate
* ETag / Last-Modified conditional requests
* 35+ tracking params stripped from URLs

## Deep Research: 5-Phase Lifecycle

Myrm's Deep Research runs a structured **CLARIFY → PLAN → EXPLORE → RESEARCH → REPORT** pipeline — not a single-shot "search and summarize":

| Phase        | What happens                                                                                                           | Why it matters                                                                                                  |
| ------------ | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **CLARIFY**  | Structured form asks the user to confirm intent                                                                        | 100% accurate understanding vs AI guessing                                                                      |
| **PLAN**     | Generates ≤6-step research plan, **pauses for user review** (approve / edit / skip)                                    | Ensures systematic coverage; user controls direction before spending tokens                                     |
| **EXPLORE**  | Auto-queries your Wiki knowledge base via FTS5 full-text search (zero LLM cost)                                        | Skips redundant web searches for topics you already have notes on; sub-agents focus on **new** information only |
| **RESEARCH** | Parallel sub-agents search + self-reflect via `think` tool; local context injected into orchestrator prompt            | Faster with quality checkpoints at each cycle; known information excluded from search tasks                     |
| **REPORT**   | Multi-page structured report with inline citations【1】【2】, Information Integrity Rules, and Limitations/Gaps disclosure | Verifiable, honest about unknowns, zero extra LLM calls                                                         |

### Research Studio (Three-Column Workbench)

The `/research` route provides a dedicated three-column layout optimized for deep research workflows:

| Column                  | Purpose           | Key features                                                       |
| ----------------------- | ----------------- | ------------------------------------------------------------------ |
| **Left: Resource Pool** | Curate inputs     | Wiki concept search, file upload with progress, checkbox selection |
| **Center: Chat**        | Converse with AI  | Full ChatWindow with all Agent capabilities                        |
| **Right: Output**       | Preview artifacts | ArtifactRenderer + PortalTabs + download + save to Wiki            |

* **PC**: three columns side-by-side with draggable dividers (left 240–480px, right 280–600px)
* **Mobile**: auto-degrades to tab switching (768px breakpoint)
* **Context sync**: checking a resource in the pool automatically injects it into the chat context via `mentionReferences`; unchecking removes it without affecting other @ references
* **Output actions**: download artifacts or save them to your Wiki with one click (scoped to the active agent)

### Mid-Research Controls

* **Plan confirmation gate** — review, edit, or approve the research plan before execution begins
* **HITL callback** — pause after any cycle to adjust direction
* **Budget guard** — `max_budget_usd` + threshold warnings
* **Cancel + keep** — stop anytime, partial results preserved
* **ReplanMiddleware** — auto-detects drift and self-corrects

### vs MiroFlow (GAIA/BrowseComp benchmark leader)

|                      |                      Myrm                     |                    MiroFlow                   |
| -------------------- | :-------------------------------------------: | :-------------------------------------------: |
| Task understanding   |          ✅ User confirms via CLARIFY          |      ⚠️ AI guesses "traps" (\$0.5–2/call)     |
| Research planning    |               ✅ Structured plan               |                     ❌ None                    |
| Self-reflection      |                 ✅ `think` tool                |                     ❌ None                    |
| Mid-research control |            ✅ HITL + cancel + budget           |                     ❌ None                    |
| Parallel sub-agents  |             ✅ Semaphore-controlled            |             ❌ Single serial agent             |
| Source dedup         |                ✅ SourceTracker                |                     ❌ None                    |
| Extra cost           |                      \$0                      |            \$0.5–2 per o3 hint call           |
| Report output        | ✅ Multi-page report + inline citations + gaps | ⚠️ `\boxed{42}` short answer (benchmark-only) |
| Report integrity     |    ✅ Strict source-based, unverified marked   |                     ❌ None                    |

## vs Competitors

|                      |                   Myrm                   |              Hermes             |                OpenClaw                |       OpenCode       |
| -------------------- | :--------------------------------------: | :-----------------------------: | :------------------------------------: | :------------------: |
| Built-in search      |                     ✅                    |                ✅                |                    ✅                   |           ✅          |
| Result filtering     |              ✅ BM25/Reranker             |        ❌ API passthrough        |              ❌ raw snippet             |   ❌ API passthrough  |
| Built-in fetch       | ✅ 3-tier local + L4 Firecrawl v2 keyless | ⚠️ Firecrawl+LLM (key required) |    ⚠️ HTTP/Firecrawl (key required)    |     ⚠️ HTTP only     |
| Vector extract mode  |           ✅ fetch\_and\_extract          |         ❌ (LLM summary)         |                    ❌                   |           ❌          |
| DOM pruning          |                     ✅                    |                ❌                |              ❌ regex/text              | ❌ Turndown full page |
| Chinese search       |              ✅ SearxNG+Baidu             |       ⚠️ backend-dependent      |               ❌ DDG poor               |     ⚠️ cloud API     |
| Image search         |  ❌ (web\_search + web\_fetch / browser)  |                ❌                |                    ❌                   |           ❌          |
| Monthly cost (local) |   **\$0** (keyless free tier included)   |     API fees (key required)     | Firecrawl fallback fees (key required) |   Exa/Parallel fees  |

### Where Hermes Differs (not stronger)

* Plugin backends (Exa/Tavily/Firecrawl) — more cloud vendors, but all require API keys
* `web_extract` skips local embedding setup by using **LLM summarization instead** — easier setup, **costs tokens per page**
* SSRF + URL secret blocking — mature, same class as Myrm (not a differentiator)

### Auto Strip Ads & Redundancy

| Cleanup                            |           Myrm           |  OpenClacky  |       OpenCode       |        Hermes        |
| ---------------------------------- | :----------------------: | :----------: | :------------------: | :------------------: |
| Remove nav/sidebar/footer          |     ✅ DOM tree prune     | ❌ full regex | ❌ Turndown full page | ⚠️ API/LLM dependent |
| Remove ads (link\_density scoring) |  ✅ ContentPruningFilter  |       ❌      |           ❌          |           ❌          |
| Clean search snippets              |             ✅            |       ❌      |           ❌          |           ❌          |
| Dedup multi-query results          |    ✅ URL+content hash    |       ❌      |           ❌          |           ❌          |
| Domain diversity sort              |    ✅ decay\_factor=0.8   |       ❌      |           ❌          |           ❌          |
| Drop low-relevance passages        | ✅ Reranker threshold 0.6 |       ❌      |           ❌          |           ❌          |

**Plain language:** We don't dump whole pages into the AI — we extract body text, strip ads and nav, dedupe, and keep only passages that match your question.

### Zero-Config Comparison (corrected)

|                         | Myrm                                                            | Hermes                          |
| ----------------------- | --------------------------------------------------------------- | ------------------------------- |
| web\_fetch out of box   | ✅ **Local 3-tier + L4 Firecrawl v2 keyless**, no API key needed | ❌ Needs Firecrawl/Exa API key   |
| web\_search out of box  | ✅ GUI one-click SearxNG/DuckDuckGo                              | ⚠️ `hermes tools` + backend key |
| Smart long-page extract | ✅ fetch\_and\_extract (BM25+vector+Reranker)                    | ⚠️ LLM summary (costs tokens)   |

**Myrm is more zero-config on fetch** — cleaning works locally without cloud APIs.

## Retrieval Sufficiency Guard (RSG)

In **deep search** mode, Myrm automatically verifies whether retrieved content is sufficient to answer the user's question — before the main LLM starts composing a response.

### How It Works

After each `web_search` or `web_fetch` call (when `search_depth == "deep"`):

1. A **lightweight LLM** (your configured `lite_model`) evaluates the retrieved snippets against the original query
2. Returns a structured verdict: sufficient / insufficient + missing aspects + suggested follow-up queries
3. Detects **negative constraint violations** (e.g. "exclude React" but results contain React content)
4. Result is appended to tool output for the main Agent to act on

### Key Design Choices

| Aspect     | Decision                         | Why                                                          |
| ---------- | -------------------------------- | ------------------------------------------------------------ |
| Activation | `deep` mode only                 | Normal search stays fast; deep search gets quality assurance |
| Model      | `lite_model_cfg` (Flash/Haiku)   | Keeps cost \< \$0.001/evaluation                             |
| Output     | JSON Schema enforced             | Reliable structured output even from local models            |
| Failure    | Fail-open (`is_sufficient=True`) | Never blocks the main workflow                               |
| Threshold  | `confidence >= 0.7` to pass      | Below threshold → treated as insufficient                    |

### Negative Constraints (Unique Capability)

Vector search and BM25 cannot handle exclusion criteria like *"recommend frameworks except React"*. RSG explicitly extracts these constraints and checks results against them:

```
Query: "Compare backend frameworks, exclude Node.js"
Retrieved: Article mentioning Express.js (Node.js)
RSG verdict: negative_constraint_violations = ["contains Node.js/Express content"]
```

The Agent sees this violation and knows to search again with better filtering.

### vs Competitors

No competitor (Hermes, OpenClaw, Claude Code, Perplexity) has an equivalent post-retrieval sufficiency check. They rely entirely on the main LLM to judge whether it has enough information — which is unreliable because:

1. LLMs tend to answer confidently even with insufficient data (hallucination)
2. No explicit negative constraint checking is possible without a dedicated evaluator
3. No structured guidance for what to search next

## PTC Integration

In Programmatic Tool Calling scripts:

```python theme={null}
results = await tools.web_search("competitor pricing 2026", max_results=5)
page = await tools.web_fetch("https://example.com/pricing")
```

No extra API round-trips — search and fetch run inside the sandbox.

## Configuration

1. Open **Settings → Search** — pick engine, API keys, SearxNG URL
2. Enable **Reranker** in retriever settings for Precision mode on long docs
3. **Web Fetch** works out of the box (browser tier uses Patchright if installed)

## Migration Tips

| From        | Action                                                                   |
| ----------- | ------------------------------------------------------------------------ |
| Hermes      | Import config; disable Firecrawl-only web\_extract; use local web\_fetch |
| OpenClaw    | Import config; remove manual Tavily/baidu Skills                         |
| Claude Code | Enable SearxNG for self-hosted search; configure same models via LiteLLM |

See [Competitor Comparison](/docs/getting-started/competitor-comparison#web-search--web-fetch--dual-engine-vs-hermes--openclaw--claude-code) for full migration benefits.