Web Search & Fetch

Myrm includes web_search and web_fetch as built-in tools. They work together: search finds sources, fetch reads pages — both filter content before it reaches the LLM.

Why It Matters

Most agents pass raw search snippets or full HTML into the model. That wastes tokens and hurts answer quality. Myrm runs a local filter pipeline:

Query → multi-engine search → BM25/Reranker → relevant snippets
URL   → 3-tier fetch → DOM prune → (optional) BM25+vector+Reranker → clean text

Result: ~40–50% fewer tokens on web-heavy tasks, $0/month local operation. :::note Honest comparison Hermes, OpenClaw, OpenCode, and Claude Code all have basic web search/fetch tools. Myrm’s advantage is the full local filter pipeline — not merely having the tools, but cleaning content before it hits the LLM. :::

Web Search

Engines

Supports 7 providers (configure in Settings → Search):

Provider	Best for
SearxNG (self-hosted)	Privacy, 70+ aggregated engines including Baidu
Tavily	General research
Exa	Semantic/neural search
Perplexity	Q&A style
Google PSE	Custom site search
DataForSEO	SEO/data tasks
Firecrawl	Search-as-a-service fallback

X (Twitter) Live Search

Myrm offers dedicated X/Twitter real-time search via xAI’s Responses API — retrieve posts, threads, and trending discussions with inline citations. Two authentication paths:

Method	Setup	Best for
API Key	Paste xAI API key in Settings → Models & Providers	Developers with paid xAI API
SuperGrok OAuth	One-click device-code login in Settings → Integrations	SuperGrok / X Premium+ subscribers (no extra cost)

Capabilities: date range filters, handle inclusion/exclusion (max 10), image/video understanding in matching posts, inline citations with source URLs, degraded-result detection (warns when filters yield no sourced data). OAuth tokens are AES-256-GCM encrypted at rest, auto-refreshed before expiry, and reused across 5 xAI services (X Search, Image Gen, Video Gen, TTS, STT) — one login unlocks the full xAI multimodal stack.

Intent Detection (Zero LLM Cost)

Myrm auto-detects 7 intent types and adjusts engine parameters:

Code, News, Academic, Finance, Security, Social, General

Say “latest AI security CVEs” — no mode picker needed.

5-Layer Intelligent Routing

Unlike competitors that use a single retrieval path, Myrm employs 5 layers of routing intelligence — all without a separate Planner LLM call:

Layer	Mechanism	Benefit
1	LLM natural selection	Agent decides which tools to call per step
2	Tool description guidance	Each tool states when it should be invoked
3	Intent optimizer	Zero-cost keyword routing to engine categories
4	Capability discovery	`discover_capability_tool` progressive loading for MCP skills (all builtin tools Turn1 eager; no mid-thread bind_tools mutate)
5	RSG post-verification	Evaluates quality and guides follow-up searches

Web Fetch vs Browser (Smart Routing)

When both web_fetch and browser tools are enabled, Myrm guides the Agent without coupling to System Prompt:

Task	Preferred tool	Why
Read article / docs / blog (no clicks)	web_fetch	Faster, cheaper; JS rendering handled internally
Login, forms, clicks, infinite scroll	browser	Full interaction + session reuse via Extension Bridge
Find URLs first	web_search	Discovery before fetch or browse

How it works: Tool descriptions + Dynamic Hints (injected only when the paired tool is available) + symmetric Loop Guard suggestions + separate browser pool isolation (CRAWL vs AGENT). Hermes only hints browser→web_extract; OpenClaw has no cross-tool hints. This means: no extra latency, no prompt cache break, and more flexible than a static pre-planner.

Retrieval Modes

Mode	When	Pipeline
Basic (default)	Most queries	BM25 + RRF multi-query fusion → smart truncation
Precision	Long documents	Chunk → BM25 top-50 → Reranker top-20 → merge adjacent

Precision mode uses semantic reranking (score threshold 0.6) — low-relevance chunks are dropped.

Domain Diversity Sort

Multi-query results are automatically reordered to prevent any single domain from dominating. Each subsequent result from the same domain is penalized by an exponential decay factor (default 0.8):

1st result from example.com — full score
2nd result from example.com — score × 0.8
3rd result from example.com — score × 0.64

This ensures the Agent sees diverse sources rather than multiple pages from one website. No configuration needed — enabled by default in combine_search_results_unified.

Visual references (no dedicated image search tool)

When reference images are needed, the agent uses web_search → web_fetch / browser to obtain image URLs from pages, then image_tool for generation — same pattern as OpenClaw / Hermes; no separate DuckDuckGo image-search dependency.

Resilience

Engine fallback — primary fails → automatic switch
15-minute result cache — repeat queries cost nothing
30s health probe — detects unavailable engines early

Web Fetch

3-Tier Architecture

Tier	Speed	Use case
L1 HTTP	~100ms	Static pages, APIs
L2 Browser	~1–3s	JavaScript-rendered pages
L3 Stealth	~3–5s	Cloudflare, anti-bot sites

AdaptiveRouter learns per-domain costs and picks the optimal tier automatically.

Optional L4 Remote Fallback (Default OFF)

When L1–L3 all fail, you can opt in to a remote reader fallback (Settings → Search → Web Fetch Remote Escalation):

Provider	Role
Jina Reader	First remote attempt (optional API key; free tier when available)
Firecrawl v2	Second attempt — Keyless (1,000 req/mo free) or bring your own API key for higher quotas

Firecrawl v2 highlights:

Zero-config start — works without an API key; Firecrawl’s free tier gives you 1,000 requests/month
Custom API Base — point to your own self-hosted Firecrawl instance for enterprise privacy or higher throughput
One-click verify — the Settings UI tests connectivity against your configured endpoint (including self-hosted)

Design guarantees (vs competitors that always hit cloud APIs):

Default OFF — no extra Turn1 tools, prompt cache unchanged
Per-session cap (1–50) — prevents runaway remote billing
SSRF guard before any remote call
No built-in bulk crawl — use search+fetch+browser locally, or Firecrawl MCP for site-wide indexing
Plane deny switch — MYRM_WEB_FETCH_ESCALATION=denied for enterprise hard blocks

OpenClaw/Hermes often depend on Firecrawl for hard pages and always require an API key; Myrm keeps local-first, supports keyless free-tier out of the box, and treats cloud as an explicit, capped last resort.

Privacy Protection

Explicit Opt-In Design: Search only activates when you manually configure and enable a provider in Settings. Unlike some alternatives that silently send queries to free engines (e.g., DuckDuckGo) behind your back, Myrm never makes network calls without your explicit configuration. No API key configured = no search at all. Transparent Fallback: When a primary provider fails and falls back to your configured backup, you receive a real-time notification in the chat — never a silent switch. When a proxy is configured, L2/L3 automatically enable DNS over HTTPS (DoH) — DNS queries are encrypted so ISPs and network administrators cannot see which sites you visit. Zero configuration, zero performance overhead.

Ad/Tracker Domain Blocking

L2 and L3 automatically block 3500+ known advertising and tracking domains at the network level (Peter Lowe’s curated list). Ad scripts never load — pages render faster, DOM stays clean, and token consumption drops. Zero configuration, enabled by default in standard mode.

fetch_and_extract (Smart Extraction)

When reranker + embedding are configured, web_fetch supports fetch_and_extract:

3-tier crawl (HTTP → Browser → Stealth)
Chunk page content
BM25 + vector hybrid retrieval (Qdrant embeddings)
Reranker re-ranking → top relevant passages only

This replaces cloud LLM summarization (e.g. Hermes web_extract + Gemini) with zero-LLM local filtering.

Content Cleaning

Before text enters the Agent context:

DOM pruning — removes nav, ads, footer, sidebar
HTML → Markdown — structured, LLM-friendly
Smart truncation — max_chars with was_truncated flag
Binary routing — PDFs parsed separately (no garbled HTML)
YouTube fast-path — auto-detects YouTube URLs, extracts timestamped subtitles directly via API; falls back to HTML fetcher if transcript unavailable
Bilibili fast-path — auto-detects Bilibili video URLs, extracts timestamped subtitles via public API (view + player); supports AI-generated subtitles with SessionVault cookies; falls back to browser crawl if unavailable

Caching

Request coalescing (concurrent same-URL = one fetch)
Stale-While-Revalidate
ETag / Last-Modified conditional requests
35+ tracking params stripped from URLs

Deep Research: 5-Phase Lifecycle

Myrm’s Deep Research runs a structured CLARIFY → PLAN → EXPLORE → RESEARCH → REPORT pipeline — not a single-shot “search and summarize”:

Phase	What happens	Why it matters
CLARIFY	Structured form asks the user to confirm intent	100% accurate understanding vs AI guessing
PLAN	Generates ≤6-step research plan, pauses for user review (approve / edit / skip)	Ensures systematic coverage; user controls direction before spending tokens
EXPLORE	Auto-queries your Wiki knowledge base via FTS5 full-text search (zero LLM cost)	Skips redundant web searches for topics you already have notes on; sub-agents focus on new information only
RESEARCH	Parallel sub-agents search + self-reflect via `think` tool; local context injected into orchestrator prompt	Faster with quality checkpoints at each cycle; known information excluded from search tasks
REPORT	Multi-page structured report with inline citations【1】【2】, Information Integrity Rules, and Limitations/Gaps disclosure	Verifiable, honest about unknowns, zero extra LLM calls

Research Studio (Three-Column Workbench)

The /research route provides a dedicated three-column layout optimized for deep research workflows:

Column	Purpose	Key features
Left: Resource Pool	Curate inputs	Wiki concept search, file upload with progress, checkbox selection
Center: Chat	Converse with AI	Full ChatWindow with all Agent capabilities
Right: Output	Preview artifacts	ArtifactRenderer + PortalTabs + download + save to Wiki

PC: three columns side-by-side with draggable dividers (left 240–480px, right 280–600px)
Mobile: auto-degrades to tab switching (768px breakpoint)
Context sync: checking a resource in the pool automatically injects it into the chat context via mentionReferences; unchecking removes it without affecting other @ references
Output actions: download artifacts or save them to your Wiki with one click (scoped to the active agent)

Mid-Research Controls

Plan confirmation gate — review, edit, or approve the research plan before execution begins
HITL callback — pause after any cycle to adjust direction
Budget guard — max_budget_usd + threshold warnings
Cancel + keep — stop anytime, partial results preserved
ReplanMiddleware — auto-detects drift and self-corrects

vs MiroFlow (GAIA/BrowseComp benchmark leader)

	Myrm	MiroFlow
Task understanding	✅ User confirms via CLARIFY	⚠️ AI guesses “traps” ($0.5–2/call)
Research planning	✅ Structured plan	❌ None
Self-reflection	✅ `think` tool	❌ None
Mid-research control	✅ HITL + cancel + budget	❌ None
Parallel sub-agents	✅ Semaphore-controlled	❌ Single serial agent
Source dedup	✅ SourceTracker	❌ None
Extra cost	$0	$0.5–2 per o3 hint call
Report output	✅ Multi-page report + inline citations + gaps	⚠️ `\boxed{42}` short answer (benchmark-only)
Report integrity	✅ Strict source-based, unverified marked	❌ None

vs Competitors

	Myrm	Hermes	OpenClaw	OpenCode
Built-in search	✅	✅	✅	✅
Result filtering	✅ BM25/Reranker	❌ API passthrough	❌ raw snippet	❌ API passthrough
Built-in fetch	✅ 3-tier local + L4 Firecrawl v2 keyless	⚠️ Firecrawl+LLM (key required)	⚠️ HTTP/Firecrawl (key required)	⚠️ HTTP only
Vector extract mode	✅ fetch_and_extract	❌ (LLM summary)	❌	❌
DOM pruning	✅	❌	❌ regex/text	❌ Turndown full page
Chinese search	✅ SearxNG+Baidu	⚠️ backend-dependent	❌ DDG poor	⚠️ cloud API
Image search	❌ (web_search + web_fetch / browser)	❌	❌	❌
Monthly cost (local)	$0 (keyless free tier included)	API fees (key required)	Firecrawl fallback fees (key required)	Exa/Parallel fees

Where Hermes Differs (not stronger)

Plugin backends (Exa/Tavily/Firecrawl) — more cloud vendors, but all require API keys
web_extract skips local embedding setup by using LLM summarization instead — easier setup, costs tokens per page
SSRF + URL secret blocking — mature, same class as Myrm (not a differentiator)

Auto Strip Ads & Redundancy

Cleanup	Myrm	OpenClacky	OpenCode	Hermes
Remove nav/sidebar/footer	✅ DOM tree prune	❌ full regex	❌ Turndown full page	⚠️ API/LLM dependent
Remove ads (link_density scoring)	✅ ContentPruningFilter	❌	❌	❌
Clean search snippets	✅	❌	❌	❌
Dedup multi-query results	✅ URL+content hash	❌	❌	❌
Domain diversity sort	✅ decay_factor=0.8	❌	❌	❌
Drop low-relevance passages	✅ Reranker threshold 0.6	❌	❌	❌

Plain language: We don’t dump whole pages into the AI — we extract body text, strip ads and nav, dedupe, and keep only passages that match your question.

Zero-Config Comparison (corrected)

	Myrm	Hermes
web_fetch out of box	✅ Local 3-tier + L4 Firecrawl v2 keyless, no API key needed	❌ Needs Firecrawl/Exa API key
web_search out of box	✅ GUI one-click SearxNG/DuckDuckGo	⚠️ `hermes tools` + backend key
Smart long-page extract	✅ fetch_and_extract (BM25+vector+Reranker)	⚠️ LLM summary (costs tokens)

Myrm is more zero-config on fetch — cleaning works locally without cloud APIs.

Retrieval Sufficiency Guard (RSG)

In deep search mode, Myrm automatically verifies whether retrieved content is sufficient to answer the user’s question — before the main LLM starts composing a response.

How It Works

After each web_search or web_fetch call (when search_depth == "deep"):

A lightweight LLM (your configured lite_model) evaluates the retrieved snippets against the original query
Returns a structured verdict: sufficient / insufficient + missing aspects + suggested follow-up queries
Detects negative constraint violations (e.g. “exclude React” but results contain React content)
Result is appended to tool output for the main Agent to act on

Key Design Choices

Aspect	Decision	Why
Activation	`deep` mode only	Normal search stays fast; deep search gets quality assurance
Model	`lite_model_cfg` (Flash/Haiku)	Keeps cost < $0.001/evaluation
Output	JSON Schema enforced	Reliable structured output even from local models
Failure	Fail-open (`is_sufficient=True`)	Never blocks the main workflow
Threshold	`confidence >= 0.7` to pass	Below threshold → treated as insufficient

Negative Constraints (Unique Capability)

Vector search and BM25 cannot handle exclusion criteria like “recommend frameworks except React”. RSG explicitly extracts these constraints and checks results against them:

Query: "Compare backend frameworks, exclude Node.js"
Retrieved: Article mentioning Express.js (Node.js)
RSG verdict: negative_constraint_violations = ["contains Node.js/Express content"]

The Agent sees this violation and knows to search again with better filtering.

vs Competitors

No competitor (Hermes, OpenClaw, Claude Code, Perplexity) has an equivalent post-retrieval sufficiency check. They rely entirely on the main LLM to judge whether it has enough information — which is unreliable because:

LLMs tend to answer confidently even with insufficient data (hallucination)
No explicit negative constraint checking is possible without a dedicated evaluator
No structured guidance for what to search next

PTC Integration

In Programmatic Tool Calling scripts:

results = await tools.web_search("competitor pricing 2026", max_results=5)
page = await tools.web_fetch("https://example.com/pricing")

No extra API round-trips — search and fetch run inside the sandbox.

Configuration

Open Settings → Search — pick engine, API keys, SearxNG URL
Enable Reranker in retriever settings for Precision mode on long docs
Web Fetch works out of the box (browser tier uses Patchright if installed)

Migration Tips

From	Action
Hermes	Import config; disable Firecrawl-only web_extract; use local web_fetch
OpenClaw	Import config; remove manual Tavily/baidu Skills
Claude Code	Enable SearxNG for self-hosted search; configure same models via LiteLLM

See Competitor Comparison for full migration benefits.

​Web Search & Fetch

​Why It Matters

​Web Search

​Engines

​X (Twitter) Live Search

​Intent Detection (Zero LLM Cost)

​5-Layer Intelligent Routing

​Web Fetch vs Browser (Smart Routing)

​Retrieval Modes

​Domain Diversity Sort

​Visual references (no dedicated image search tool)

​Resilience

​Web Fetch

​3-Tier Architecture

​Optional L4 Remote Fallback (Default OFF)

​Privacy Protection

​Ad/Tracker Domain Blocking

​fetch_and_extract (Smart Extraction)

​Content Cleaning

​Caching

​Deep Research: 5-Phase Lifecycle

​Research Studio (Three-Column Workbench)

​Mid-Research Controls

​vs MiroFlow (GAIA/BrowseComp benchmark leader)

​vs Competitors

​Where Hermes Differs (not stronger)

​Auto Strip Ads & Redundancy

​Zero-Config Comparison (corrected)

​Retrieval Sufficiency Guard (RSG)

​How It Works

​Key Design Choices

​Negative Constraints (Unique Capability)

​vs Competitors

​PTC Integration

​Configuration

​Migration Tips

Web Search & Fetch

Why It Matters

Web Search

Engines

X (Twitter) Live Search

Intent Detection (Zero LLM Cost)

5-Layer Intelligent Routing

Web Fetch vs Browser (Smart Routing)

Retrieval Modes

Domain Diversity Sort

Visual references (no dedicated image search tool)

Resilience

Web Fetch

3-Tier Architecture

Optional L4 Remote Fallback (Default OFF)

Privacy Protection

Ad/Tracker Domain Blocking

fetch_and_extract (Smart Extraction)

Content Cleaning

Caching

Deep Research: 5-Phase Lifecycle

Research Studio (Three-Column Workbench)

Mid-Research Controls

vs MiroFlow (GAIA/BrowseComp benchmark leader)

vs Competitors

Where Hermes Differs (not stronger)

Auto Strip Ads & Redundancy

Zero-Config Comparison (corrected)

Retrieval Sufficiency Guard (RSG)

How It Works

Key Design Choices

Negative Constraints (Unique Capability)

vs Competitors

PTC Integration

Configuration

Migration Tips