Web Search & Fetch

Myrm includes web_search and web_fetch as built-in tools. They work together: search finds sources, fetch reads pages — both clean up content before it reaches the LLM.

Why It Matters

Most agents pass raw search snippets or full HTML into the model. That wastes tokens and hurts answer quality. Myrm cleans content locally before it reaches the LLM:

Query → multi-engine search → merge + dedup (round-robin across questions) → top results
URL   → 3-tier fetch → DOM prune → (optional) BM25+vector+Reranker → clean text

Result: ~40–50% fewer tokens when reading pages (DOM pruning in web_fetch), $0/month local operation. :::note Honest comparison Hermes, OpenClaw, OpenCode, and Claude Code all have basic web search/fetch tools. Myrm’s advantage is the full local filter pipeline — not merely having the tools, but cleaning content before it hits the LLM. Hermes + wigolo MCP users: the community often adds npx wigolo for multi-engine search, fetch, and research. Myrm ships this built-in — no extra Node sidecar; same-session search uses in-process coalescing; cross-session recall uses sessions corpus; page watch via Cron PollTrigger. See competitor comparison. :::

Web Search

Engines

Supports 7 providers (configure in Settings → Search):

Provider	Best for
SearxNG (self-hosted)	Privacy, 70+ aggregated engines including Baidu
Tavily	General research
Exa	Semantic/neural search
Perplexity	Q&A style
Google PSE	Custom site search
DataForSEO	SEO/data tasks
Firecrawl	Search-as-a-service fallback

X (Twitter) Live Search

Myrm offers dedicated X/Twitter real-time search via xAI’s Responses API — retrieve posts, threads, and trending discussions with inline citations. Two authentication paths:

Method	Setup	Best for
API Key	Paste xAI API key in Settings → Models & Providers	Developers with paid xAI API
SuperGrok OAuth	One-click device-code login in Settings → Integrations	SuperGrok / X Premium+ subscribers (no extra cost)

Capabilities: date range filters, handle inclusion/exclusion (max 10), image/video understanding in matching posts, inline citations with source URLs, degraded-result detection (warns when filters yield no sourced data). OAuth tokens are AES-256-GCM encrypted at rest, auto-refreshed before expiry, and reused across 5 xAI services (X Search, Image Gen, Video Gen, TTS, STT) — one login unlocks the full xAI multimodal stack.

Intent Detection (Zero LLM Cost)

Myrm auto-detects 7 intent types and adjusts engine parameters:

Code, News, Academic, Finance, Security, Social, General

Say “latest AI security CVEs” — no mode picker needed.

5-Layer Intelligent Routing

Unlike competitors that use a single retrieval path, Myrm employs 5 layers of routing intelligence — all without a separate Planner LLM call:

Layer	Mechanism	Benefit
1	LLM natural selection	Agent decides which tools to call per step
2	Tool description guidance	Each tool states when it should be invoked
3	Intent optimizer	Zero-cost keyword routing to engine categories
4	Capability discovery	`discover_capability_tool` progressive loading for MCP skills (all builtin tools Turn1 eager; no mid-thread bind_tools mutate)
5	RSG post-verification	Evaluates quality and guides follow-up searches

Web Fetch vs Browser (Smart Routing)

When both web_fetch and browser tools are enabled, Myrm guides the Agent without coupling to System Prompt:

Task	Preferred tool	Why
Read article / docs / blog (no clicks)	web_fetch	Faster, cheaper; JS rendering handled internally
Login, forms, clicks, infinite scroll	browser	Full interaction + session reuse via Extension Bridge
Find URLs first	web_search	Discovery before fetch or browse

How it works: Tool descriptions + Dynamic Hints (injected only when the paired tool is available) + symmetric Loop Guard suggestions + separate browser pool isolation (CRAWL vs AGENT). Hermes only hints browser→web_extract; OpenClaw has no cross-tool hints. This means: no extra latency, no prompt cache break, and more flexible than a static pre-planner.

Multi-Query Merge

When one call carries several questions, results are merged round-robin across the questions, so no single question crowds out the others. Duplicates are removed by URL and content, then the merged list goes through the domain diversity sort below and the first 10 results reach the model.

Domain Diversity Sort

Search results are automatically reordered to prevent any single domain from dominating. Each subsequent result from the same domain is penalized by an exponential decay factor (default 0.8):

1st result from example.com — full score
2nd result from example.com — score × 0.8
3rd result from example.com — score × 0.64

This ensures the Agent sees diverse sources rather than multiple pages from one website. No configuration needed — every search call goes through apply_domain_diversity_sort.

Structured Hit Fields

Search results from providers that expose rich metadata (e.g. Volcengine Doubao) carry structured fields through the entire pipeline:

Field	Example	Benefit
`site_name`	”GitHub”, “Wikipedia”, “StackOverflow”	LLM sees source identity, not just a URL
`authority_description`	”官方”, “媒体”, “个人”	Authority signal for answer prioritization
`date`	”2026-07-30”	Recency-aware reasoning; outdated sources deprioritized

These fields survive deduplication, domain-diversity sort, and multi-query fusion — guaranteed by the combine_search_results_unified metadata-preservation pipeline. Competitors (Hermes, OpenClaw, DeerFlow) discard provider metadata or flatten it into snippet strings.

Citation Hover UI

In-chat citation markers (e.g. 【1】) reveal a hover card with rich source details:

Info	Effect
Site name	Shows “GitHub”, “Wikipedia” instead of raw URLs
Authority badge	Color-coded label (“Official”, “Media”) for quick trust assessment
Relative freshness	”3 days ago”, “2 weeks ago” — internationalized, instant recency awareness

Touch-friendly (tap on iPad/mobile). Supports 5 languages (zh/en/ja/ko/de). Competitor DeerFlow shows URL + snippet only; the other 6 competitors have no citation hover UI at all.

Agent Explicit Search Parameters

The Agent can explicitly restrict search results by time through time_range on web_search_tool:

Parameter	Values	Effect
`time_range`	`day`, `week`, `month`, `year`, or `YYYY-MM-DD..YYYY-MM-DD`	Restricts results to the specified time window

For official or authoritative sources, add site: operators in the query (e.g. site:gov.cn, site:github.com). Three-layer parameter priority fusion:

Agent explicit (highest) — parameters the Agent passes directly
Intent auto-detection — zero-cost keyword-based inference (News → 1 day, Finance → 1 week)
Config defaults (lowest) — provider-level extra_params

Parameter normalization (_normalize_explicit_params) translates semantic parameters into provider-specific formats:

Semantic	Volcengine	SearxNG	Tavily
`time_range="week"`	`TimeRange="OneWeek"`	`time_range="week"`	`days=7`
`time_range="2026-01-01..2026-07-30"`	Direct passthrough	Graceful fallback	Graceful fallback

Strategy D (Parameter Escalation): When the Retrieval Sufficiency Guard reports insufficient results, the Agent autonomously retries with adjusted queries or a narrower time_range — no RSG code changes needed. Per-turn search budget: inside each delegated sub-agent run (for example a Deep Research worker), ToolTurnBudgetGuard limits web_search_tool to 20 search queries (harness SECURITY_SYSTEM.md §6.2). The main agent’s own searches are bounded by LoopGuard and the iteration budget.

Visual references (no dedicated image search tool)

When reference images are needed, the agent uses web_search → web_fetch / browser to obtain image URLs from pages, then image_tool for generation; there is no separate DuckDuckGo image-search dependency.

Resilience

Engine fallback — primary fails → automatic switch
15-minute result cache — repeat queries cost nothing
30s health probe — detects unavailable engines early

X Live Search (xAI x_search)

Beyond generic web search, Myrm ships a built-in x-live-search skill wired directly to xAI’s official x_search tool on the Responses API — live posts from X, not stale search-engine caches. What it does:

Author & date filtering — pin queries to specific handles (up to 20 accounts) and from_date/to_date ranges (YYYY-MM-DD), so “what did @handle say this week” is one call
Image/video understanding — posts containing media are analyzed natively (enable_image_understanding / enable_video_understanding, the latter exclusive to X Search), no separate OCR pipeline
Readable citations — xAI returns url_citation titles as bare inline numbers; Myrm rewrites them to clickable @handle post labels so answers stay traceable at a glance
Honest-citation guard — when a search returns no citations, the skill explicitly warns the answer may rest on general knowledge instead of pretending every claim is grounded
Protocol-current — model grok-4.6, full parameter surface, 180s timeout tuned for long agentic retries

Credential handling (the differentiator): your xAI key is stored in the encrypted credential database and injected into the sandboxed execution environment only for the lifetime of the call (XAI_API_KEY / XAI_BASE_URL). Scripts, logs, and chat history never see a plaintext key — unlike plaintext .env exports required by third-party X-search MCP plugins across competitor harnesses. Availability gating: with zero xAI credentials configured, the skill reports available: false (with an unavailable_reason linking to Settings → Skills) and is excluded from the prompt entirely — no dead tool signatures, no prompt pollution. Add the provider once in Settings and the skill lights up on the next session.

Web Fetch

3-Tier Architecture

Tier	Speed	Use case
L1 HTTP	~100ms	Static pages, APIs
L2 Browser	~1–3s	JavaScript-rendered pages
L3 Stealth	~3–5s	Cloudflare, anti-bot sites

AdaptiveRouter learns per-domain costs and picks the optimal tier automatically.

Optional L4 Remote Fallback (Default OFF)

When L1–L3 all fail, you can opt in to a remote reader fallback (Settings → Search → Web Fetch Remote Escalation):

Provider	Role
Jina Reader	First remote attempt (optional API key; free tier when available)
Firecrawl v2	Second attempt — Keyless (1,000 req/mo free) or bring your own API key for higher quotas

Firecrawl v2 highlights:

Zero-config start — works without an API key; Firecrawl’s free tier gives you 1,000 requests/month
Custom API Base — point to your own self-hosted Firecrawl instance for enterprise privacy or higher throughput
One-click verify — the Settings UI tests connectivity against your configured endpoint (including self-hosted)

Design guarantees (vs competitors that always hit cloud APIs):

Default OFF — no extra Turn1 tools, prompt cache unchanged
Per-session cap (1–50) — prevents runaway remote billing
SSRF guard before any remote call
No built-in bulk crawl — use search+fetch+browser locally, or Firecrawl MCP for site-wide indexing
Plane deny switch — MYRM_WEB_FETCH_ESCALATION=denied for enterprise hard blocks

OpenClaw/Hermes often depend on Firecrawl for hard pages and always require an API key; Myrm keeps local-first, supports keyless free-tier out of the box, and treats cloud as an explicit, capped last resort.

Privacy Protection

Explicit Opt-In Design: Search only activates when you manually configure and enable a provider in Settings. Unlike some alternatives that silently send queries to free engines (e.g., DuckDuckGo) behind your back, Myrm never makes network calls without your explicit configuration. No API key configured = no search at all. Transparent Fallback: When a primary provider fails and falls back to your configured backup, you receive a real-time notification in the chat — never a silent switch. When a proxy is configured, L2/L3 automatically enable DNS over HTTPS (DoH) — DNS queries are encrypted so ISPs and network administrators cannot see which sites you visit. Zero configuration, zero performance overhead.

Ad/Tracker Domain Blocking

L2 and L3 automatically block 3500+ known advertising and tracking domains at the network level (Peter Lowe’s curated list). Ad scripts never load — pages render faster, DOM stays clean, and token consumption drops. Zero configuration, enabled by default in standard mode.

fetch_and_extract (Smart Extraction)

When the embedding and reranking models are configured and Advanced Retrieval is on in Preferences, web_fetch supports fetch_and_extract:

3-tier crawl (HTTP → Browser → Stealth)
Chunk page content
BM25 + vector hybrid retrieval narrows large chunk sets in memory (50 chunks or fewer by default skip straight to reranking)
Reranker re-ranking → top relevant passages only

Unlike cloud LLM summarization (e.g. Hermes web_extract + Gemini), this is zero-LLM local filtering.

Content Cleaning

Before text enters the Agent context:

DOM pruning — removes nav, ads, footer, sidebar
HTML → Markdown — structured, LLM-friendly
Smart truncation — max_chars with was_truncated flag
Binary routing — PDFs parsed separately (no garbled HTML)
YouTube fast-path — auto-detects YouTube URLs, extracts timestamped subtitles directly via API; falls back to HTML fetcher if transcript unavailable
Bilibili fast-path — auto-detects Bilibili video URLs, extracts timestamped subtitles via public API (view + player); supports AI-generated subtitles with SessionVault cookies; falls back to browser crawl if unavailable

Caching

Request coalescing (concurrent same-URL = one fetch)
Stale-While-Revalidate
ETag / Last-Modified conditional requests
35+ tracking params stripped from URLs

Deep Research: 5-Phase Lifecycle

Myrm’s Deep Research runs a structured CLARIFY → PLAN → EXPLORE → RESEARCH → REPORT pipeline — not a single-shot “search and summarize”:

Phase	What happens	Why it matters
CLARIFY	Structured form asks the user to confirm intent	100% accurate understanding vs AI guessing
PLAN	Generates ≤6-step research plan, pauses for user review (approve / edit / skip)	Ensures systematic coverage; user controls direction before spending tokens
EXPLORE	Auto-queries your Wiki knowledge base via FTS5 full-text search (zero LLM cost)	Skips redundant web searches for topics you already have notes on; sub-agents focus on new information only
RESEARCH	Parallel sub-agents search + self-reflect via `think` tool; local context injected into orchestrator prompt	Faster with quality checkpoints at each cycle; known information excluded from search tasks
REPORT	Multi-page structured report with inline citations【1】【2】, Information Integrity Rules, and Limitations/Gaps disclosure	Verifiable, honest about unknowns, zero extra LLM calls

Research Studio (Three-Column Workbench)

The /research route provides a dedicated three-column layout optimized for deep research workflows:

Column	Purpose	Key features
Left: Resource Pool	Curate inputs	Wiki concept search, file upload with progress, checkbox selection
Center: Chat	Converse with AI	Full ChatWindow with all Agent capabilities
Right: Output	Preview artifacts	ArtifactRenderer + PortalTabs + download + save to Wiki

PC: three columns side-by-side with draggable dividers (left 240–480px, right 280–600px)
Mobile: auto-degrades to tab switching (768px breakpoint)
Context sync: checking a resource in the pool automatically injects it into the chat context via mentionReferences; unchecking removes it without affecting other @ references
Output actions: download artifacts or save them to your Wiki with one click (scoped to the active agent)

Mid-Research Controls

Plan confirmation gate — review, edit, or approve the research plan before execution begins
HITL callback — pause after any cycle to adjust direction
Budget guard — max_budget_usd + threshold warnings
Cancel + keep — stop anytime, partial results preserved
ReplanMiddleware — auto-detects drift and self-corrects

vs MiroFlow (GAIA/BrowseComp benchmark leader)

	Myrm	MiroFlow
Task understanding	✅ User confirms via CLARIFY	⚠️ AI guesses “traps” ($0.5–2/call)
Research planning	✅ Structured plan	❌ None
Self-reflection	✅ `think` tool	❌ None
Mid-research control	✅ HITL + cancel + budget	❌ None
Parallel sub-agents	✅ Semaphore-controlled	❌ Single serial agent
Source dedup	✅ SourceTracker	❌ None
Extra cost	$0	$0.5–2 per o3 hint call
Report output	✅ Multi-page report + inline citations + gaps	⚠️ `\boxed{42}` short answer (benchmark-only)
Report integrity	✅ Strict source-based, unverified marked	❌ None

vs Competitors

	Myrm	Hermes	OpenClaw	OpenCode
Built-in search	✅	✅	✅	✅
Multi-query merge + dedup	✅ round-robin + URL/content dedup	❌ API passthrough	❌ raw snippet	❌ API passthrough
Built-in fetch	✅ 3-tier local + L4 Firecrawl v2 keyless	⚠️ Firecrawl+LLM (key required)	⚠️ HTTP/Firecrawl (key required)	⚠️ HTTP only
Vector extract mode	✅ fetch_and_extract	❌ (LLM summary)	❌	❌
DOM pruning	✅	❌	❌ regex/text	❌ Turndown full page
Chinese search	✅ SearxNG+Baidu	⚠️ backend-dependent	❌ DDG poor	⚠️ cloud API
Image search	❌ (web_search + web_fetch / browser)	❌	❌	❌
Monthly cost (local)	$0 (keyless free tier included)	API fees (key required)	Firecrawl fallback fees (key required)	Exa/Parallel fees

Where Hermes Differs (not stronger)

Plugin backends (Exa/Tavily/Firecrawl) — more cloud vendors, but all require API keys
web_extract skips local embedding setup by using LLM summarization instead — easier setup, costs tokens per page
SSRF + URL secret blocking — mature, same class as Myrm (not a differentiator)

Auto Strip Ads & Redundancy

Cleanup	Myrm	OpenClacky	OpenCode	Hermes
Remove nav/sidebar/footer	✅ DOM tree prune	❌ full regex	❌ Turndown full page	⚠️ API/LLM dependent
Remove ads (link_density scoring)	✅ ContentPruningFilter	❌	❌	❌
Clean search snippets	✅	❌	❌	❌
Dedup multi-query results	✅ URL+content hash	❌	❌	❌
Domain diversity sort	✅ decay_factor=0.8	❌	❌	❌
Keep only relevant passages (fetch_and_extract)	✅ BM25+vector+Reranker	❌	❌	❌

Plain language: We don’t dump whole pages into the AI — we extract body text, strip ads and nav, dedupe, and keep only passages that match your question.

Zero-Config Comparison (corrected)

	Myrm	Hermes
web_fetch out of box	✅ Local 3-tier + L4 Firecrawl v2 keyless, no API key needed	❌ Needs Firecrawl/Exa API key
web_search out of box	✅ GUI one-click SearxNG/DuckDuckGo	⚠️ `hermes tools` + backend key
Smart long-page extract	✅ fetch_and_extract (BM25+vector+Reranker)	⚠️ LLM summary (costs tokens)

Myrm is more zero-config on fetch — cleaning works locally without cloud APIs.

Retrieval Sufficiency Guard (RSG)

In deep search mode, Myrm automatically verifies whether retrieved content is sufficient to answer the user’s question — before the main LLM starts composing a response.

How It Works

After each web_search or web_fetch call (when search_depth == "deep"):

A lightweight LLM (your configured lite_model) evaluates the retrieved snippets against the original query
Returns a structured verdict: sufficient / insufficient + missing aspects + suggested follow-up queries
Detects negative constraint violations (e.g. “exclude React” but results contain React content)
Result is appended to tool output for the main Agent to act on

Key Design Choices

Aspect	Decision	Why
Activation	`deep` mode only	Normal search stays fast; deep search gets quality assurance
Model	`lite_model_cfg` (Flash/Haiku)	Keeps cost < $0.001/evaluation
Output	JSON Schema enforced	Reliable structured output even from local models
Failure	Fail-open (`is_sufficient=True`)	Never blocks the main workflow
Threshold	`confidence >= 0.7` to pass	Below threshold → treated as insufficient

Negative Constraints (Unique Capability)

Vector search and BM25 cannot handle exclusion criteria like “recommend frameworks except React”. RSG explicitly extracts these constraints and checks results against them:

Query: "Compare backend frameworks, exclude Node.js"
Retrieved: Article mentioning Express.js (Node.js)
RSG verdict: negative_constraint_violations = ["contains Node.js/Express content"]

The Agent sees this violation and knows to search again with better filtering.

vs Competitors

No competitor (Hermes, OpenClaw, Claude Code, Perplexity) has an equivalent post-retrieval sufficiency check. They rely entirely on the main LLM to judge whether it has enough information — which is unreliable because:

LLMs tend to answer confidently even with insufficient data (hallucination)
No explicit negative constraint checking is possible without a dedicated evaluator
No structured guidance for what to search next

PTC Integration

In Programmatic Tool Calling scripts:

results = await tools.web_search("competitor pricing 2026", max_results=5)
page = await tools.web_fetch("https://example.com/pricing")

No extra API round-trips — search and fetch run inside the sandbox.

Configuration

Open Settings → Search — pick engine, API keys, SearxNG URL
Configure the embedding and reranking models, then turn on Advanced Retrieval in Preferences so web_fetch extracts only the relevant passages of long pages
Web Fetch works out of the box (browser tier uses Patchright if installed)

Migration Tips

From	Action
Hermes	Import config; disable Firecrawl-only web_extract; use local web_fetch
OpenClaw	Import config; remove manual Tavily/baidu Skills
Claude Code	Enable SearxNG for self-hosted search; configure same models via LiteLLM

See Competitor Comparison for full migration benefits.

​Web Search & Fetch

​Why It Matters

​Web Search

​Engines

​X (Twitter) Live Search

​Intent Detection (Zero LLM Cost)

​5-Layer Intelligent Routing

​Web Fetch vs Browser (Smart Routing)

​Multi-Query Merge

​Domain Diversity Sort

​Structured Hit Fields

​Citation Hover UI

​Agent Explicit Search Parameters

​Visual references (no dedicated image search tool)

​Resilience

​X Live Search (xAI x_search)

​Web Fetch

​3-Tier Architecture

​Optional L4 Remote Fallback (Default OFF)

​Privacy Protection

​Ad/Tracker Domain Blocking

​fetch_and_extract (Smart Extraction)

​Content Cleaning

​Caching

​Deep Research: 5-Phase Lifecycle

​Research Studio (Three-Column Workbench)

​Mid-Research Controls

​vs MiroFlow (GAIA/BrowseComp benchmark leader)

​vs Competitors

​Where Hermes Differs (not stronger)

​Auto Strip Ads & Redundancy

​Zero-Config Comparison (corrected)

​Retrieval Sufficiency Guard (RSG)

​How It Works

​Key Design Choices

​Negative Constraints (Unique Capability)

​vs Competitors

​PTC Integration

​Configuration

​Migration Tips

Web Search & Fetch

Why It Matters

Web Search

Engines

X (Twitter) Live Search

Intent Detection (Zero LLM Cost)

5-Layer Intelligent Routing

Web Fetch vs Browser (Smart Routing)

Multi-Query Merge

Domain Diversity Sort

Structured Hit Fields

Citation Hover UI

Agent Explicit Search Parameters

Visual references (no dedicated image search tool)

Resilience

X Live Search (xAI x_search)

Web Fetch

3-Tier Architecture

Optional L4 Remote Fallback (Default OFF)

Privacy Protection

Ad/Tracker Domain Blocking

fetch_and_extract (Smart Extraction)

Content Cleaning

Caching

Deep Research: 5-Phase Lifecycle

Research Studio (Three-Column Workbench)

Mid-Research Controls

vs MiroFlow (GAIA/BrowseComp benchmark leader)

vs Competitors

Where Hermes Differs (not stronger)

Auto Strip Ads & Redundancy

Zero-Config Comparison (corrected)

Retrieval Sufficiency Guard (RSG)

How It Works

Key Design Choices

Negative Constraints (Unique Capability)

vs Competitors

PTC Integration

Configuration

Migration Tips