Skip to main content

​
Vision & Image Understanding

Myrm lets you attach images in chat, on channels, and in desktop uploads. If your main model does not support vision, Myrm automatically routes images through a Vision Fallback model and injects a text description — so you never have to switch models manually.

​
How It Works

  1. You paste, drag, or attach any file (image, PDF, document, video, audio, code) in the WebUI (or send one on a supported channel).
  2. Myrm checks whether the selected main model supports vision (supports_vision).
  3. If yes — the image is sent to the main model as a native multimodal block.
  4. If no — Myrm shows a live “Analyzing image…” status, calls your configured Vision Fallback model, replaces the image with a concise text description, then continues the conversation with your main model.
The same pipeline applies to video: native video-capable models receive the file directly; others get frame extraction plus vision analysis.

​
Setup

  1. Open Settings → Models.
  2. Pick your Main chat model (any provider).
  3. Set Vision Fallback to a vision-capable model (e.g. GPT-4o, Gemini Flash, Qwen-VL).
  4. Click Test vision chain to verify the full ordered fallback chain (shows resolved_model and endpoint on failure).
  5. If your main model is text-only and no fallback is set, Myrm may offer Use this model — one click to pick the first vision-capable model from your enabled providers.
  6. Optional: set Video Fallback separately when your main model is text-only but you upload .mp4 files — Myrm routes video analysis through that slot before falling back to frame extraction.
  7. Optional: use the capability icons in the model picker — models with the eye icon support vision natively.
Myrm auto-detects model capabilities via LiteLLM and models.dev. You can override per model in the model card if needed.

​
When Vision Is Not Configured

If you attach an image or video but neither your main model nor Vision Fallback can analyze it, Myrm shows a warning toast with Go to Settings — deep-linking to Settings → Models → Default so you can set Vision Fallback in one click. Upload is not blocked (same philosophy as web search config gap): you can still send files, but analysis requires configuration.

​
What You Can Do

  • Screenshot Q&A — paste a screenshot and ask what is wrong or what to click next.
  • 6-Tool Image Annotator — open any image or screenshot in the built-in editor with freehand, rectangle, ellipse, arrow, text, and blur tools. Annotated images upload as attachments and the agent reads them as multimodal input.
  • Point-to-Edit Element Picker — in HTML artifact previews, toggle picker mode and click any element. Myrm captures the CSS selector, DOM breadcrumb, and outerHTML, then lets you type a one-line instruction so the agent modifies that exact element.
  • Select-to-AI Toolbar — select code in the Monaco editor (Modify / Explain / Optimize / Comment) or text in document previews (Modify / Explain / Rewrite). The selection context including artifact ID and line range is sent to the agent automatically.
  • VisualApproval — when the agent requests permission for a browser or desktop action, a coordinate-highlighted screenshot shows exactly where the action targets, so you can approve or reject with full visual context.
  • Non-vision main model — use a cheap text model for reasoning while vision fallback handles images.
  • Channel images — Telegram, Discord, iMessage, and other channels deliver images into the same pipeline.
  • PDF & documents — scanned or image-heavy PDFs can route through vision when text extraction is sparse.
  • Agent-active workspace vision — enable the vision-toolkit prebuilt skill so the agent can call vision_semantic_tool on files already in the sandbox (OCR, region read, two-image compare) and vision_geometry_tool for pixel-level diffs — no manual re-upload from chat.
  • Transparent downgrade badge — when fallback runs, chat shows vision_backend: vlm (full vision model), native_video (model reads video directly), or frame (FFmpeg frame extraction).

​
Status & Caching

While fallback analysis runs, the chat shows an Analyzing image (or Analyzing video) indicator. When done, it clears automatically. Identical images in the same session are cached by content hash — repeat uploads do not trigger duplicate vision API calls.

​
Async Image Generation

When your agent has the image generation tool enabled and credentials are ready, saying “draw a banner” does not block the chat for 30–120 seconds.
  1. The agent enqueues a background task and returns a task_id immediately.
  2. An ImageTaskCard appears in the message thread with live progress (SSE, with polling fallback).
  3. You can keep chatting, refine the prompt, or start another task while images render.
  4. Finished images appear in the card and can flow into the media library.
Security (cloud & desktop): API keys and Gateway tokens are AES-GCM sealed before the task is written to local tasks.db — backups and sandbox volumes never store plaintext secrets. Enable image models under Settings → Models (or your agent’s tool panel) and pick a provider (20+ via LiteLLM, with automatic failover).

​
Tips

  • Configure a fast, cost-effective model for Vision Fallback if you send many screenshots.
  • For large images, Myrm compresses automatically before calling the vision model.
  • If analysis fails, you get a clear error message; the rest of your message still processes.