Skip to main content

​
Vision & Image Understanding

Myrm lets you attach images in chat, on channels, and in desktop uploads. If your main model does not support vision, Myrm automatically routes images through a Vision Fallback model and injects a text description — so you never have to switch models manually.

​
How It Works

  1. You paste, drag, or attach any file (image, PDF, document, video, audio, code) in the WebUI (or send one on a supported channel).
  2. Myrm checks whether the selected main model supports vision (supports_vision).
  3. If yes — the image is sent to the main model as a native multimodal block.
  4. If no — Myrm shows a live “Analyzing image…” status, calls your configured Vision Fallback model, replaces the image with a concise text description, then continues the conversation with your main model.
The same pipeline applies to video: native video-capable models receive the file directly; others get frame extraction plus vision analysis.

​
Setup

  1. Open Settings → Models.
  2. Pick your Main chat model (any provider).
  3. Set Vision Fallback to a vision-capable model (e.g. GPT-4o, Gemini Flash, Qwen-VL).
  4. Click Test vision chain to verify the full ordered fallback chain (shows resolved_model and endpoint on failure).
  5. If your main model is text-only and no fallback is set, Myrm may offer Use this model — one click to pick the first vision-capable model from your enabled providers.
  6. Optional: set Video Fallback separately when your main model is text-only but you upload .mp4 files — Myrm routes video analysis through that slot before falling back to frame extraction.
  7. Optional: use the capability icons in the model picker — models with the eye icon support vision natively.
Myrm auto-detects model capabilities via LiteLLM and models.dev. You can override per model in the model card if needed.

​
When Vision Is Not Configured

If you attach an image or video but neither your main model nor Vision Fallback can analyze it, Myrm shows a warning toast with Go to Settings — deep-linking to Settings → Models → Default so you can set Vision Fallback in one click. Upload is not blocked (same philosophy as web search config gap): you can still send files, but analysis requires configuration.

​
What You Can Do

  • Lossless Long Screenshot Adaptive Slicing — when attaching tall screenshots (e.g. multi-screen chat logs, tall documents, or transaction records), the system automatically triggers in-memory adaptive slicing with boundary overlap buffers, concurrent multi-tile vision recognition, and seamless markdown transcript reconstruction, completely avoiding the downsampling blur seen in traditional approaches.
  • Screenshot Q&A — paste a screenshot and ask what is wrong or what to click next.
  • 6-Tool Image Annotator — open any image or screenshot in the built-in editor with freehand, rectangle, ellipse, arrow, text, and blur tools. Annotated images upload as attachments and the agent reads them as multimodal input.
  • Point-to-Edit Element Picker — in HTML artifact previews, toggle picker mode and click any element. Myrm captures the CSS selector, DOM breadcrumb, and outerHTML, then lets you type a one-line instruction so the agent modifies that exact element.
  • Select-to-AI Toolbar — select code in the Monaco editor (Modify / Explain / Optimize / Comment) or text in document previews (Modify / Explain / Rewrite). The selection context including artifact ID and line range is sent to the agent automatically.
  • VisualApproval — when the agent requests permission for a browser or desktop action, a coordinate-highlighted screenshot shows exactly where the action targets, so you can approve or reject with full visual context.
  • Non-vision main model — use a cheap text model for reasoning while vision fallback handles images.
  • Channel images — Telegram, Discord, iMessage, and other channels deliver images into the same pipeline.
  • PDF & documents — scanned or image-heavy PDFs can route through vision when text extraction is sparse.
  • Agent 原生多模态看图与脚本处理 — Agent 直接通过 file_read_tool 读取工作区图片（大模型原生看图，纯文本主模型自动 Fallback），高级像素运算与图像裁切直接通过沙箱 Python 脚本完成 — 零学习成本，零 Prompt 冗余。
  • Transparent downgrade badge — when fallback runs, chat shows vision_backend: vlm (full vision model), native_video (model reads video directly), or frame (FFmpeg frame extraction).

​
Status & Caching

While fallback analysis runs, the chat shows an Analyzing image (or Analyzing video) indicator. When done, it clears automatically. Identical images in the same session are cached by content hash — repeat uploads do not trigger duplicate vision API calls.

​
Async Image Generation

When your agent has the image generation tool enabled and credentials are ready, saying “draw a banner” does not block the chat for 30–120 seconds.
  1. The agent enqueues a background task and returns a task_id immediately.
  2. An ImageTaskCard appears in the message thread with live progress (SSE, with polling fallback).
  3. You can keep chatting, refine the prompt, or start another task while images render.
  4. Finished images appear in the card and can flow into the media library.
Security (cloud & desktop): API keys and Gateway tokens are AES-GCM sealed before the task is written to local tasks.db — backups and sandbox volumes never store plaintext secrets. Enable image models under Settings → Models (or your agent’s tool panel) and pick a provider (20+ via LiteLLM, with automatic failover).

​
Tips

  • Client-Side Image Admission: When dragging or pasting multi-megabyte ultra-HD images, the client leverages OffscreenCanvas to downsample and lossily convert to WebP in milliseconds (volume reduced by 98% down to ~400KB), completely eliminating gateway HTTP 413 rejections with zero UI blocking.
  • Prompt Cache Prefix Lock: In multi-turn conversations with multiple images, Myrm’s 2MB quantum offload buffer evicts Base64 payloads from older history while injecting structured metadata placeholders (dimensions and sandbox storage paths). This prevents visual amnesia, keeps Prompt Cache hit rates above 90%, and slashes vision token costs by up to 80%.
  • Configure a fast, cost-effective model for Vision Fallback (e.g. Gemini 2.5 Flash or GPT-4o-mini) if you send frequent code screenshots.
  • If analysis fails, you get a clear error message; the rest of your message still processes.