​

Eval Lab

​

Key capabilities

• Single-profile eval — Run a test suite against your current agent configuration and see pass/fail results in real time.
• Cross-profile matrix eval — Run the same test cases across multiple agent profiles to detect regressions before switching models or prompts.
• Environment reproducibility — Every eval run captures a frozen EvalManifest snapshot (model, tools, profile, benchmark mode) so results are always reproducible.
• Benchmark mode — One-click toggle that strips user customizations (skills, MCPs, memory, web search) to produce a fair baseline score using only core tools.
• Historical trend tracking — View pass-rate trends over time with interactive charts. History table shows Profile, Model, pass rate, average time, and token usage per run.
• Click-to-detail reports — Click any historical run to load the full report with per-case results, environment snapshot, and diff view.

​

Getting started

1. Navigate to Eval Lab from the sidebar (or go to /eval-lab).
2. In the Config tab, select a dataset and optionally enable Benchmark Mode.
3. Click Run to start evaluation. Progress updates in real time.
4. View results in the Report tab — summary cards, per-case table, and environment snapshot.
5. Switch to the History tab to compare across runs.

​

WorkBuddy Bench benchmark

​

Download a subset

1. Open Eval Lab and switch to the WorkBuddy Bench tab.
2. Each track shows its task count, approximate size, and local status (Not downloaded / Downloaded).
3. Click Download to fetch the archive from Hugging Face in the background. Progress is streamed via SSE; the button disables while a download is running.
4. Archives are verified against the official SHA-256 checksums and installed atomically — a corrupt or partial archive is never used.
5. Use Refresh anytime to re-read local disk state; a just-finished download immediately flips the button to Downloaded.

​

Run a benchmark

1. Click Run on a downloaded track to evaluate every task in that subset.
2. A Stop control is available while the evaluation is running — abort is honored mid-run and mid-download.
3. When finished, the run produces a reproducible report (model, tools, profile, scoring mode recorded in the manifest).

​

International authority benchmarks (BrowseComp)

• Sampled runs first — big benchmarks auto-suggest a small sample (e.g. 20 of 1,266) so you validate the whole pipeline at a fraction of the token cost before committing to a full run. Clear the sample to run the full set. Reports badge runs that truly sampled (sampled), so the label is never a guess.
• Fair baseline by design — benchmark mode strips your skills, MCPs and memory, so the score reflects the model plus core tools only.
• Pre-flight checks — the run verifies the search / embedding services the benchmark needs are configured and reachable before spending any tokens.

​

Deterministic scoring: layered acceptance (CompositeVerifier)

• Run the task’s own tests — the test_suite assertion executes the task-bundled tests against the agent’s workspace output. Results are parsed from JUnit XML or a plain reward-script exit code.
• Per-turn pass_rate — multi-turn conversations score each turn separately, so a task that passes on turn 1 but degrades later is caught early and stops wasting tokens (configurable on_turn_fail strategy).
• No hidden LLM judge — parsing is deterministic: pass/fail comes from the test runner, not from a model reading the output. This keeps scores reproducible and immune to judge-prompt drift.
• Partial credit, not binary — skipped tests are excluded from the denominator so an uncollectable flaky test can’t tank a real pass; reward-script outputs map to graduated scores.
• Test-level visibility — each track card shows both the run pass rate and the test pass rate (avg_pass_rate over turns), and every report carries the aggregated test-level average.

​

Judge model configuration

​

Scoring diagnostics: failures never go silent

• Attributed failures — execution-level failures (security block, timeout, crash) are reported as such, instead of being misreported as an “unreadable reward file”. You can immediately tell why a grade failed: the policy blocked it, it timed out, or the command itself errored.
• stdout tail attached — every “command failed”, “reward file unreadable”, and “JUnit file unreadable” message carries up to 800 characters of the grading command’s real stdout, so you see what actually happened instead of guessing.
• Counts-based reward fallback — reward payloads that only carry tests_passed/tests_total counters (or a tests[] array), rather than the full score.json/reward.json schema, are now parsed and scored correctly instead of being rejected as unreadable. This mirrors the official WBBench runner’s counting semantics while staying compatible with lightweight scorer scripts.

​

Layered evaluation: how much does each harness layer actually contribute?

​

The layer chain

​

How to run

1. Open Eval Lab → Matrix tab.
2. Pick a downloaded benchmark and start a Layered run.
3. Progress streams live (download bytes → per-layer case progress), the report renders in the familiar matrix view, and every run lands in history for later comparison.

​

Why you can trust the numbers

• Fingerprints pin the exact switches — each layer’s benchmark_mode / skills_enabled / memory_enabled combination is locked by a SHA-256 fingerprint in the report, so the layer definitions stay comparable across harness upgrades.
• Full model disclosure — reports record agent_model, judge_model, and harness_version (same honesty contract as Memory A/B and benchmark manifests), so a score curve can only drift when real capability changed — never because the ruler moved.
• Behavior evidence, not just scores — each layer reports how many times the agent actually called memory tools (memory_tool_calls), so “memory didn’t help” is never confused with “memory was never used”.
• Isolated measurement — all layers run with shared contexts disabled and throwaway memory volumes (SQLite + embedded Qdrant) that are evicted and deleted after the run; the memory layer’s gain reflects the memory mechanism itself, not your real shared memory.

​

Cross-profile matrix eval

1. In the Config tab, select 2 or more profiles from the chip-based multi-selector.
2. A Matrix mode badge appears automatically when 2+ profiles are selected.
3. Click Run Matrix to start. The dashboard switches to the Matrix tab showing real-time progress:
   • Which profile is currently being evaluated
   • Profile progress (e.g., 2/3)
   • Case completion progress bar
4. When complete, the Matrix tab displays:
   • Summary cards — total cases, stable rate, regression count, total time
   • Per-profile table — pass rate, tokens, cost, and time for each profile
   • Case × Profile grid — color-coded matrix showing each case’s status on each profile (green = stable, amber = regression, red = all failed)
5. Cases are classified as:
   • Stable — passes on ALL selected profiles (safe to switch models)
   • Regression — passes on some but fails on others (risky areas)
   • All failed — fails on every profile (needs investigation regardless)

​

Memory A/B: prove memory’s value with numbers

​

How it works

1. Open Eval Lab → WorkBuddy Bench and pick a downloaded track.
2. Click Memory A/B on a track card — a confirmation dialog shows exactly what will run.
3. The dialog probes your embedding model first: memory retrieval depends on embeddings, so a missing or unreachable embedding model is reported before you waste a run.
4. Myrm runs the same tasks twice — once with enable_memory=True, once with enable_memory=False — everything else identical.
5. Live progress streams over SSE; the header Stop button aborts the run and cleans up mid-run.

​

The report

• Pass rate — task completion rate with vs without memory
• Tokens & cost — the price of remembering
• memory_tool_calls — how many times memory tools were actually invoked. If memory helped, you can see it was really used — no false positives from “enabled but never used” memory.

​

History

​

Isolation & safety

​

Test coverage

​

Eval workspace lifecycle: isolated sandboxes that self-clean

• Destroyed after every run — success, failure, abort, or crash all land in the same cleanup path (create_session and execute-only runs alike), so long-running servers never accumulate disk.
• Crash self-healing — any workspace left behind by a dead process is swept automatically at the next server startup; nothing to clean by hand.
• Failure-proof cleanup — each teardown step (memory-volume eviction, directory removal, per-profile workspace cleanup) is independently guarded, so one failing step can never leak the rest.

​

Evaluation integrity: decontamination, budgets & trajectory disclosure

​

Contamination-proof runs (HF leakage protection)

• Web fetch — any hostname on the benchmark’s blocklist raises a benchmark_blocked tool error the moment it’s hit.
• Web search — results from blocklisted hosts are silently dropped before ranking/formatting, and queries matching blocklisted terms fail fast.
• Shell / code execution — the network policy defaults to no outbound access (or a strict allowlist without Hugging Face), so there is no side channel to exfiltrate the answer key.
• Reports record whether decontamination was active (decontam_active), and the Report tab shows it as a badge — so a “clean” score is provable, not assumed.

​

Declared run budgets

• A stuck agent can’t burn tokens forever — the tool-call middleware and the engine’s recursion budget enforce the caps.
• The moment a cap is hit, the run stops and the per-case report records exactly which limit stopped it (limit_reached, e.g. max_tool_calls or max_iterations). Truncated cases are never silently indistinguishable from finished ones.
• The manifest records the declared budget, and the Report tab shows it as Budget · N calls / M iterations.

​

Trajectory disclosure

• Tool call details — how many tool calls the agent made (N× badge with tooltip).
• Blocked count — how many attempts the contamination guard intercepted (Blocked N badge).
• Limit reached — which budget stopped the run (Limit badge with the specific type in the tooltip).
• Judge transparency — semantic (LLM-as-a-judge) assertions accept a configurable judge_prompt, recorded in the report; agent_model / judge_model are disclosed for every run (task-native grades honestly mark none). Exact-match answers short-circuit the judge entirely, so trivial hits never spend judge tokens.

​

Environment snapshot

​

Benchmark mode

• System prompt → empty
• Tools → core-only (no MCP, no skills, no sub-agents)
• Memory, web search, replan, compression → disabled

​

Writing test cases

{"message": "What is 2+2?", "expected": "4"}

{
  "message": "Write a haiku about coding",
  "assertions": [
    {"type": "semantic", "criteria": "Output is a valid haiku with 5-7-5 syllable structure"}
  ]
}

{
  "turns": [
    {"message": "Create a file called test.txt", "assertions": [{"type": "tool", "tool_name": "write_file"}]},
    {"message": "Read the file back", "expected_contains": "test.txt"}
  ],
  "on_turn_fail": "stop"
}

​

Skill publish quality guard

1. EvolutionScreener — 5-phase screening (locked → force retry → cooldown → rejection history → LLM confirmation) blocks invalid evolution attempts before they consume resources.
2. EvalCase Regression Gate — Runs bound EvalCases against candidate variants. Applies score penalties proportional to failed cases; hard-filters variants that fail 100% of cases.
3. Improvement Gate — Injects the original skill as a baseline competitor. Only variants that genuinely score higher than the original survive.
4. SandboxValidator — Executes candidates in an isolated sandbox to verify they actually work.
5. ConfidenceApprovalFlow — Multi-signal risk control (diff ratio, effective rate, confidence threshold). Red flags downgrade to human Diff Review in the GUI.

​

History & degradation

• Old reports (created before manifest tracking) display - for Profile and Model columns — no data loss.
• The history table supports horizontal scrolling on narrow screens.
• Failed report loads show a toast notification instead of failing silently.