Benchmark

Benchmarks the models configured in Clapilot against real workspace workflows with tool calls (storyboards, invoices, quarterly reports, task boards, and more) and computes a score per model.

What it does

benchmark is a bundled iframe module that measures how well the models configured in Clapilot (agent_provider_configs) complete typical Clapilot workflows that require tool calls. A benchmark run executes every selected scenario once per selected model as a real ClapilotAICore agent run, observes the tool-call stream, and computes a normalized score (0–100) per scenario and model, plus a per-model leaderboard for the whole run.

The module is not installed by default. It appears in the Module Store as a bundled module and must be installed explicitly; a global migration (db/migrations/281_benchmark_module_default_off.sql) seeds a module_installs override so it also stays off on instances using the legacy_all install policy. Installing the module runs its schema migration (benchmark_runs, benchmark_results).

Scenarios

Scenarios live in src/lib/benchmark/scenarios.ts. Each one defines an agent prompt, a scenario-scoped allowedToolNames allowlist, expected tool calls with weights, an LLM-judge rubric, and the bundled modules it requires. Scenarios whose required modules are not installed are shown as unavailable and are skipped in runs.

Scenario	Workflow	Key expected tools	Requires
`video-storyboard`	Create a Video Studio storyboard project and configure voice dubbing plus generated background music (never starts final video generation)	`video_studio_create_ai_project`, `video_studio_update_ai_project`	`video-studio`
`quarterly-tax-report`	Record three business expenses for the previous quarter and generate the quarterly accounting report	`accounting_create_entry` ×3, `accounting_generate_report`	–
`invoice-canvas`	Create a reusable invoice template in Canvas, then a concrete customer invoice from it	`canvas_create_template`, `canvas_create_file_from_template`, `canvas_edit_file`	`canvas`
`project-tasks`	Set up a project board with four tasks and move one to in-progress	`aufgaben_create_board`, `aufgaben_create_task` ×4, `aufgaben_move_status`	–
`document-brief`	Create a folder with two structured documents	`documents_create_folder`, `documents_create` ×2	–
`email-draft`	Draft (never send) a professional project status email	`emails_create_draft`	–
`calendar-planning`	Plan a workshop day with three calendar events	`calendar_create_event` ×3	–

Email sending, video generation, and destructive tools are excluded from every scenario allowlist.

Scoring

Each result is scored out of 100, composed of:

Expected tool checks (scenario-specific weights): each expected tool earns its points proportionally to min(successful calls / required calls, 1), counted from tool.end stream events without errors.
Run completion (10 points): the agent turn completed without a runtime error.
Tool reliability (10 points): scaled by the fraction of tool calls that did not fail.
LLM judge (20 points, optional per run): the instance default model scores the final answer 0–10 against the scenario rubric via the agent chat-completions endpoint. If the judge is disabled or fails, its points are removed from the denominator instead of scoring zero.

Metrics recorded per result: duration, tool call and error counts, input/output tokens (summed from agent_model_request_logs by run id), provider cost (agent_runs.cost_usd), and the effective model that actually served the run (fallbacks can differ from the requested model — the leaderboard reports effective_model per result for that reason).

Run mechanics

Runs execute sequentially (one scenario × model at a time) in the Next.js server process; one run may be active at a time. A staleness sweep marks runs older than three hours as failed after a server restart.
Each item runs in a throwaway agent session (includeStoredHistory: false, lightContext, compact tool profile, explicitModelSelection: true) that is deleted afterwards.
Scenario prompts embed a unique BM-XXXXXX marker. Created workspace artifacts (boards, documents, drafts, canvas files, projects, events, accounting entries) are kept and identifiable by that marker.
Cancelling a run stops before the next item; the in-flight scenario finishes first.

API

See API Reference: GET /api/benchmark/overview, GET/POST /api/benchmark/runs, GET/DELETE /api/benchmark/runs/:id, POST /api/benchmark/runs/:id/cancel. Reading requires an authenticated user; starting, cancelling, and deleting runs require the admin role. All routes return 404 while the module is not installed.

Agent tools

benchmark_list_scenarios, benchmark_list_runs, benchmark_get_run, and benchmark_start_run are install-gated on this module; benchmark_start_run additionally requires an admin session. See Agent Tool Contracts for the full contracts. There is intentionally no Live Voice wiring: benchmark runs take many minutes and are not a realtime-voice workflow.

Apple clients

The Benchmark module is a web-only admin/diagnostics surface. The iOS and macOS clients have no corresponding native module screen; results are viewed in the web app.