Benchmark

What it does

benchmark is a bundled iframe module that measures how well the models configured in Clapilot (agent_provider_configs) complete typical Clapilot workflows that require tool calls. A benchmark run executes every selected scenario for every selected model (optionally repeated) as a real ClapilotAICore agent run, observes the tool-call stream, reads the created objects back through the owning modules, scores each cell with versioned weights, and reports per-model coverage, technical success rate, and quality score separately. Every created object is registered as a run artifact and can be deleted selectively or automatically.

The module is not installed by default. It appears in the Module Store as a bundled module and must be installed explicitly; a global migration (db/migrations/281_benchmark_module_default_off.sql) seeds a module_installs override so it also stays off on instances using the legacy_all install policy. Installing the module runs its schema migrations (benchmark_runs, benchmark_results, benchmark_artifacts, benchmark_cleanup_audit). Instances that installed the module before migration 002_benchmark_validity_artifacts.sql existed are upgraded lazily on the first benchmark API call (src/lib/benchmark/schema.ts applies pending files from bundled-modules/benchmark/migrations/ using the same module_schema_migrations bookkeeping as the installer).

Scenarios

Scenarios live in src/lib/benchmark/scenarios.ts. Each one defines a version, an agent prompt, a scenario-scoped allowedToolNames allowlist, expected tool calls with weights, machine-checkable expected artifacts (object type, minimum count, required fields, minimum content length), weighted judge criteria, the bundled modules it requires, and data preconditions. Scenarios whose required modules are not installed (or whose preconditions fail) are shown as unavailable and their cells are excluded before the run starts.

Email sending, video generation, and destructive tools are excluded from every scenario allowlist. The invoice rubric states the arithmetically consistent totals (net 2,630.00 EUR, VAT 499.70 EUR, gross 3,129.70 EUR); a fixture test in src/lib/benchmark/scoring.test.ts pins them.

Preflight and run snapshot

Before a run is created, POST /api/benchmark/preflight (also triggered by Start benchmark in the UI) validates the request:

• every model ref must be present in the runtime model list (client.listModels()); an unknown model blocks the start instead of falling back to another model;
• every scenario's requiredModuleSlugs must be installed and resolvable on disk, and its preconditions (currently mandant_exists) must hold; otherwise every cell of that scenario is planned as excluded;
• the judge model (explicit judgeModelRef, else the first catalog model of the default provider) must be in the model list when the judge is enabled.

The UI shows executable and excluded cells (with the reason) and requires confirmation before starting. Excluded cells are stored as skipped results with a failureKind (module_unavailable, precondition_failed, model_unavailable); they never contribute a 0 score.

Each run stores an immutable snapshot (benchmark_runs.snapshot): methodology and scoring version, the weights, per-scenario version + content hash (prompt, system prompt, allowlist, expected tools/artifacts, rubric, criteria, timeout), judge model/prompt/schema version, target models with provider, module availability, and repetitions. The derived methodologyHash is shown in the run list; only runs with an identical hash are directly comparable, and the UI never mixes runs into a cross-run ranking. Runs created before snapshots existed are flagged Legacy: methodology not fully versioned.

Cell states and reproducibility

Every scenario × model × attempt is a cell (benchmark_results row) with one of the states queued, running, completed, failed, skipped, judge_pending, judge_failed, cancelled.

• Technical failures (timeout, runtime error, tool error, module unavailable, model unavailable, model fallback) end in failed with a metrics.failureKind; such cells keep their checks but never receive a quality score.
• The requested model and the model that actually served the run (agent_runs.effective_model) are both recorded. A mismatch for a non-virtual ref (anything but route/* and moa/*) is treated as a silent fallback and fails the cell (model_fallback).
• The judge runs after the technical outcome and the artifact validation are persisted (judge_pending), so a judge crash leaves a retryable cell instead of losing the agent output.
• Retry (POST /api/benchmark/runs/:id/retry) re-queues only failed cells and re-judges only judge_failed cells from their stored output; completed cells are never touched.
• repetitions (1-3) runs each cell several times; the matrix and leaderboard show mean, median, and spread (population standard deviation).
• Runs execute sequentially in the Next.js server process; one run may be active at a time. A staleness sweep marks runs (and their active cells) older than three hours as failed after a server restart.
• Each cell runs in a throwaway agent session (includeStoredHistory: false, lightContext, compact tool profile, explicitModelSelection: true) that is deleted afterwards.

Scoring and judge

Scoring is deterministic and server-side (src/lib/benchmark/scoring.ts, BENCHMARK_SCORING_VERSION). Each completed cell is scored out of 100 from:

• Expected tool checks (scenario-specific weights): each expected tool earns points proportionally to min(successful calls / required calls, 1), counted from tool.end stream events without errors.
• Run completion (10 points) and tool reliability (10 points, scaled by the fraction of tool calls that did not fail).
• Artifact validation (10 points per scenario): the created objects are read back through the owning module and checked against the scenario criteria (type, minimum count, required fields, minimum content).
• LLM judge (20 points, optional per run): the frozen judge model scores every scenario criterion from 0 to 10 in a strict JSON schema. The server validates the response (only known criterion ids, each exactly once, finite numbers in range, no extra keys, optional total must equal the weighted sum) and computes the weighted score itself; the calculation path is stored with the verdict. Any invalid response ends in judge_failed (retryable) — a missing judge verdict is never counted as 0 or as passed, and the cell has no final score.

The run summary (aggregateRunSummary) reports per model: executable/excluded cells, coverage (completed / executable), technical success rate, quality mean/median/spread over completed cells only, failed/skipped/judge-failed counts, duration, and cost. A comparability verdict states whether coverage is complete, whether the models are comparable (same executable cells, no missing judge verdicts), and whether a ranking is allowed; ranks are only assigned when it is. Otherwise the UI shows a prominent "incomplete / not comparable" warning with the reasons, and no winner is presented.

Artifact registry and cleanup

After each cell, the runner reads every expected artifact type back through the owning module (documents, folders, boards, tasks, calendar entries, drafts, video projects, Canvas files/templates via the Canvas module API, accounting entries via the Accounting module API) and registers matches in benchmark_artifacts. Attribution requires all of: the cell marker (BM-XXXXXX) in the title/name/subject/ description, creation after the cell start, and ownership by the benchmark user (tasks on a benchmark-created board are attributed by board). Pre-existing or foreign resources are never registered. Provider media referenced by video projects (generated frame images/clips) is registered as not deletable because no module delete API exists.

The run detail shows Benchmark-Artefakte grouped by documents, files, images, videos, and records, each with type, module, resource id, cell, and delete status. Benchmark-Artefakte löschen opens a confirmation dialog (count, irreversibility warning); the default selection is every unambiguously attributed, deletable artifact. Deletion runs exclusively through module APIs with an agent system token scoped to the run creator (children before parents: documents before folders, tasks before boards; drafts are discarded, which is the email module's delete semantics). Results are recorded per artifact (deleted, missing when the resource was already gone, failed with the error) and in benchmark_cleanup_audit (actor, mode, time, counts). Failed entries stay selectable and can be retried; repeating a cleanup is idempotent. The run and its scores are kept; deleted artifacts are marked.

Auto cleanup is a start option ("Erzeugte Benchmark-Dokumente und Dateien nach dem Lauf automatisch löschen", default off) with an optional retention in minutes. It runs once no cell is queued, running, or judge_pending — also for cancelled or partially failed runs — never before judge and validator, and reports pending → running → completed / partial_failed in the run's cleanup state (errors are logged with the [benchmark] prefix). Retention timers are re-armed after a server restart by a lazy sweep on the next run listing. Cleanup is refused with 409 while a cell is active or another cleanup is running.

UI and export

The run detail renders the comparability banner, the leaderboard (rank only when rankable, quality ± spread, coverage, technical success rate, failures, skipped, judge failures, duration, cost), the scenario × model matrix with cell states, the cell detail (state, failure kind, requested vs. served model, checks grouped by tool/technical/artifact/judge, judge criteria table with weights and calculation path, reasoning, agent answer), the run snapshot, and the artifact registry with cleanup log. Export (JSON) (GET /api/benchmark/runs/:id/export) contains snapshot, preflight, summary, comparability warnings, results, artifacts, and cleanup audit. Legacy runs and incomplete/non-comparable runs are labelled in the run list.

API

See API Reference: GET /api/benchmark/overview, POST /api/benchmark/preflight, GET/POST /api/benchmark/runs, GET/DELETE /api/benchmark/runs/:id, POST …/cancel, POST …/retry, GET …/artifacts, POST …/cleanup, GET …/export. Reading requires an authenticated user; preflight, starting, cancelling, retrying, cleaning up, and deleting runs require the admin role. All routes return 404 while the module is not installed.

Agent tools

benchmark_list_scenarios, benchmark_list_runs, benchmark_get_run, benchmark_preflight_run, benchmark_start_run, benchmark_retry_run, benchmark_list_artifacts, and benchmark_cleanup_artifacts are install-gated on this module; preflight, start, retry, and cleanup require an admin session and cleanup additionally confirm=true. See Agent Tool Contracts for the full contracts. There is intentionally no Live Voice wiring: benchmark runs take many minutes and are not a realtime-voice workflow.

Tests

• Unit: src/lib/benchmark/scoring.test.ts (invoice rubric fixture, weighted sums, invalid judge JSON, aggregation/comparability), preflight.test.ts, artifacts.test.ts (attribution, validation), cleanup.test.ts (selection planning, module API routing, idempotent execution), and the route authorization test src/app/api/benchmark/runs/[id]/cleanup/route.test.ts.
• Live end-to-end: npm run e2e:live:benchmark-cleanup (requires a running instance with the Benchmark and Video Studio modules installed, an admin login, and a model ref; see the header of tests/e2e/benchmark-artifact-cleanup.test.ts) runs a document plus media scenario, waits for scoring, checks the cleanup preview, deletes one artifact selectively, and verifies auto cleanup on a second run.

Apple clients

The Benchmark module is a web-only admin/diagnostics surface. The iOS and macOS clients have no corresponding native module screen; results are viewed in the web app.