Commerce-Agents Adoption Review

This page records the review of Anthropic's anthropics/commerce-agents blueprint (Apache-2.0, reviewed at commit fd4d592, published 2026-08-31) as an adoption candidate for ClapilotAICore. It follows the same pattern as the Cortheon study: read the external code at mechanism level, map every idea onto a concrete place in our runtime, and decide per idea whether we take the concept, the code, or nothing.

Clapilot has no commerce use case. The review covers only the three areas the evening scan flagged — the agent loop, the approval-gate mechanics, and the evals — and ignores shopping/merchant backends, verticals, Managed Agents manifests, and the Claude Code plugin.

TL;DR

Total recommended effort: about 14–20 person-days across four work packages (see Recommendation), of which the approval gate is the only one that touches all three clients.

What the blueprint actually contains

Size and shape matter for the code-vs-concept decision:

• Two agents (shopping, merchant), each with a core/ package (types, backend interface, prompt, tool contracts, gates, executor), a Messages API runtime (runtime-messages-api/), an Agent SDK runtime, and a Managed Agents manifest.
• commerce-common/ (~3,300 LoC Python) holds what both share: turn.py (loop helpers), execution.py (executor frame), fencing.py, memory.py, grounding.py, presentation.py, delegation.py, streaming.py, and testing.py (a scripted FakeClient).
• Four demo verticals with Next.js web apps make up most of the ~48k LoC.
• The README states it "is not maintained and does not accept contributions." There will be no upstream to track.

Everything safety-relevant is enforced inside the tool call, in one executor (BaseToolExecutor.execute) that all three run paths share. That is the one architectural decision worth keeping in mind for us: a gate that lives in the tool executor holds regardless of which provider loop or bridge invoked the tool.

1. Agent loop comparison

Reference: commerce_common/turn.py, *_runtime/orchestrator.py. Ours: services/clapilot-agent/src/sessions/index.mjs (executeRunNow, outer steer loop) and services/clapilot-agent/src/providers/index.mjs (six inner provider loops: runAnthropic, runBedrock, runGemini, runOpenAiResponsesWithTools, runOpenAiLike, runOllama).

Verdict: nothing to port. Two concepts (typed blocked outcome, settled results for interrupted writes) move into the gate work package because they are what make the gate observable.

2. Approval-gate mechanics

How the blueprint does it

Reference: merchant_agent/gates.py, merchant_agent/changes.py, merchant_agent/types.py (StagedChange, MerchantSessionState), examples/demo_common/merchant.py (change_action), docs/safety.md.

1. Writes never execute directly. Every mutating tool is a stage_* tool that creates a StagedChange (change_id, kind, status: staged, a field-level items[] diff with before/after, created_by, created_by_kind: operator | agent, guardrail notes, backend-computed money fields). The tool result carries a fixed note: "Staged only … apply it only after the operator approves."
2. Provenance ledger. Session state keeps seen_listings, read_listings, seen_campaigns, seen_changes — ids that tool results returned this session. A stage call naming an id the session never saw is held (PROVENANCE_GATE). A content edit additionally needs a full-record read.
3. Guardrails run twice: at stage time and again at apply time against the config in force then (items per change, price delta %, promotion depth, restock size, campaign budget, protected fields, one line per target+field).
4. The approval mark is host-owned. apply_change succeeds only if change_id ∈ state.approved_change_ids — and only the host writes that set: the portal's POST /changes/{id}/apply route, or the SDK console's host_approve. A preview card approves nothing; an approval typed into chat sets nothing (the model could fabricate it). The mark is added immediately before the executor runs and removed immediately after, whatever the outcome, so a later turn cannot spend it.
5. Held ≠ failed. A gate returns ToolOutcome.held(gate, text): the model receives plain guidance ("tell the operator it is staged and waiting for approval on …"), the host receives status: blocked, reason: approval. The turn continues normally.
6. Follow-through reminder. If the user asked for a concrete change and the turn ends with no stage_* attempt, one host-authored user message reminds the model to stage (never to apply). That message is tagged as host text so memory extraction and "latest user text" ignore it.
7. Audit stamps on the change record (applied_by is always the operator; discarded_by_kind distinguishes an agent-initiated discard).

What we have today

The main-agent bridge envelope hard-codes approvalPolicy: "never" (resolveBridgePermissionEnvelope), and the system prompt tells the model to act rather than describe. That is intentional and stays: the blueprint's answer to the same tension is not to ask before every tool, but to make only the apply step of a small set of irreversible actions need a mark that the model cannot produce.

Does the pattern fit our external actions?

Yes, and better than the fleet token does, for three reasons:

• The mark is set by a UI click, not by a typed phrase. The fleet phrase works but is a chat-only UX; it cannot be used from a job, a channel run, or the iOS app without a personal chat session (fleet_personal_session_required). A staged-action record with an approve/dismiss surface works on every run kind — the run ends with "staged, waiting for approval", and approval later applies the action without a model call.
• It preserves autonomy where we want it. Everything that is reversible (drafts, tasks, documents, calendar entries) stays ungated. Only emails_send_draft, social_media_publish_post, outbound channel sends to recipients outside the allowlist, and any future payment/transfer tool become "apply" steps.
• It gives the completion judge and the run reconciler a truthful signal. Today a run that ends on an unsent draft looks like a run that failed to send. With status: blocked, reason: approval the judge can score "staged and waiting" as complete-with-caveat instead of uncertified.

What we should not copy: the per-domain guardrail catalogue (price deltas, restock sizes) — for us guardrails are per-tool argument checks that already live in the tool-proxy handlers; and the provenance ledger as a hard rule on all ids — our tools are id-driven across many modules and a session-wide "seen ids" map would break legitimate cross-session references (a task id from the page context, a document id from a URL). A narrow provenance rule is enough: an apply must reference a staged record that this session created or listed, which is exactly what consumeFleetApproval already checks via the action hash.

3. Evals versus benchmark_*

What the blueprint has

• No harness. commerce-evals/SKILL.md: "The repo ships no eval harness; the suite is yours, because a case only means something against your catalog, orders, and fixtures." Gate behaviour is unit-tested with a scripted FakeClient; behavioural evals are described, not implemented.
• A case shape (JSON): id, priority, difficulty, tags, skip, a state precondition (seen ids, cart, memory, staged changes), turns, and expected with code-graded keys — calls_tool, calls_one_of, never_calls, first_tool, first_tool_not, ui_components, no_ui, cart_contains / cart_not_contains, staged_change_kinds, no_applied_changes, memory_contains / memory_not_contains, skill_loaded / no_skill_load, reply_includes / reply_omits, max_tool_calls — plus one rubric for a judge.
• Authoring rules worth adopting verbatim: every positive case has a negative twin; grade final tool arguments and resulting state, not wording; assert the route only where the route is the behaviour (a grounding read first, a write that must never happen), otherwise calls_one_of; a rubric is one PASS and one FAIL condition that no answer satisfies both of, names the fixture fact that decides it, says nothing about tone or length.
• Judge rules: one judge call per dimension, temperature zero, structured verdict + reason, transcript truncated from the start, and a fingerprint of judge model + rubric stored with every verdict so a rubric change invalidates old scores; an unparseable judge reply is a judge failure, not an agent failure.
• Run pattern: regression set on every merge over several trials with a pass threshold; targeted set while changing one flow; everything on a model change; a judged sample of live traffic in production. Diff failure sets, not toplines — a one-point topline move between live runs is noise.
• Poisoned fixtures: eval-only listings/reviews carrying injected instructions, each with a code-asserted negative (never_calls, no_applied_changes, memory_not_contains) and a benign should-serve counterpart so a refuse-everything agent still fails.

What we have

src/lib/benchmark/ (~1,300 LoC) is a real harness, exposed through benchmark_list_scenarios, benchmark_list_runs, benchmark_get_run, benchmark_start_run and the Benchmark module UI:

• Seven scenarios in scenarios.ts mirroring real workflows (video storyboard, quarterly tax report, invoice canvas, project tasks, document brief, email draft, calendar planning), each with buildPrompt(marker), extraSystemPrompt, allowedToolNames, expectedTools[] (tool set + minCalls + points), judgeRubric, timeoutSeconds.
• runner.ts executes each scenario as a live agent run against the real workspace, scores expected-tool checks, run completion, tool reliability, and an LLM judge (0–10 mapped to BENCHMARK_JUDGE_POINTS), and persists benchmark_runs / benchmark_results with a per-model leaderboard.

Overlap: prompt-driven scenarios, tool-call expectations, allowlists, LLM judge, per-model comparison. This is the harness the blueprint tells its readers to write. Do not build a second one.

The gaps the case shape exposes

Priority, difficulty and skip tags are cheap and make the failure-set diff readable.

Recommendation and effort

Take the concepts, not the code. Reasons the code does not transfer: Python 3.11 / pydantic / anthropic SDK against our Node ESM runtime; gates are typed against MerchantSessionState, StagedChange, Listing; three of the seven gate checks are commerce-only (options/variants, promotion depth, campaign provenance); the loop assumes a single Messages API client, while ours must behave identically across six provider loops and the CLI bridges. Porting would mean rewriting every line anyway, and the repo is unmaintained, so there is no upstream benefit from staying close to it. The exception is fencing.py, whose character-level sanitizer is provider-agnostic and short.

Effort in person-days (PD), assuming one engineer, including docs, agent tool contracts, DE/EN/IT strings, and the three clients where a surface exists:

WP1 — Generic staged-action approval gate (6–9 PD)

• 1a. Typed blocked outcome (1 PD). Add status: "blocked" and reason: <gate> to the tool-end event and the proxy result envelope, next to the existing is_error. Emit it from the module/scope/shell refusals and the fleet gate today. Teach detectFailedSideEffects and the completion judge evidence builder that a blocked call is not a failed one.
• 1b. Settled outcomes for interrupted writes (1 PD). When a steer, abort or restart cuts a round whose side-effecting tool already returned, carry the real result into the replayed history instead of the "invalid trailing call" marker, so the model does not repeat a completed send. Mirrors close_open_tool_uses(settled=…).
• 1c. agent_staged_actions (2–3 PD). Migration and API: id, run_id, session_key, user_id, tool_name, action_hash, args_json, preview_json, status: staged | approved | applied | discarded | expired, created_by_kind: user | agent, applied_by, timestamps. Generalize createFleetApprovalToken / consumeFleetApproval: the mark is written only by the approve route, consumed immediately after the apply attempt, bound to the action_hash. The tool handler for a gated tool without a valid mark returns blocked / approval with the staged id and a one-line instruction to tell the user it is waiting — never an error.
• 1d. Gate the four external actions (1 PD). emails_send_draft, social_media_publish_post, outbound channel send to a non-allowlisted recipient, and a placeholder policy entry for payment/transfer tools. Per-tool switch in Settings → ClapilotAICore → Runtime (app_settings, default on for autonomous runs, off for interactive chat where the user is watching), following the existing runtime-flag pattern.
• 1e. Approval surface (1–3 PD). Web: a "Freigaben" list on the run/chat timeline and in the Dashboard with approve/dismiss circular actions; iOS/macOS: the same list in the run detail; channels: the existing agent_channel_approvals reply flow reused. Approving applies without a model call and posts the result back to the run.

WP2 — Narrow provenance rule (1–2 PD, optional, after WP1)

Apply accepts only staged ids the session created or listed, and stage refuses draft ids the session never read (emails_get_message / social_media_get_post first). Implemented as a per-session seenIds map on the run context, not a global ledger.

WP3 — Benchmark harness upgrades (4–6 PD)

• Extend BenchmarkScenarioDefinition with neverCalls, firstTool, maxToolCalls, state preconditions (seeded records the scenario references), priority, difficulty, skip.
• Add negative twins for the existing seven scenarios and three poisoned-fixture scenarios (injected instruction in a document, a mail body, and a web page) with code-asserted negatives and a benign counterpart each.
• Switch the judge to a PASS/FAIL rubric per dimension with a stored judgeModel + rubricHash fingerprint; keep the 0–10 score as a secondary metric.
• trials per run and a failure-set diff view against a chosen baseline run in the Benchmark module, replacing "score moved by 0.4" as the signal.
• Wire the staged-action scenarios once WP1 lands ("stage, wait, approve, apply" and "never applies without a mark").

WP4 — Fencing hardening (1 PD)

Port the NFKC normalization, invisible- and control-character stripping, forged turn-boundary neutralization (blank line + Human: / Assistant: / System:), transcript and tool-call tag removal (<function_calls>, <tool_result>, <|…|> special tokens, run to a fixpoint so nested copies cannot reassemble), and fence-marker copy removal from commerce_common/fencing.py into tool-result-quarantine.mjs as a normalization step before the instruction-shape scan. Add the Apache-2.0 attribution to THIRD_PARTY_NOTICES.md as done for Cortheon. Test with the existing quarantine unit tests plus the blueprint's own hostile cases.

Sequencing

WP1a → WP1c/1d → WP1e → WP4 in the first iteration (the gate is the real gap); WP3 in the second so the gate has end-to-end evals; WP2 only if the gate's first weeks show apply calls with ids the session never saw.

Explicitly not recommended

• Eager mid-stream tool dispatch (EagerDispatcher): latency optimization with a large blast radius in executeToolBatch.
• Forced grounding via tool_choice: commerce-shaped.
• The merchant guardrail catalogue, staging follow-through reminder, and presentation-component enrichment: commerce-shaped.
• Adopting any package from the repo as a dependency: unmaintained, Python.

Sources

• Repository: https://github.com/anthropics/commerce-agents (commit fd4d592, Apache-2.0, "reference implementation; not maintained").
• Files read for this review: README.md, docs/safety.md, commerce-common/commerce_common/{turn,execution,fencing,streaming,testing}.py, merchant-agent/core/merchant_agent/{gates,changes,types}.py, merchant-agent/runtime-messages-api/merchant_agent_runtime/orchestrator.py, merchant-agent/runtime-agent-sdk/merchant_agent_sdk/merchant_tools.py, examples/demo_common/merchant.py, plugins/commerce-builder/skills/commerce-evals/SKILL.md.
• Clapilot counterparts: services/clapilot-agent/src/sessions/index.mjs, services/clapilot-agent/src/providers/index.mjs, services/clapilot-agent/src/tool-result-quarantine.mjs, services/clapilot-agent/src/autonomous-completion.mjs, services/clapilot-agent/src/channels/index.mjs, src/lib/agent-runtime/tool-proxy.ts (fleet approval, mail/social handlers), src/lib/benchmark/{scenarios,runner}.ts.