Agent Orchestrator

Regression tasks whose description contains an exact session.*.failed production signature are not marked erledigt immediately after an implementation run. Symphony moves them to warten and persistently waits for the linked pull request to be merged. The production observation window starts at that merge timestamp, lasts 24 hours by default (SYMPHONY_REGRESSION_OBSERVATION_MS), and survives orchestrator restarts through the durable job record. Jobs with pending observations are protected from the 12-hour job TTL cleanup. Symphony only completes the task when the signature stays absent for the full post-merge window. A recurrence or an unmerged PR closure returns the task to offen.

What it does

Agent Orchestrator (agent-orchestrator) runs coding agents against Git repositories and monitors them from one place. It starts detached coding-agent jobs (codex, claude, clapilot-code), hosts interactive agent sessions backed by the native ClapilotAICore broker, and keeps a small compatibility fallback for older job paths. Clapilot-code is Clapilot's own in-process coding loop; Codex and Claude remain external harnesses. GitHub and GitLab repositories share the same repository picker and automation matrix. The runtime reviews pull requests or merge requests, implements issues on a fresh branch, replies to mentions, follows review feedback, and fixes default-branch CI failures.

How to open / enable it

• Agent Orchestrator is a Developer-mode module: the web module and the Settings -> Agent Orchestrator surface are hidden while Developer mode is disabled (src/lib/module-store/developer-mode-modules.ts).
• With Developer mode enabled, open it from the module menu; the route is /modules/agent-orchestrator. GitHub automations and runtime settings live under Settings -> Agent Orchestrator (src/app/(app)/(einstellungen)/settings/agent-orchestrator/page.tsx).
• Manifest: bundled-modules/agent-orchestrator/module.json (slug agent-orchestrator, entry index.html, renderer react, icon bot). The React UI is src/components/modules/agent-orchestrator-module.tsx, hosted by src/app/(app)/modules/[slug]/page.tsx; bundled frontend assets remain in bundled-modules/agent-orchestrator/ but the app renders the module directly in the main React tree.
• Auth for the module API: web users authenticate with the clapilot_session cookie; legacy compatibility skills/scripts use a short-lived machine Bearer token from POST /api/auth/agent/system-token; remote runners use Hub HMAC auth with the shared CLAPILOT_HUB_SHARED_SECRET.

Key workflows

Start a detached coding job

Use jobs when the user wants a detached run such as "go implement this", "prepare a PR", or "review this repo".

• + New Agent opens a compact chat-style composer. With Local selected it creates an interactive workspace session; its footer contains attachment, repository, icon-prefixed model, optional speech-to-text, and send controls. The model menu combines available Codex, Claude, and Clapilot Code models and infers the harness from the selected model.
• When at least one online Codex runner is available, the web composer adds a runner selector between repository and model. Auto lets any online runner claim the work, while choosing a named runner pins it to that machine. A remote selection forces the Codex provider and creates a decoupled coding job instead of an interactive session; the queued job remains pinned even if that runner later goes offline.
• The Apple client's Neuer Agent surface mirrors the same session-first contract and single-row footer. It hides the generic composer emoji and keyboard-hint controls so repository and model selection remain visible without a second selector row.
• Detached and remote jobs remain supported for the web New Agent composer, automations, GitHub observers, remote runners, API/tool calls, existing activity rows, and job follow-ups.
• Codex-backed detached jobs keep an internal linked orchestrator session, so the same background run can be resumed later with a follow-up prompt from the web module, the native Apple client, or agent tools. Plain CLI jobs without a linked session, including Claude CLI jobs, can also receive follow-up prompts; those run in the original job workspace and append their output to the same job log. Remote Codex jobs requeue follow-up prompts for the remote runner and reuse the remote job workspace.

Model selection

• provider=clapilot-code runs through Clapilot's in-process coding loop instead of the Codex app-server or Claude CLI. It reuses the native embedded_pi adapter internally. The web creation UI shows a model picker backed only by usable non-subscription entries in the configured Provider & Modelle catalog; session creation fails clearly when none are available instead of silently falling back.
• pi, embedded_pi, embedded-pi, and clapilot_code remain accepted input aliases for clapilot-code. New jobs, session metadata, and automation settings use the canonical clapilot-code provider value.
• The web Neuer Agent panel shows model pickers for claude and codex too, backed by GET /coding-models. The picker preselects the model a job would run with by default (runtime config override, then the provider catalog default). Options merge the configured agent_provider_configs catalog with the current CLI-native model lineup, so models the CLI accepts stay selectable even when the Clapilot catalog only lists a subset.
• The current Codex lineup includes gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.4, gpt-5.4-mini, and the subscription-only gpt-5.3-codex-spark. Spark is a current selectable model and is not normalized through the legacy GPT-5.3 fallback rule.
• Detached Claude jobs always resolve to a concrete model: an explicit request wins, then the Settings -> Agent Orchestrator Claude runtime model, then the configured Anthropic provider catalog default. The current built-in fallback is claude-opus-5; the resolved ref (for example claude-default/claude-opus-5) is stored on the job and shown on activity cards, matching interactive sessions.
• Claude CLI jobs run with --output-format stream-json --verbose, so the job log streams live assistant text and tool activity while the job runs, and the job's model field is corrected to the model the CLI actually reports in its init event.
• The job follow-up composer (web terminal panel and Apple job detail) has a model picker limited to the running job's provider catalog, preselected to the model the job last ran with. The chosen model is sent as model on POST /jobs/{id}/follow-up; detached CLI follow-ups re-run the CLI with that model, linked-session follow-ups forward it as a per-turn model.
• The Codex app-server bridge resolves the Codex service tier for each model. Codex-auth OpenAI providers can set a per-model Codex-Tier (Global, Fast, Flex, Aus) in ClapilotAICore settings; these values are stored as provider metadata under modelServiceTiers. The model picker reads Codex app-server model/list and uses the returned speed-tier metadata for the per-model tier menu. Models left on Global use CLAPILOT_AGENT_CODEX_SERVICE_TIER when it is set; otherwise the tier stays unset and Codex chooses.
• Agent Orchestrator runtime settings accept max and ultra reasoning in addition to the older effort values. Codex currently advertises max for the GPT-5.6 family and ultra for Sol and Terra; unsupported model/effort combinations remain subject to the Codex runtime's own validation.

Run interactive sessions

Use sessions when the user wants to stay in the same thread/workspace and continue iteratively:

• resumable repo conversations
• thread-bound Telegram/Slack/WhatsApp coding sessions
• live turn + terminal streaming
• fork/archive lifecycle

Interactive session adapters remain codex_app_server, acp_agent, and embedded_pi. provider=auto resolves to Codex unless an explicit adapter override is provided, provider=clapilot-code defaults to the internal adapter=embedded_pi, and an explicit provider=claude defaults to adapter=acp_agent. The acp_agent adapter remains scaffolded for local ACP-compatible sidecars/agents and is still experimental. The web + New Agent panel offers interactive execution for the production-ready Codex and Clapilot-code paths.

Monitor work in the module

The React module shows detached jobs, interactive sessions, and native runtime sessions from ClapilotAICore so admins can inspect normal Assistant chats, Clapilot-code coding sessions, and bridge runtime activity without switching to the dedicated Sessions diagnostics page.

The activity rail uses a canonical read-only thread projection over the existing job, orchestrator-session, native runtime-session, and remote-session records. The projection classifies each source as chat, coding, scheduled, system, or remote, preserves its original session/chat/job/external identifiers, and advertises capabilities such as conversation inspection, reply, diagnostics, workspace files, diffs, terminal, approvals, checkpoints, and Git actions. It does not migrate or replace stored sessions.

Normal web, group, and channel chats remain first-class entries in Sessions. Their persisted chatTranscript stays the authoritative conversation, while model runs, tool events, memory/context diagnostics, raw events, and metadata remain available in the detail inspector. Coding-only workspace capabilities are enabled only for repository/workspace-backed projections, so later Changes, Files, Terminal, Preview, and checkpoint panels cannot replace or hide ordinary chat history.

The Apple client applies the same projection on iPhone, iPad, and macOS. Selecting a native runtime chat loads its full persisted transcript instead of stopping at the list summary; supported chats can continue through the existing runtime-session turn endpoint, while fork, close, terminal, diff, checkpoint, and Git actions remain limited to session types that actually support them.

• The left rail is a dense, searchable activity navigator with counted filters for Coding Agents, Sessions, Alle, Remote Agents, and Analytics. Each row keeps category, provider, repository/session identity, title, latest output, resolved model, status, latest tool use, and update time scannable without competing with the selected-thread detail pane.
• Selected-thread headers lead with the human-readable session/channel title and retain the technical session id as secondary metadata. This keeps normal chat inspection understandable without removing diagnostic identity.
• Detached-job details use the same compact identity hierarchy as session details: task title, repository/model context, provider mark, status dot, and an optional stop action. A segmented detail control separates Conversation, Activity, Terminal, Changes, and Files; changed files and diffs never render inline in chat history. Do not add a Terminal/Prompt switch, a second "Current Terminal" title, explanatory subtitle, status badge, or always-visible job/workspace metadata row above it. The original task remains visible as a user message.
• Coding Jobs shows detached coding jobs plus repository-backed interactive and runtime sessions, so coding work stays grouped with repository context.
• Session Jobs bundles chat- and runtime-backed session work: normal Assistant sessions (runtimePath=native), native Clapilot-code coding sessions (runtimePath=embedded_pi), and native sessions whose active execution harness resolves to the Claude CLI Bridge. Hidden follow-up sessions that belong to detached coding jobs are intentionally excluded; they stay attached to their owning Coding Job detail view.
• Alle mixes session-backed work and detached coding jobs in one rail.
• Session overview requests are summary-only by default: they carry compact status/model/activity previews but omit per-session event and chat-history arrays. The web module refreshes visible overview lists every 30 seconds, skips hidden-tab refreshes, and suppresses overlapping requests. Full history is loaded only for the selected session.
• Remote Agents switches the rail to a remote-machine dropdown, then lists that machine's local Codex CLI and clapilot-code sessions from the latest runner heartbeat. Clapilot Code rows carry a harness label. Selecting a remote session opens its metadata and transcript once the runner has answered the detail request in a later heartbeat. A composer at the bottom creates a decoupled resume job pinned to that machine, so replies queue even while the runner is stale or offline and continue through the matching CLI harness when a compatible runner reconnects.
• Analytics opens the internal Agent Costs & Yield dashboard for repo-bound Orchestrator runs. The first MVP derives run/yield data from existing jobs and interactive sessions, including repo, issue/PR context when detectable, provider/model, status category, duration, and simple waste hints such as failed runs or completed runs without a detected PR. Cost and token fields are explicitly marked as unavailable until reliable provider usage data exists; this analytics view does not render raw logs.

Detail-pane behavior:

• Session-backed details preserve each data mode without changing the underlying records. Conversation is the default and shows only chronological user/assistant turns, Activity isolates tool calls, terminal output, model runs, and lifecycle events, and Details exposes identifiers, runtime metadata, and diagnostics. Local repository-backed coding sessions additionally expose capability-gated Terminal, Changes, and Files modes. Terminal isolates command cards and raw terminal deltas already emitted by the agent runtime; Changes shows the current branch, Git status, staged diff, and working-tree diff; Files shows a bounded Git-ignore-aware relative inventory. Normal chats never receive these workspace modes.
• Workspace inspection is lazy-loaded from GET /sessions/{id}/workspace or GET /jobs/{id}/workspace. Each local job records its starting Git commit; Changes compares the current checkout against that baseline, so committed and pushed agent edits remain visible instead of appearing clean. Before an automation checkout is removed, the Orchestrator persists an immutable final workspace snapshot and serves it to both the owning job and linked session. Historical PR jobs created before snapshot support can recover a read-only changed-file view from the GitHub pull-request files API. Ownership and trusted-root checks still match workspace file references, and remote-runner workspaces remain unavailable. The Terminal inspector is read-only: it reuses the selected session's persisted and SSE-streamed command/terminal events and does not expose the unrestricted Developer-mode admin terminal or introduce an arbitrary-shell endpoint.
• Local coding sessions expose an Approvals mode on web and Apple. Codex app-server runs use approvalPolicy=on-request, approvalsReviewer=user, and workspace-write; command and file-change server requests remain pending in the native broker until the owner explicitly approves or declines them through GET/POST /sessions/{id}/approvals. The same mode can create bounded read-only workspace checkpoints through GET/POST /sessions/{id}/checkpoints; checkpoint data is stored in Clapilot runtime state outside the repository and never mutates or rolls back the worktree. Preview, checkpoint restore, and mutating Git actions remain unavailable until their separate guarded contracts are implemented.
• The Changes inspector exposes guarded Git actions through POST /sessions/{id}/git-actions and the equivalent job endpoint. Stage/unstage accepts one validated repository-relative path, commit accepts only a bounded message and requires staged changes, and push is fixed to a non-force origin push of the current branch. Every action rechecks the owner/trusted workspace plus the branch and HEAD the user reviewed; stale requests are rejected. Web uses a two-click push confirmation, and Apple mirrors the same controls. Destructive discard/reset, force push, arbitrary remotes/refspecs, and checkpoint restore are intentionally unsupported.
• Wrapper preferences persist without changing chat semantics: web retains the activity filter and resizable detail width in local storage, while Apple retains the activity filter in AppStorage. Selecting any other session still resets the detail mode to Conversation, so a previously inspected Terminal/Changes/Approvals pane can never hide a normal chat transcript. Web exposes Cmd/Ctrl+K for activity search, Cmd/Ctrl+Shift+N for a new agent, and Escape to clear search; macOS mirrors the search and new-agent shortcuts.
• Runtime conversations render oldest-to-newest across providers. Reply composers are available only in Conversation, preventing prompts from being entered while inspecting diagnostics; detached jobs retain their existing prompt/terminal workflow.
• Interactive Codex sessions render a typed timeline instead of a flat log wall: assistant replies are unboxed markdown prose, user messages are compact trailing bubbles, and Codex tool/command lifecycle items are quiet expandable rows. Tool details expose command/cwd, duration, exit code, output, and raw payload only on demand; unmatched low-level output is collapsed behind a bounded Terminal output disclosure.
• Multi-turn sessions remain one continuous chronological conversation. Turn IDs, event counts, provider avatars, and timestamps are not persistent section headers; timestamps stay secondary, while user prompts and assistant prose establish the turn boundaries naturally.
• Detached Codex jobs run codex exec --json. The job log preserves the legacy source and message fields and additionally carries typed eventType, itemType, itemId, turnId, itemStatus, item, and usage fields. Web and Apple render agent_message, command_execution, file_change, MCP/dynamic tool, and web-search items directly instead of classifying CLI stdout heuristically.
• Detached-job detail views use the same visual language on web and Apple: task and follow-up prompts appear as user messages, assistant text renders once as prose, the newest two tool actions stay visible, and older tool calls fold behind a +N previous tool calls disclosure. Routine runtime initialization/status chatter stays out of the primary transcript. The currently open job/session follows its SSE stream, preserves structured event fields across reconnect replay, and reconnects after transient transport failures. When a job is linked to a canonical session, the linked transcript owns user/assistant prose and job logs contribute only non-conversation activity, preventing duplicate answers.
• Repository-backed web timelines show a compact changed-files card with file count and diff totals; View diff opens the existing guarded Changes inspector. The composer remains anchored below the scroll content as a rounded floating surface, with attachments and the provider-scoped model selector inside its footer.
• Bridge runtime tool calls (tool.start / tool.end rows from agent_events, e.g. Claude CLI Bridge and Codex bridge runs) are reconstructed into the same expandable tool rows: calls are paired by toolCallId, labeled with the tool name plus an args-derived summary, and expose the executed command ($ …), result preview, and full raw JSON payload on demand. Older consecutive tool calls collapse while the newest actions remain visible. run.postprocess.* chatter is hidden unless it failed. The Apple client mirrors the same behavior.
• Clapilot-code entries use the same oldest-to-newest transcript direction as Codex and Claude detail panes, auto-scroll to the newest turn, and include a footer composer for follow-up prompts into the same runtime session. Raw runtime events and metadata stay hidden behind the provider icon in the pane header.
• Claude bridge entries expose attached model/tool runs plus the persisted Claude harness session id. The module footer can send follow-up prompts back into the same Claude bridge session; if the Claude CLI process is still running, the prompt is steered into the live stream, otherwise a new runtime turn resumes the persisted Claude session.
• The read-only native session detail pane shows readable timestamps on each model run and nests per-run tool usage events inside the corresponding turn/run card. When no classic chat_nachrichten transcript row exists for a native runtime session, the pane reconstructs the visible session transcript from the persisted run history instead of showing an empty transcript block. For native runtime runs that used a Mixture-of-Agents (MoA) preset, the run view also lists the reference model outputs (per-reference provider/model and response) alongside the aggregated answer.
• Interactive sessions continue directly from the terminal pane footer (no separate lower controls composer). Detached jobs expose a follow-up composer directly in the job detail pane: linked-session jobs continue the same hidden session, while plain CLI jobs continue in the same workspace and append output to the job log. The follow-up composers reuse the floating-chat input styling, minus voice/model controls, and support the same inline image/file attachments for first and follow-up turns. The new-agent composer carries repository and inferred harness/model selection plus optional speech-to-text in one footer row.
• User-owned local coding sessions and local job follow-ups support workspace file references in the composer. Type @ (or use the @ action), search the active working directory, and select a relative path. The request carries the selected paths as fileReferences[]; Codex, Claude CLI, and Clapilot-code are instructed to inspect those files relative to their active cwd without uploading or embedding the file contents. New-agent drafts have no checkout yet, legacy ownerless sessions are not claimed implicitly, and remote runner workspaces are not server-local, so the picker is intentionally unavailable in those cases.
• The web module keeps the selected interactive or linked-job session attached to GET /sessions/{id}/stream?replay=0 even while the session is ready or waiting for another turn. The initial SSE snapshot supplies recent history once; replay=0 prevents the same historical events from being emitted a second time during reconnects. New events then update the open detail pane without overview polling.
• Selected Clapilot-code and Claude bridge entries subscribe to the SSE stream of their linked external session and throttle runtime-detail reconciliation while events are arriving. Legacy runtime rows without a linked external stream, or rows whose persisted stream is no longer available after a runtime restart, use a 15-second selected-detail fallback poll instead of rapid SSE reconnects.
• The Apple client subscribes to the same SSE endpoints (GET /jobs/{id}/stream and GET /sessions/{id}/stream) for the selected activity item. Session list calls receive the new compact summary view by default, while streamed job logs and linked-session events remain the live detail path.

Run jobs on remote machines

• Detached jobs can set executionTarget: "remote" with provider codex or auto, and optionally remoteRunnerId to wait for a specific registered machine; otherwise any online Codex runner can claim the job. The web + New Agent panel exposes local execution, automatic remote claiming, and pinned remote runners through its runner dropdown.
• Remote execution is pull-based: scripts/agent-orchestrator-codex-remote-runner.mjs runs on the remote machine, connects outbound to Clapilot with CLAPILOT_HUB_SHARED_SECRET, claims queued Codex work, checks the repository out from a local shared bare cache into a per-job worktree, runs codex, and reports logs/status back to the same job stream.
• The runner includes a read-only CLI-session snapshot in its heartbeat. It scans the remote machine's Codex home (CLAPILOT_REMOTE_RUNNER_CODEX_HOME, CODEX_HOME, or ~/.codex) for state_*.sqlite thread metadata and Codex CLI/Desktop history files (session_index.jsonl plus sessions/**/rollout-*.jsonl). It also scans ~/.clapilot-code/sessions/*.jsonl, or CLAPILOT_REMOTE_RUNNER_CLAPILOT_CODE_SESSIONS when configured. Both harnesses are exposed per machine through GET /remote-runners/{runnerId}/codex-sessions; selecting a session requests transcript detail, which the runner includes in a later heartbeat, so detail loading is asynchronous and outbound-only.
• Replying to a scanned Codex session requires the Node runner codex-remote-runner/0.2.3 or newer and runs codex exec resume with the explicit Codex home. Replying to a scanned Clapilot Code session requires codex-remote-runner/0.2.4 or newer and runs clapilot-code exec resume. Both paths use the scanned cwd when it still exists, without cloning or creating a repository worktree. The macOS Swift runner does not claim session-resume assignments.
• macOS machines can instead run the signed Clapilot Remote Runner menu-bar app. Its popup has an Overview / Connection / Log segmented control: Overview shows live worker stats, active sessions, and recent finished sessions (succeeded/failed/cancelled) with timing; Connection holds the Clapilot URL, Hub shared secret, runner id/label, workspace path, polling interval, slot count, and the Save/Test heartbeat/Quit actions; the secret is stored in Keychain. Start/stop and heartbeat status stay in the popup header and status bar.
• Settings -> Agent Orchestrator lists registered remote machines with online/stale status, slots, capabilities, version, running jobs, and last heartbeat.
• The Clapilot server does not need SSH access or an inbound port on the remote machine. Treat connected runners as trusted, because repo jobs receive a GitHub token for cloning private repositories.

GitHub and GitLab automations

Configure named GitHub and GitLab accounts in Settings -> App Verbindungen, then select repositories in Settings -> Agent Orchestrator. GitLab connections accept GitLab.com or a self-managed instance URL. The settings page and the New Agent composer show one combined repository list; each row carries its GitHub or GitLab suffix icon and retains the source connection so same-named repositories do not collide. The picker caches the last loaded union locally, and a refresh button remains available.

The repository automation matrix selects a concrete model independently for PR review, issue implementation, and mention handling on every enabled repository. The picker reuses the same Codex, Claude, and Clapilot Code model catalog as New Agent; users select only the model, and the model's provider determines the coding harness. Main-CI failure fixes and Symphony coding tasks dispatched from that repository's mapped task board use the repository's issue-implementation model. Existing rows without model metadata remain backward-compatible with the former global provider defaults until the configuration is saved, at which point enabled rows persist concrete { provider, model } selections as prReviewModel, issueObserverModel, and mentionObserverModel inside agent_orchestrator_repo_automation_config.

For each repository you can independently enable:

• automatic pull-request or merge-request reviews
• automatic issue observer runs
• automatic mention replies
• default-branch CI failure fixes from GitHub or GitLab webhooks

Trigger mode is configured per repository as polling or webhook. Webhook rows show a repo-specific tokenized URL for the matching GitHub or GitLab project webhook; polling rows run from the interval loop. GitLab merge-request, issue, note, pipeline, and push hooks are normalized to the same scoped automation families as their GitHub equivalents.

Integrations and providers:

• Each repository retains its named GitHub or GitLab connection. Review scans, checkout, review submission, issue assignment, follow-up tracking, mentions, and CI inspection all use that same provider-specific connection.
• GitLab parity covers nested group paths, GitLab.com and self-managed base URLs, merge requests, issues, notes/discussions, approvals, pipeline jobs, emoji reactions, and project webhooks.
• Legacy github_token / github_pr_review_token values still exist as compatibility fallbacks, but the intended admin flow is named GitHub integrations.
• GitHub automations can select codex, claude, or local clapilot-code; the OpenClaw compatibility fallback is not used for review, implementation, or mention-reply submission.

PR review: scans non-draft open PRs, deduplicates by repo + PR number + head SHA, and derives a GitHub review decision from the generated findings:

• REQUEST_CHANGES when any finding is tagged critical or the review explicitly asks for changes
• APPROVE when the review reports no issues or only medium / minor findings
• COMMENT remains the fallback when the review output cannot be classified

Issue observer: assigns the issue to the configured GitHub identity when a coding run starts, comments that implementation has started, and closes the issue immediately after pickup so it cannot be picked up by another observer run. On success it persists the issue-to-PR link and comments the PR link back onto the issue. If another active implementation session already owns the same issue, the duplicate pickup is treated as already in progress and the issue stays closed. If implementation fails and no PR can be recovered, it reopens the issue and records the failure with the normal retry backoff. Codex-backed issue implementations default to interactive sessions so the same session can be resumed for later PR follow-up work. Issue observer settings include an Agent Goal toggle: when enabled and the implementation provider resolves to Codex or Claude, implementation and PR creation runs start with a /goal command derived from the issue task. PR creation prompts require PR bodies to include a closing issue reference plus a ## Testing Instructions section; custom issue-observer prompts still receive this mandatory PR body contract after template expansion.

Main CI failure fix: webhook-only. When enabled for a webhook-mode repository, completed failing workflow_run, check_run, or check_suite events on the repository default branch start a coding run from that default branch. The run must push a branch, open a PR, include testing instructions, and then enters the same tracked-PR follow-up loop as issue-observer PRs. Pull-request branch failures stay on the existing tracked-PR follow-up path.

Tracked PRs: PRs opened by orchestrator-managed coding runs or interactive sessions are tracked durably in the database until they merge or close.

• If an issue observer run fails after a linked PR was already opened, or a later manual/chat run opens a PR that closes the watched issue, the next issue scan recovers that PR into the same durable tracked-PR follow-up state instead of leaving it unmonitored.
• While a tracked PR remains open, the orchestrator watches for merge conflicts, failing GitHub checks, PR review comments, top-level PR review bodies, GitHub Actions CI report comments, and mention activity on the open PR branch. Failing check follow-up is not gated only by the PR thread updated_at timestamp, so edited GitHub Actions smoke/e2e reports still retrigger the tracked PR worker when the check run fails again.
• When a tracked PR still has an owning interactive session, follow-up events are redirected into that same session as another turn so the original agent keeps its context.
• If no usable session exists, CHANGES_REQUESTED reviews, merge conflicts, and critical comments trigger an immediate follow-up run, while non-critical comments keep the guarded triage fallback. Triage can choose reply for direct questions, status requests, or clarification comments and post a GitHub PR conversation reply without changing code; it chooses follow_up for small safe code fixes (which can also post a result-specific PR conversation reply after updating the branch); ignore is reserved for noise or duplicate comments.
• Tracked PR status exposes recent follow-up trigger outcomes and failures for debugging (lastScanResults, recent handled triggers, recent failures).
• Active tracked-PR turns are checkpointed before agent execution. After a runtime restart the original trigger is restored once with the explicit requeued_after_restart status and dispatched with the same composite idempotency key. Review/conversation replies carry a hidden operation marker, so replay after a crash reuses an already posted GitHub comment or GitLab merge-request note instead of posting it twice.
• Detached/manual jobs that explicitly implement a GitHub issue share the same issue-level lock as the issue observer: a second job for the same repo issue is rejected while an implementation session is active or once an open PR already closes the issue.

Mention observer: watches configured issue and PR conversations for @<github-login> mentions of the GitHub identity behind the configured token, adds a lightweight reaction, and then either redirects the mention into the tracked PR session or starts a target-specific session/workspace for that PR or issue.

Symphony task-board dispatch

• Symphony's poll loop is controlled by a persisted enable/disable flag in app_settings.agent_orchestrator_symphony_enabled. POST /orchestrator/start enables and starts the loop, POST /orchestrator/stop disables it and clears queued retries. GET /status includes enabled; disabled loops do not dispatch aufgaben candidates. The UI control lives under Settings -> Agent Orchestrator, not in the user-facing module view.
• The Symphony task-board toggle only controls aufgaben coding task dispatch; GitHub automations continue to run from the per-repository automation matrix even when Symphony is disabled or no task board is selected.
• Dispatch observes the configured default task board in app_settings.agent_orchestrator_symphony_task_board_id plus any repo-specific boards selected in the Settings GitHub automation matrix. If Symphony is enabled without a default board or a repo board mapping, no task dispatch happens.
• Tasks inside repo-specific boards inherit the mapped GitHub repository automatically. Tasks inside the default board still need a concrete repo:owner/name reference in the task description so the agent can safely clone the repository.
• Repository coding tasks dispatched from a mapped board resolve the same concrete issueObserverModel used for issue implementation, including its implied Codex, Claude, or Clapilot Code harness. The job record stores the requested provider/model and the provider-reported effective model remains visible in Agent Activity. Rows without a concrete model retain the legacy global issue-observer provider fallback.
• Symphony only appends mandatory branch/commit/PR instructions for explicit repository coding tasks. General aufgaben records such as document or payroll review tasks without a repo stay in the non-repository task path and must not create GitHub PRs.
• Symphony-created PR bodies must include both the Requested-by: Symphony task ... line and a Clapilot task: .../aufgaben/{id} link for traceability (using the configured public app URL, not internal container service URLs), plus a ## Testing Instructions section with concrete validation commands or manual checks. The coding job retains the originating task ID. After detecting the created PR, Symphony adds a system comment with its canonical URL, PR number, and open status to that task, localized from the task creator's persisted UI language (de, en, or it; private-board owner and German fallbacks). Separate PRs accumulate as separate comments; a typed UUID advisory lock and URL lookup run as separate statements in one transaction so retries and concurrent completion paths do not add the same canonical PR URL twice. Linking still runs when optional tracked-PR follow-up registration fails.
• Symphony tasks and repository automations share one resource-aware dispatch budget. A slot is reserved synchronously before asynchronous workspace preparation, preventing one poll from exceeding the configured limit. The safe default is one local run. The effective limit is also capped by total and currently free memory so coding workers cannot consume the runtime's safety reserve; GET /status exposes effectiveMaxConcurrentAgents and the capacity snapshot.
• Container bootstrap does not start Symphony by default. Set CLAPILOT_AGENT_ORCHESTRATOR_BOOTSTRAP_ENABLED=true only when a deployment should force the poll loop on during boot.

How the agent can drive it (tools)

Native ClapilotAICore tool contracts cover repository job orchestration and interactive session orchestration (services/clapilot-agent/src/sessions/index.mjs):

• agent_orchestrator_list_repos
• agent_orchestrator_start_job
• agent_orchestrator_list_jobs
• agent_orchestrator_get_job
• agent_orchestrator_follow_up_job
• agent_orchestrator_stop_job
• agent_orchestrator_start_session
• agent_orchestrator_list_sessions
• agent_orchestrator_get_session
• agent_orchestrator_send_turn
• agent_orchestrator_fork_session
• agent_orchestrator_close_session

Wiring notes:

• src/lib/agent-runtime/tool-proxy.ts forwards agent_orchestrator_* tool calls into this module API on behalf of the linked Clapilot user.
• Repository start tools forward the provider, named connection, instance URL, and clone URL returned by agent_orchestrator_list_repos; the module re-resolves the named connection and derives authenticated remote URLs server-side before using its token.
• services/clapilot-agent/src/orchestrator-sessions/index.mjs brokers interactive session lifecycle, Codex app-server integration, and event streaming into agent_events.
• Native/channel agent runs should use these tools for generic repo coding and PR-preparation work instead of legacy gateway execution paths.
• Website Canvas uses the same interactive session broker internally for iterative website edits, but keeps repo sync, preview, commit, and push ownership inside Website Canvas.

Configuration & limits

Repository checkout and cleanup:

• Server-side local automation keeps shared bare repository caches under .agent-orchestrator/repos and creates per-job/per-session Git worktrees under the existing job roots (manual-job-workspaces, github-automation-workspaces/*, and .agent-orchestrator/sessions).
• GitHub and GitLab tokens are supplied to Git fetches through temporary http.extraheader arguments instead of being persisted in remote URLs.
• Repository-backed Clapilot-code sessions keep the initiating Clapilot user id on initial and resumed turns, so user-scoped Agent Orchestrator tools remain available inside the linked coding session. Older sessions acquire and persist that binding when they are resumed from the signed-in module UI.
• Repository shells receive the selected credential only as ephemeral process environment: GitHub uses GH_TOKEN, GitLab uses GL_TOKEN and GITLAB_HOST, and Git push uses temporary process-local Git config. Named-connection and GitLab sessions use the embedded native runtime so credentials remain scoped to one session. Tokens are never written to the checkout remote URL, job log, or external-session metadata.
• Completed local automation worktrees are removed at job completion where possible, and the instance cleanup worker removes stale PR review, issue, main-CI, tracked PR, manual-job, and Symphony workspaces while protecting active work. Native interactive session roots are excluded because the session subsystem owns their resumable lifetime.
• The crash-recovery reaper runs at startup and hourly. AGENT_ORCHESTRATOR_WORKSPACE_TTL_HOURS controls the orphan TTL (default 24); an atomic PID lock prevents concurrent reapers. Lease state plus the in-memory queued/running inventory protects active workspaces, and terminal jobs with linked interactive sessions stay protected for follow-up turns. Paused or other non-terminal Symphony task states remain active; only configured terminal task states become eligible after the TTL.
• Every GC emits structured agent_orchestrator_workspace_gc telemetry with workspace counts and bytesBefore, bytesAfter, and bytesFreed. AGENT_ORCHESTRATOR_WORKSPACE_HARD_LIMIT_GIB optionally rejects every new local manual or GitHub-automation worktree and emits agent_orchestrator_workspace_backpressure once orchestrator storage reaches the configured limit.
• Run-local dependency trees are never copied between worktrees. Removing a worktree recursively removes its node_modules; package managers should use their host/container-level shared content store (for pnpm, the configured global pnpm store).
• Bare repo caches are retained for reuse and are pruned by age only when Git reports no linked worktrees. The default cache TTL is 336 hours.
• Remote runners keep their own sibling repos cache next to the configured workspaces directory and remove stale completed job workspaces after 24 hours by default.

Detached Claude auth:

• Detached Claude jobs resolve auth from the configured Anthropic provider in Clapilot's database before falling back to legacy app settings or local CLI state.
• Claude subscription auth uses the stored Claude setup-token and maps it to the Claude CLI internally; it does not rely on ANTHROPIC_OAUTH_TOKEN being injected via Docker .env.

Supervisor:

• An optional supervisor loop periodically reviews orchestrator state (for example stale runs) and can be triggered or toggled via GET|POST /supervisor/run, POST /supervisor/start, and POST /supervisor/stop.
• Its configuration is persisted in app_settings (agent_orchestrator_supervisor_enabled, ..._provider, ..._model, ..._interval_ms, ..._stale_run_ms); the interval floor is 1 minute and the stale-run floor is 15 minutes.

Optional env overrides:

• AGENT_ORCHESTRATOR_CODEX_ARGS
• CLAPILOT_AGENT_CODEX_SERVICE_TIER
• AGENT_ORCHESTRATOR_CLAUDE_ARGS
• OPENCLAW_PRIMARY_MODEL
• CLAPILOT_AGENT_ORCHESTRATOR_BOOTSTRAP_ENABLED (force Symphony on at container boot)
• SYMPHONY_MAX_CONCURRENT (configured local automation concurrency, default 1, maximum 8)
• AGENT_ORCHESTRATOR_MIN_FREE_MEMORY_MIB (memory reserve protected from new local automation runs, default 3072)
• AGENT_ORCHESTRATOR_RUN_MEMORY_BUDGET_MIB (estimated memory required per new local automation run, default 4096)

API endpoints exposed by module

Base: /api/modules/agent-orchestrator/api — implementation: bundled-modules/agent-orchestrator/api/handler.mjs; reference: bundled-modules/agent-orchestrator/README.md.

• GET /tools
• GET /auth
• POST /auth
• DELETE /auth
• POST /repos
• GET /status
• GET /poll
• POST /config
• GET /webhook/{token}
• POST /webhook/{token}
• POST /orchestrator/start
• POST /orchestrator/stop
• GET|POST /supervisor/run, POST /supervisor/start, POST /supervisor/stop
• GET /remote-runners
• POST /remote-runners/heartbeat
• POST /remote-runners/claim
• GET /remote-runners/{runnerId}/codex-sessions
• GET /remote-runners/{runnerId}/codex-sessions/{sessionId}
• POST /remote-runners/{runnerId}/codex-sessions/{sessionId}/follow-up
• POST /remote-runners/jobs/{id}/events
• GET /coding-models (model options plus resolved default for claude, codex, and clapilot-code; the Clapilot-code list contains only usable non-subscription catalog models)
• GET /jobs
• POST /jobs (supports an optional model for all coding providers — Clapilot-code direct provider models plus concrete Claude/Codex CLI models; executionTarget: "remote" plus optional remoteRunnerId remains Codex-only, and optional codexGoalEnabled enables Codex/Claude first-turn /goal bootstrapping)
• GET /jobs/{id}
• GET /jobs/{id}/stream
• GET /jobs/{id}/workspace-files?q={query}&limit={count} (local jobs only; linked jobs resolve the linked coding session cwd)
• POST /jobs/{id}/follow-up (supports an optional model override and validated relative fileReferences[] for local follow-up runs; remote jobs reject file references)
• DELETE /jobs/{id}
• GET /sessions
• POST /sessions (supports canonical provider: "clapilot-code" plus model; legacy pi/embedded_pi aliases normalize to Clapilot-code and use the internal embedded_pi adapter)
• GET /sessions/{id}
• DELETE /sessions/{id}
• GET /sessions/{id}/workspace-files?q={query}&limit={count} (returns bounded, Git-ignore-aware relative file suggestions for an owned local workspace)
• POST /sessions/{id}/turns (supports an optional per-turn model override plus validated relative fileReferences[], forwarded to the native session broker; stale or invalid selected paths reject the request)
• GET /sessions/{id}/stream
• POST /sessions/{id}/fork
• POST /sessions/{id}/archive

Troubleshooting

• The module is not visible: Agent Orchestrator is hidden while Developer mode is disabled; enable Developer mode first.
• Symphony is enabled but no tasks dispatch: check that a default task board (app_settings.agent_orchestrator_symphony_task_board_id) or a repo-specific board mapping is configured, and that default-board tasks contain a concrete repo:owner/name reference.
• A second job for the same GitHub issue is rejected: the issue-level lock is intentional — one implementation session or open closing PR owns the issue at a time.
• Remote job never starts: verify the runner heartbeat in Settings -> Agent Orchestrator (online/stale status) and that the runner shares the correct CLAPILOT_HUB_SHARED_SECRET; remote execution is outbound-only from the runner.