GitHub Traffic Control

This page describes how Clapilot routes GitHub REST API traffic and how rate limits are handled. It is for developers touching Issue Reporter, Hub issue approval, or Agent Orchestrator GitHub calls. Part of this design is shipped (the shared githubTrafficFetch helper and the write queue table); the gateway budgets, queue drain worker, and shared webhook ingestion are still planned and are marked as such below.

Architecture Decision

Clapilot will route internal GitHub REST API calls through a central GitHub Traffic Control layer. The first implementation scope is intentionally narrow:

• normalize GitHub rate-limit detection for Issue Reporter and Hub approval writes
• log GitHub rate-limit headers from the shared web helper
• queue Hub approval issue-creation writes when GitHub returns a primary or secondary rate limit
• document the current GitHub call inventory and rollout path for Agent Orchestrator traffic

The long-term gateway owns request budgeting, read caching, write queues, and webhook ingestion. Existing Agent Orchestrator code already has a local githubApiFetch wrapper with cooldown and poll caches; that should be folded into the shared gateway instead of expanded as a second permanent abstraction.

Current Call Inventory

Gateway Contract

Every product GitHub request should record:

• method and normalized endpoint
• actor, for example issue_reporter, hub_approval, agent_orchestrator_pr_review, or supervisor
• token or installation identity without storing the secret
• x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-reset, x-ratelimit-used, x-ratelimit-resource
• x-github-request-id
• response status and rate-limit classification

Rate-limit errors must map to a user-facing 429 or queued response, not a generic 502. The message should include the reset timestamp and GitHub request id when available.

Queue

github_traffic_control_queue stores write work that should not be dropped on rate limit. The first queued action is hub_issue_create; it records the Hub issue id and reviewer in the JSON payload with status waiting_for_rate_limit.

Queue states are:

• pending
• waiting_for_rate_limit
• failed_permanent
• completed

Status: enqueueing works today (enqueueGithubWrite in src/lib/github-traffic-control.ts), but the drain worker is not implemented yet. A follow-up worker should drain due rows by priority, call the same gateway, increment attempts, and mark permanent failures only after non-rate-limit errors or an attempt cap. Until that worker exists, queued rows stay in the table and need manual follow-up.

Request Budgets

Priority order:

1. user-triggered writes
2. active coding/debug jobs
3. user-visible reads
4. background sync and supervisor polling

When remaining budget is low, background polling should skip or extend its interval before user-triggered actions are affected. Agent Orchestrator already caches some poll reads; the next slice should add a shared budget check before runSymphonyPollTick scans GitHub repos.

Read Caching Strategy

Recommended cache windows:

• repo metadata: 10 to 60 minutes
• issue and PR metadata: 60 to 120 seconds
• check status: 30 to 90 seconds
• comments and reviews: 60 to 180 seconds
• workflow logs: explicit request only, or fetched when a failed check requires diagnostics

In-flight request coalescing should key by token identity, method, endpoint, and query params so parallel agents share the same read result.

Webhook Follow-up

Today, GitHub webhooks are received only by the Agent Orchestrator module (public module endpoint /api/modules/agent-orchestrator/webhook); there is no shared product-level webhook route yet. The planned shared gateway should add a GitHub webhook endpoint for:

• issues
• issue_comment
• pull_request
• pull_request_review
• check_run
• check_suite
• workflow_run

Webhook events should invalidate gateway caches and enqueue agent work. Polling remains a fallback and supervisor recovery path, but it should not be the primary freshness mechanism for active repositories. Agent Orchestrator now also uses completed failing workflow_run, check_run, and check_suite webhook payloads to start a configured default-branch CI fix run that opens a PR, while PR-branch failures continue through tracked PR follow-up.