Clapilot Docs
Docs

RAG Implementation

Technical implementation of local document RAG in Clapilot.

RAG (retrieval-augmented generation) lets chat answers cite the user's own documents: instead of hoping the model remembers a file, Clapilot embeds document chunks up front and injects the best-matching chunks into the prompt at question time. Clapilot implements this local-first on PostgreSQL with pgvector (the Postgres extension for vector-similarity search), a queue-driven background indexer, and chat-time retrieval injection. No external vector database is involved.

INDEXING (background) dokumente change DB trigger enqueues job document_index_jobs polled by indexer worker extract + chunk embed via selected embedding provider document_ chunks RETRIEVAL (chat time) /api/chat question embed query text cosine search Mandant scope only thresholds top_k, min_score, per-doc cap RAG context in system prompt chunks searched at chat time

Components

  • Retrieval path: src/app/api/chat/route.ts + src/lib/rag/retrieve.ts
  • Background indexer: scripts/document-rag-indexer.mjs
  • Runtime bootstrap: entrypoint.sh
  • Database schema: db/migrations/012_rag_pgvector.sql
  • Admin control endpoints:
    • GET /api/admin/rag/status
    • POST /api/admin/rag/reindex

Data model

The RAG schema is implemented in Postgres:

  • document_index_jobs
    • Queue for indexing work (pending, processing, processed, failed)
    • Tracks attempts, errors, timestamps
  • document_chunks
    • Chunk text, metadata, and embedding vector
    • embedding vector(1536) for OpenAI text-embedding-3-small
    • HNSW index for vector similarity search

Indexing is enqueued automatically from dokumente changes by DB trigger.

Indexing flow

  1. A document insert/update in dokumente enqueues an index job.
  2. document-rag-indexer.mjs polls document_index_jobs.
  3. The worker resolves the document file from dokumente.file_path.
  4. Text is extracted/chunked and embeddings are generated.
  5. Existing chunks for that document are replaced atomically.
  6. Job status is updated to processed or failed.

Notes:

  • Missing files produce failed jobs with ENOENT details.
  • The worker is non-blocking for app startup and runs continuously.

Chat retrieval flow

/api/chat attempts retrieval before sending the prompt to the gateway:

  1. Build the query embedding through the embedding provider/model selected in ClapilotAICore -> Provider & Modelle (src/lib/rag/provider-embeddings.ts); if no embedding provider is configured, retrieval is skipped
  2. Resolve the active Mandant scope from the explicit request context or the current /mandanten/:id route
  3. Search document_chunks by cosine distance inside that Mandant scope only
  4. Apply thresholds (top_k, min_score, per-document cap)
  5. Inject [Clapilot RAG Context] as system message with source labels ([Q1], [Q2])
  6. Continue normal streaming response path

If no active Mandant is selected or no relevant chunks match, chat proceeds without document context (RAG remains enabled, but cross-Mandant retrieval stays blocked).

Runtime configuration

Main env toggles:

  • RAG_ENABLED (chat retrieval injection)
  • RAG_INDEXER_ENABLED (background indexing worker)
  • RAG_INDEXER_POLL_SECONDS
  • RAG_INDEXER_BATCH
  • DOC_VISION_FALLBACK_ENABLED (optional legacy/env override for document vision fallback; default is disabled unless explicitly enabled)
  • DOC_VISION_MODEL (optional legacy/env override for the selected vision model ref)
  • RAG_CHUNK_SIZE
  • RAG_CHUNK_OVERLAP
  • RAG_RETRIEVAL_TOP_K
  • RAG_RETRIEVAL_MIN_SCORE
  • RAG_RETRIEVAL_MAX_PER_DOC

Current docker defaults are RAG_ENABLED=true and RAG_INDEXER_ENABLED=true. Embedding provider + model selection now primarily comes from ClapilotAICore -> Provider & Modelle, not from a fixed OpenAI env key. The legacy RAG_EMBEDDING_MODEL env is only used as a final fallback when no native provider selection exists yet. Document vision fallback is local-first. In ClapilotAICore -> RAG Index, the OCR & Vision block configures the normal RAG vision fallback/model and a dedicated OCR model (receipts/scans). The dedicated app_settings.document_ocr_model is optional: empty inherits rag_document_vision_model; a selected OCR model takes precedence and enables vision fallback for receipt/scan extraction even when the general RAG vision toggle is off. The effective model remains part of the OCR cache configuration hash.

MM-Bridge

files_visualize is the shared multimodal bridge for Chat, Dokumente workflows, and repository-backed Clapilot Code sessions (clapilot-cli files visualize --file-path <path>). It accepts a workspace file path or dokumente_id; runtime-state paths, dotfiles, and dot-directories are rejected after canonical path resolution. Provider/model resolution happens only on the server through the existing OCR/Vision model setting; no provider credential is returned in tool output or placed in the calling model's context.

The bridge is deliberately text-first. Source code, TXT/Markdown, CSV/TSV, JSON/XML/YAML, HTML, SRT/VTT, LaTeX, SVG/DrawIO XML, XLS/XLSX/ODS, DOCX, and Markdown/code cells from IPYNB are extracted locally and do not call a vision model. Images and PDFs use local text/OCR plus the vision sidecar where visual understanding is required. Presentations and legacy Office files are converted with headless LibreOffice before bounded vision analysis. Videos are sampled with ffmpeg.

Default budgets are 40,000 returned characters, at most 2 PDF pages, and at most 6 video frames (one frame per ten seconds). Operators can tune them with MM_BRIDGE_MAX_CHARS, DOC_VISION_MAX_PAGES, and MM_BRIDGE_MAX_FRAMES; all values are hard-capped (120,000 characters, 6 pages, 12 frames). Tool responses include the effective limits. Missing libreoffice, ffmpeg, mammoth, or xlsx support returns a named MM_BRIDGE_DEPENDENCY_MISSING error. A missing sidecar returns MM_BRIDGE_VISION_NOT_CONFIGURED instead of an empty result.

Operations and validation

Check runtime

docker compose exec -T clapilot env | rg '^RAG_|^OPENAI_API_KEY|^CLAPILOT_AGENT_'
docker compose logs -f clapilot | rg -n 'rag-indexer|chat-rag'

Check indexing state

docker compose exec -T postgres psql -U "${POSTGRES_USER:-clapilot}" -d "${POSTGRES_DB:-clapilot}" -c \
"SELECT status, count(*) FROM document_index_jobs GROUP BY status ORDER BY status;
 SELECT count(*) AS chunks FROM document_chunks;"

Admin runtime controls

  • /settings/clapilotaicore/rag shows RAG Index Health
  • Manual reindex options are available from the ClapilotAICore admin UI
  • The same panel exposes the OCR & Vision settings. Both model selects use vision-capable entries from Provider & Modelle; the receipt/scan OCR select includes an inherit option for the RAG vision model

Failure modes

  • Missing embedding provider/model:
    • Indexer logs a configuration error and does not process new embeddings until a provider is selected in ClapilotAICore -> Provider & Modelle
  • Missing document file:
    • Job marked failed with ENOENT; fix file_path or restore file, then reindex
  • No context on a specific question:
    • Usually threshold/filter behavior, not a RAG disable state
  • Repeated document OCR / repeated image uploads:
    • Check for stale _inbox/.process-now.failed.json artifacts and failed queue items; document OCR results are cached by file hash + extraction config, and trigger files are claimed atomically to avoid blind retries across replicas