Model Training and Evaluation

Settings -> ClapilotAICore -> Training manages the full LoRA workflow against the existing Spark training service: create a dataset, start and monitor a run, cancel it, load the completed adapter, compare it with a base or gateway model, and verify whether vLLM is really applying it.

Only one setting is required: the training API base URL. It is stored in app_settings.model_training_api_base_url. An empty value resolves to http://192.168.178.39:8010. The browser never calls the LAN service directly; every operation passes through admin-only routes under /api/clapilotaicore/model-training/**.

Dataset bridge

The primary dataset path is Build from chat export. Instance, source, and date filters match the existing Hub chat export. Clapilot calls the shared transcript-export builder in src/lib/chat-transcript-export.ts, produces messages JSONL with the available tool-call/tool-result trace and tool schemas, and uploads that JSONL directly to Spark's /v1/files endpoint with purpose=fine-tune. There is no download/re-upload step.

The returned dry-run summary includes conversation and message counts, tool calls, conversations with tools, turns dropped because their tool trace could not be reconstructed, incomplete traces, derived schemas, and assistant responses that look like tool-derived data without a recorded call. Zero tool calls is a prominent warning: training on tool-free examples teaches a model to fabricate tool output instead of invoking tools. Operators may continue after reviewing the warning.

JSONL can also be uploaded directly. Either path selects the resulting file ID in the new-run form.

Run lifecycle

A run selects serveable or full target layers plus rank, alpha, learning rate, sequence length, batch size, gradient accumulation, maximum steps, and epochs. The practical defaults are serveable, rank 16, sequence length 8192, and 200 steps.

Spark accepts only one training run at a time. A create response with HTTP 409 becomes the stable training_already_running error and the UI says that another training is active. It is not presented as a generic upstream failure.

The stream route polls training stats and runs every three seconds and emits a combined SSE snapshot. The UI shows progress, step count, ETA, elapsed time, latest loss, base model, hyperparameters, and a compact loss graph. Active runs can be cancelled.

Operational facts that must be understood:

• A run can remain in starting for roughly seven minutes while the model loads without emitting progress. That silent interval is expected and is not by itself a failure.
• A sequence length below 4096 is risky. In the verified Clapilot export, length 2048 truncated 84 percent of the tokens.
• Starting training automatically stops the comparison server. It must be started again after the run.
• The upstream supports only one run at a time; retrying does not create a queue.

Deploy and compare

The comparison status reports up and running separately because they describe different conditions: reachability and process state. Starting and stopping the comparison server are explicit actions. A completed run can be hot-loaded as ft:<run_id> and used in either side of the comparison beside the base and gateway models. Each response includes elapsed seconds and tokens per second.

Hot-loaded adapters do not survive a comparison-server restart. Every required adapter must be loaded again after restart.

Effectiveness verification

A successful load response is not proof that vLLM applies the adapter. The verified upstream can log Loaded new LoRA adapter while silently producing byte-identical base-model output with no error or warning.

The Verify adapter action is therefore the acceptance check. Clapilot resolves the upstream base model, generates from the base and adapter with the same prompt, temperature 0, and a fixed seed, then compares the returned content byte for byte. identical: true is a warning: the adapter is loaded but inert. A difference is evidence that the adapter is participating in generation. A temperature-zero diff is the only trustworthy check provided by this upstream contract because server state and load logs alone do not prove application.

Agent access

Nine native tools expose the workflow: status, list runs, get run, start, cancel, build dataset, deploy adapter, comparison chat, and adapter verification. Start, cancel, dataset build, and deployment enforce administrator status inside their handlers. Results are compact; lists are capped and loss history is limited to the latest 20 points.