# Local AI Models — fine-tune, convert to GGUF, and swap safely

This guide covers the platform's **local LLM**: what it actually does (and does
not do), how to fine-tune it on the platform's own resolved outcomes, how to
convert/quantize a trained model to the `.gguf` file the runtime loads, and the
**safe swap procedure** — register → verify → activate → test → rollback — using
the built-in model manager. Everything here describes what the code really does;
every command matches the real CLI (`--help`-verified).

Companion guides:

* **[Data Guide](DATA_GUIDE.md)** — the databases the fine-tune corpus is mined from.
* **[Reading the Stats](READING_THE_STATS.md)** — how the SR | PSR | DSR(N) gate that
  governs the LLM's live weight is interpreted.

---

## 1. How the local model is used

The platform's default model is **Qwen3-4B-Instruct-2507** as a quantized
**Q4_K_M GGUF**, run entirely **on your machine** by `llama-cpp-python`
(`core/local_ai/runtime.py`). Nothing is sent to any cloud API. The previous
default, **Qwen2.5-7B-Instruct** (`models/Qwen2.5-7B-Instruct-Q4_K_M.gguf`),
stays registered as the legacy/rollback model — every example below that names
the 7B applies unchanged to whichever model is active.

It plays exactly two roles:

**a) One voter in the trading ensemble** (`core/local_ai/agent.py`)

* `llm_signal(symbol, context)` emits a strict-JSON structured opinion —
  `{"action": "BUY"|"SELL"|"HOLD", "confidence": 0..1, "rationale": "<=160 chars"}`
  — that joins the weighted vote in the algo, TPT, and futures bots as a
  **veto/confirm** seat. The combined decision still passes the risk kernel.
* **It never places orders.** The agent has no broker/exec surface of any kind.
* **It earns its weight.** While unproven (too few resolved votes), it rides at a
  small fixed floor (`LLM_FLOOR_WEIGHT = 0.05` — a single LLM ballot cannot
  dominate the ensemble). Once enough of its logged votes resolve against
  realized returns, the weight becomes fully data-driven
  (`llm_ensemble_weight()`: IC > 0, Brier reliability, and a positive deflated
  SR | PSR | DSR(N) triplet are all required). A measurably bad model earns
  **less** than the floor — down to 0. Swapping in a fine-tune does **not**
  grant it weight; it re-earns it live.
* **It abstains by design.** Missing library, missing model, GPU busy with a TNN
  retrain, timeout, or unparseable output all degrade to "no vote" (weight 0)
  rather than an error. Votes are cached per symbol (~900 s TTL) so at most one
  slow inference runs per symbol per window.

**b) The AI Analyst tab** (`core/local_ai/analyze.py`)

Modes `full / bull_bear / risk / options / explain / chat`, grounded in the real
platform context, with a self-critique (reflection) pass and per-symbol memory.
The analyst path is display-only and never touches the vote or trade path.

**Honest limits** (measured on this stack, not marketing):

* It is an **explanation/reasoning overlay** over the platform's validated
  numerical signals — **not a signal generator**.
* It is a poor **large-schema JSON filler**: asked for ~10 required fields it
  reliably fills only 2–3, so the platform never asks it for more than a small
  JSON object and uses deterministic fallbacks elsewhere.
* On GPU a vote takes ~2 s warm; pinned to CPU the same call can take 15–130 s
  and multi-call analyst modes will time out (see Troubleshooting §5).
* Its self-reported confidence is only recalibrated (isotonic) after enough of
  its votes have resolved; before that the raw confidence is passed through
  and the weight floor keeps its influence small.

### Install constraint on this machine (read before touching pip)

`llama-cpp-python` **must** be the prebuilt **abetlen cu124 CUDA wheel**:

```bash
.venv/Scripts/python.exe -m pip install llama-cpp-python \
    --only-binary llama-cpp-python --force-reinstall --no-cache-dir \
    --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124
```

* **Never** run a bare `pip install -U llama-cpp-python`: PyPI serves a source
  build that needs MSVC (absent here), and pip **uninstalls the working wheel
  first**, leaving the LLM dead.
* The abetlen **CPU** wheels use AVX-512, which this CPU (AMD Zen 3) lacks —
  they crash at model load with `0xc000001d` (illegal instruction).
* The CUDA wheel needs the CUDA runtime DLLs at import time; the runtime
  handles this automatically by registering PyTorch's bundled CUDA DLLs before
  importing `llama_cpp` (`runtime._register_cuda_dll_dirs()`), so no system
  CUDA toolkit is required.

---

## 2. Fine-tuning on the platform's own outcomes

The training data is **never fabricated**: it is mined from the platform's own
resolved records — resolved LLM votes in the feedback-loop ledger, resolved
brain observations, and adaptive-backtest trades — plus a small authored
curriculum. The model input is exactly what the live agent sees (same
`SYSTEM_PROMPT` + `build_user_prompt`); realized outcomes set the **target
label only** and are never rendered into the prompt (leak-safety is asserted
and recorded in the manifest).

**Inference is local; training is a cloud step.** Your 8 GB GPU can *infer* the
7B Q4_K_M model but cannot *train* a 7B (LoRA training needs ~14–16 GB). The
pipeline builds the corpus locally and trains on a free Colab T4.

### Step 2.1 — prepare the corpus locally (no GPU needed)

```bash
.venv/Scripts/python.exe scripts/finetune/prepare_finetune.py
```

This one command builds `data/finetune/{train,eval}.jsonl` + `manifest.json`,
**hard-gates** on the manifest, runs the cloud trainer's import preflight, and
generates a ready-to-run Colab notebook
(`scripts/finetune/qwen7b_finetune_colab.ipynb`). Exit codes: `0` ready, `2`
not ready (too few resolved outcomes — wait, nothing is padded), `3` leak
detected (do **not** train), `4` build failed. `--check` runs the gate only
(no notebook write); `--out DIR` changes the corpus directory.

Before training, the manifest must say `"status": "ready"` and
`"leak_safe": true`, with an achieved reasoning fraction around 0.85.

### Step 2.2 — train on Colab (free T4)

Upload `train.jsonl`, `eval.jsonl`, and `scripts/finetune/train_qwen7b_unsloth.py`
(or open the generated notebook) and run:

```bash
python train_qwen7b_unsloth.py \
    --train /content/train.jsonl --eval /content/eval.jsonl \
    --out /content/qwen7b-quant --epochs 2
```

It is an Unsloth 4-bit **LoRA** SFT of Qwen2.5-7B (T4-safe defaults are
auto-clamped on small GPUs) that exports three artifacts under `--out`: the
LoRA adapter, a merged 16-bit HF model, and — the one you need — the
**`q4_k_m` GGUF** in `gguf/`. Download that file. Full click-path and
honesty notes: `docs/FINETUNE_RUNBOOK.md`.

### Advanced: the local small-model QLoRA path

There is also an older, fully-local path (`scripts/diagnostics/llm_finetune.py
build` / `train`) that QLoRA-trains a **small** base model (e.g.
Qwen2.5-1.5B-Instruct) on the numeric SFT/DPO dataset under
`data/llm_finetune/` — this *does* fit on an 8 GB GPU, but the result is a
different, weaker model than the shipped 7B and its adapter must be merged and
converted manually (see §3). Treat it as advanced/experimental; the supported
path is the 7B Colab pipeline above.

---

## 3. Converting and quantizing to `.gguf`

**Default path: you don't need this.** The Colab trainer already exports the
`q4_k_m` GGUF (`--gguf-quant q4_k_m`, the default) — skip straight to §4.

The manual path exists for the advanced cases (a locally-trained adapter from
§2's QLoRA path, or re-quantizing a merged HF model). It uses the
[llama.cpp](https://github.com/ggml-org/llama.cpp) tooling, which is **not
bundled with this platform** — clone/download it separately (flags below are
the standard llama.cpp usage; confirm against your checkout's `--help`, they
occasionally change between releases):

```bash
# 0) if starting from a LoRA adapter: merge it into its base model first
python -c "from peft import AutoPeftModelForCausalLM; \
  m = AutoPeftModelForCausalLM.from_pretrained('models/adapters/<name>'); \
  m.merge_and_unload().save_pretrained('models/merged/<name>')"

# 1) convert the merged HF model to a full-precision GGUF
git clone https://github.com/ggml-org/llama.cpp
pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
python llama.cpp/convert_hf_to_gguf.py models/merged/<name> \
    --outfile <name>-f16.gguf --outtype f16

# 2) quantize with llama-quantize (a prebuilt binary from the llama.cpp
#    releases page, or built from source)
llama-quantize <name>-f16.gguf <name>-Q4_K_M.gguf Q4_K_M
```

Note the split: `convert_hf_to_gguf.py` produces full/high-precision GGUFs
(`f16`/`bf16`/`q8_0`); the K-quants like `Q4_K_M` come from the separate
`llama-quantize` step.

### Which quant level for an 8 GB GPU

The GGUF shares the card with the PyTorch brain, and the KV cache costs VRAM on
top of the weights (the platform defaults to a 4096-token context for exactly
this reason — an 8192 KV cache is ~1.3 GB and overflowed 8 GB alongside the
7B weights).

| Quant | 7B file size | On 8 GB, next to the brain | Verdict |
|---|---|---|---|
| `Q4_K_M` | ~4.7 GB | fits with headroom | **Recommended — the platform standard** |
| `Q5_K_M` | ~5.4 GB | tight; expect OOM step-downs | only if you must |
| `Q8_0` / `f16` | 8–15 GB | does not fit | no |
| `Q3_*` / `Q2_*` | smaller | fits easily | reasoning quality degrades — not worth it |

---

## 4. Safe swap — register → verify → activate → test → rollback

The runtime resolves which GGUF to load in this priority order
(`runtime._find_model_path()`):

1. the **active entry in the model registry** (`weights/llm_registry.json`),
   if the file exists;
2. else the `LOCAL_AI_MODEL` env var (typically set via `.env`), if it points
   at an existing file;
3. else the first `models/*.gguf`, sorted.

The registry is the recommended mechanism — it is versioned, hashed, and has
one-step rollback. It is managed with the built-in CLI
(`python -m core.local_ai.model_manager`; subcommands: `register`, `activate`,
`rollback`, `verify`, `remove`, `list`, `current`).

> **Golden rule: never delete or overwrite the stock GGUF**
> (`models/Qwen2.5-7B-Instruct-Q4_K_M.gguf`). It is your rollback.

### 4.1 Register the new model (hashes it, does not activate)

```bash
# put the new file in the gitignored models/ dir first, then:
.venv/Scripts/python.exe -m core.local_ai.model_manager register \
    "models/Qwen2.5-7B-Instruct-quant-ft-20260702-Q4_K_M.gguf" \
    --name qwen7b-ft-20260702 --version v20260702 --kind finetuned
```

`register` records the path, a **sha256 hash**, size, version, and timestamps.
Optional flags: `--activate` (register + activate in one step), `--no-hash`
(skip the slow hash — not recommended), `--kind {finetuned,base}`.

### 4.2 Verify integrity

```bash
.venv/Scripts/python.exe -m core.local_ai.model_manager verify qwen7b-ft-20260702
```

Re-hashes the file and checks it still exists. Prints `ALL OK` (exit 0) or
`INTEGRITY FAILURE` (exit 2 — the file is corrupt/moved; re-download and
re-register before going further). Run `verify` with no name to check every
registered model.

### 4.3 Activate

```bash
.venv/Scripts/python.exe -m core.local_ai.model_manager activate qwen7b-ft-20260702
.venv/Scripts/python.exe -m core.local_ai.model_manager current   # confirm
```

`activate` flips the registry's `active` pointer and records the previous one
for rollback. **Restart the app** so the runtime singleton reloads — the log
line `LocalLLM loaded: <filename>` must name your new file.

### 4.4 Test-vote before trusting it

```bash
# smoke: 10 canned symbols through the REAL production vote path (read-only,
# records nothing to the ledger) — shows each vote/abstain:
.venv/Scripts/python.exe scripts/diagnostics/test_finetuned_live.py

# gate: CoT-survival validation on the held-out eval split at the production
# vote budget (valid-JSON / carried-reasoning / abstain-rate / action mix):
.venv/Scripts/python.exe scripts/finetune/validate_finetune.py --max-rows 20
```

`validate_finetune.py` exits `0` on PASS, `1` on FAIL (a metric collapsed —
roll back), `2` if it could not run. Even a PASS grants no ensemble weight:
the model still earns its live weight through the gate (§1).

### 4.5 Rollback if bad

```bash
.venv/Scripts/python.exe -m core.local_ai.model_manager rollback
```

One step: the previously-active model is active again (running `rollback` a
second time swaps back — it is a toggle between the last two). Restart the app.
If the registry is somehow empty/broken, the runtime automatically falls back
to `LOCAL_AI_MODEL` / the stock `models/*.gguf` — it never wedges on a stale
pointer.

### What the registry file records

`weights/llm_registry.json` (gitignored; override the path with the
`LOCAL_AI_REGISTRY` env var):

```json
{
  "active": "C:\\...\\models\\Qwen3-4B-Instruct-2507-quant-ft-20260710-Q4_K_M.gguf",
  "prev_active": "C:\\...\\models\\Qwen3-4B-Instruct-2507-quant-ft-20260710-Q4_K_M.gguf",
  "models": [
    {
      "path": "C:\\...\\models\\Qwen2.5-7B-Instruct-quant-ft-20260626v-Q4_K_M.gguf",
      "name": "qwen25-7b-ft-20260626",
      "kind": "finetuned",
      "version": "v1",
      "size_bytes": 4683073536,
      "sha256": "",
      "added_ts": 1783444000.837,
      "updated_ts": 1783444000.837,
      "notes": "previous champion (rollback target)"
    },
    {
      "path": "C:\\...\\models\\Qwen3-4B-Instruct-2507-quant-ft-20260710-Q4_K_M.gguf",
      "name": "qwen3-4b-ft-20260710",
      "kind": "finetuned",
      "version": "v4",
      "size_bytes": 2497280288,
      "sha256": "facfd920…",
      "added_ts": 1783724504.224,
      "updated_ts": 1783724504.224,
      "notes": "PROMOTED 2026-07-11: Gate-2 vote eval PASS + CoT-survival PASS",
      "eval_cot_ok": true,
      "registered_at": "2026-07-11T15:55:48Z"
    }
  ]
}
```

(Base-model entries — e.g. the stock `qwen3-4b-instruct-2507` — sit in the same
`models` array and are elided here for brevity. The example matches the live
registry: the **active** model is the Qwen3-4B fine-tune `qwen3-4b-ft-20260710`;
the earlier 7B fine-tune stays registered as a rollback target.)

Entries registered programmatically via `runtime.set_active_model()` can carry
extra provenance (`base`, `trained_at`, `dataset_manifest`, `eval_cot_ok`); a
model whose `eval_cot_ok` is not `true` is reported as **unverified** by
`runtime.active_model_info()` (what the AI Analyst banner shows). Writes are
atomic; a corrupt registry is ignored, never crashes the app.

**Legacy `.env` path:** `scripts/finetune/install_finetuned_gguf.py` copies a
downloaded GGUF into `models/` under a dated canonical name and writes
`LOCAL_AI_MODEL` into `.env` (`--rollback` removes the key; `--rollback
--to-stock` repoints it at the stock model). It still works — but remember the
registry **outranks** `.env`: while a registry entry is active,
`LOCAL_AI_MODEL` is ignored. Pick one mechanism; prefer the registry.

---

## 5. Troubleshooting

**Crash `0xc000001d` (STATUS_ILLEGAL_INSTRUCTION) at model load** — you have an
abetlen **CPU** wheel; those use AVX-512, which this CPU does not have. Fix:
reinstall the **cu124** wheel with the exact command in §1. This is also the
first thing to check after any `pip install`/`pip upgrade` touched the venv.

**`ImportError: ... llama.dll (or one of its dependencies)`** — the CUDA wheel
cannot find the CUDA runtime DLLs. The platform registers PyTorch's bundled
DLLs automatically, but only when `llama_cpp` is imported *via*
`core.local_ai.runtime` and torch is installed in the same venv. Standalone
scripts should `from core.local_ai.runtime import get_local_llm` rather than
importing `llama_cpp` directly.

**CUDA out-of-memory / model refuses to load** — the 8 GB budget is: 7B Q4_K_M
weights ~4.7 GB + KV cache (~context-size-dependent; the default
`QP_LLM_N_CTX=4096` is chosen so it fits) + whatever the PyTorch brain holds.
Fixes, in order: use `Q4_K_M` (not Q5/Q8), lower `QP_LLM_N_CTX`, and let the
runtime's automatic GPU-layer sizing/step-down work (do **not** hard-pin
`QP_LLM_GPU_LAYERS`). If a new model still will not load, `rollback` (§4.5) —
the platform is fully functional on the stock model.

**Votes are slow (tens of seconds) or analyst modes time out** — the model is
running on CPU. Check `.env` for a `QP_LLM_GPU_LAYERS=0` pin and remove it
(env wins over the runtime's auto-sizing); `0` forces CPU, `-1` means all
layers on GPU. A healthy GPU vote is ~2 s warm.

**Everything abstains instantly / "analyst offline"** — the runtime is
unavailable: either `llama-cpp-python` is not importable (§1) or no GGUF
resolves. Run `python -m core.local_ai.model_manager current` (shows the
active entry, or `(no active model; using base)`) and `list` (a `(MISSING)`
tag means the registered file is gone from disk), and check `models/` still
contains a `.gguf`.

**New model loads but the votes look wrong** (JSON validity collapsed,
reasoning vanished, everything is BUY) — that is the documented fine-tune
failure mode `validate_finetune.py` exists to catch. Roll back (§4.5), rebuild
the corpus with more resolved outcomes / a higher reasoning fraction, retrain.

**`verify` prints `INTEGRITY FAILURE`** — the file on disk no longer matches
the sha256 recorded at registration (truncated download, moved/edited file).
Do not activate it; re-download and `register` again.