# Data Guide — what every database actually contains

This guide explains the data your platform records: which SQLite databases exist,
what a row in each important table *means*, how a prediction goes from `open` to
`evaluated`, and why the sample counts (N) you see are smaller than the raw row
counts. Everything here describes what the code really does — table and column
names match the live schemas.

Companion guides:

* **[Reading the Stats](READING_THE_STATS.md)** — how to interpret SR | PSR | DSR(N),
  verdicts, and calibration.
* **[Data Center How-To](DATA_CENTER_HOWTO.md)** — browsing these tables and running
  your own read-only SQL.

---

## 1. Where the data lives

All platform data is stored in plain SQLite files in your per-user QuantSuite data
folder (NOT the read-only install directory):

- **Windows:** `%LOCALAPPDATA%\QuantSuite\data`
- **macOS:** `~/Library/Application Support/QuantSuite/data`
- **Linux:** `~/.local/share/QuantSuite/data`

(In a dev checkout run from the repo, these resolve to the repo's own `data/`.) The
exact path is also shown in **Settings → Data Location**, with an **OPEN DATA FOLDER**
button. Nothing is uploaded anywhere; you can open any of these files with any SQLite
tool (read-only recommended — see the How-To).

The Data Center registers these databases (key → file → purpose):

| Key | File | Purpose | Update cadence |
|---|---|---|---|
| `algo_trades` | `algo_trades.db` | Algo-bot trade ledger + agent blend weights | live |
| `feedback_loop_ledger` | `data/feedback_loop_ledger.db` | The closed feedback loop: signals, options recs, flow ratings, edge metrics | daily-pipeline |
| `brain_knowledge` | `data/brain_knowledge.db` | Brain knowledge base: observations, paper trades, risk decisions | daily-pipeline |
| `trial_ledger` | `data/trial_ledger.db` | Trial Ledger — the N used for deflated statistics | live |
| `quant_neural` | `data/quant_neural.db` | Neural-net training telemetry (epochs, runs, checkpoints) | weekly-manual |
| `feature_store` | `data/feature_store.db` | TTL feature cache | daily-pipeline |
| `historical_store` | `data/historical_store.db` | OHLCV bar storage + data-quality log | on-demand |
| `propfirm_brain_v5` | `data/propfirm_brain_v5.db` | Futures (ES/NQ) prop-firm bot trades + daily stats | event-driven |
| `execution_v2` | `data/execution_v2.sqlite` | Execution order book (parent + child orders) | event-driven |
| `decision_traces` | `db/decision_traces.db` | Decision introspection (traces, counterfactuals, lessons) | event-driven |
| `market_memory` | `db/market_memory.db` | Episodic market memory | event-driven |
| `ml_arch_registry` | `data/ml_arch_registry.sqlite` | ML architecture experiment registry | on-demand |
| `slide_doctrine` | `data/slide_doctrine/doctrine.db` | Slide-Doctrine sub-app | on-demand |
| `adaptive_feedback` | `data/adaptive_feedback.db` | Legacy adaptive controller state | deprecated |
| `convergence_loop` | `data/convergence_loop.db` | Legacy convergence loop state | deprecated |
| `data_v2` | `data/data_v2.sqlite` | Legacy provider cache (superseded) | deprecated |
| `dsr_units_demo` | `data/_dsr_units_demo.db` | Unit-test fixture — ignore | deprecated |

**Cadence matters.** A database that hasn't been written in days is not
necessarily broken: `event-driven` stores only change on real activity (orders,
bot runs), `on-demand` stores only change when something requests them, and
`deprecated` stores are intentionally frozen. Only `daily-pipeline` and `live`
databases should ever look "stale". The Data Center shows the cadence class next
to each database for exactly this reason.

The four databases you will actually read most are covered in depth below:
`feedback_loop_ledger`, `brain_knowledge`, `trial_ledger`, and `algo_trades`.

---

## 2. `feedback_loop_ledger` — the measurement spine

This is the most important database on the platform. It is the append-mostly
record of every prediction the system makes (brain signals, options
recommendations, flow ratings, LLM votes and vetoes) and — crucially — the
**realized outcome** of each one, filled in later by the daily evaluator. The
loop is:

```
prediction recorded (status='open')
        │  … time passes; the horizon elapses …
        ▼
daily evaluator fetches the real price at the horizon date
        ▼
outcome written back (status='evaluated', was_correct set)
        ▼
edge statistics recomputed per scope → edge_metrics
```

### 2.1 `brain_signals` — every directional call

One row = one **standing prediction**: "symbol X will move in direction Y over
the next H days". Written by the brain, the algo bot, the neural model, and the
local LLM voter (they are distinguished by `source`).

Key columns:

| Column | Meaning |
|---|---|
| `created_at` | When the prediction was **first** made (UTC ISO). This is frozen — it anchors the forward-return window. |
| `last_seen_at` | The last scan tick that re-emitted the same standing prediction (see de-duplication, §2.7). |
| `symbol`, `action` | Ticker and the call: `BUY` / `SELL` / `HOLD`. |
| `source` | Who made it: `brain`, `algo_bot`, `neural`, `llm`, `strategies`, `futures`, `options`, `tpt`. |
| `p_up` / `p_down` / `p_flat`, `confidence` | The model's stated probabilities / conviction at prediction time. Frozen at first emit. |
| `ref_price` | The spot price when the signal was issued — the anchor the return is measured from. |
| `horizon_days`, `horizon` | The claim window (e.g. 5 days) and its bucket (`SHORT`/`MED`/`LONG`). |
| `thesis`, `features` | JSON: the contributing factors and the feature vector at signal time (used later by learners and audits). |
| `status` | Lifecycle: see §2.6. |
| `outcome_return` | The realized % return from `ref_price` to the close at the horizon date. Only set once the horizon has actually elapsed **and** that day's price exists — the evaluator never grades against "today's" price. |
| `was_correct` | 1/0: was the direction right? BUY is correct if the return was > 0, SELL if < 0, HOLD if the absolute move stayed under 0.5%. |
| `evaluated_at` | When the outcome was graded. |

**What a row means:** a falsifiable claim, graded exactly as originally made.
The prediction fields are deliberately frozen at first emit — a signal that gets
re-emitted at a drifted price later in the day does not get to update its entry
price or confidence, because that would grade a post-hoc revision as if it had
been the original forecast.

### 2.2 `options_recommendations` — every options structure suggested

One row = one standing options recommendation (e.g. "SPY bull_call_debit,
expiry 2026-06-29"). Written by the options recommender; marked to market daily
by the evaluator.

| Column | Meaning |
|---|---|
| `strategy` | Structure type: `bull_call_debit`, `bear_call_credit`, `iron_condor`, `long_strangle`, … |
| `legs` | JSON list of the actual legs (strike / type / side / qty). |
| `spot_at_rec`, `net_premium`, `max_profit`, `max_loss`, `breakevens` | The trade economics at recommendation time. Frozen at first emit. |
| `predicted_pop` | The predicted probability of profit (0–1) — this is the graded claim. |
| `expiry`, `days_to_expiry` | The structure's expiry; the loop closes at/after expiry. |
| `status`, `final` | Lifecycle (§2.6). `final=1` means the position reached expiry and its outcome is settled. |
| `last_eval_at`, `last_spot`, `est_value` | The most recent daily mark. |
| `outcome_pnl`, `outcome_pnl_pct` | Realized P&L in $ and as a **% of capital at risk** (see below). |
| `holds_true` | 1/0: did the recommendation perform (P&L > 0)? This is what the hit rate counts. |

**`outcome_pnl_pct` is risk-normalized, not return-on-debit.** It is
`pnl / abs(max_loss)` (falling back to `net_premium` when max_loss is
unavailable), clamped to [−100%, +500%]. Dividing by a tiny net debit produced
absurd artifacts (−9,539% on one spread), so the denominator is the capital
genuinely at risk. −100% means the full defined risk was lost.

### 2.3 `options_eval_log` — the daily mark-to-market trail

One row per recommendation per day: `spot`, `est_value` (a Black–Scholes
re-price of the structure with a historical-vol proxy), `pnl`, `pnl_pct`,
`days_to_expiry`, `holds_true`, `final`. A UNIQUE index on (rec_id, day) makes
same-day re-evaluation idempotent — running the daily eval twice cannot inflate
the time series. Use this table to watch how a position's estimated value
evolved between recommendation and expiry.

Honest caveat: `est_value` is a **model** price (self-contained Black–Scholes on
spot history), not a live options-chain quote. It's good enough for grading
"did this structure work", not for quoting a closing price.

### 2.4 `flow_ratings` — graded flow predictions

A flow "rating" (bullish/bearish options or dark-pool flow) is only meaningful
if it makes a falsifiable claim. Each row therefore stores a **defined event**
at rating time: "an absolute move ≥ `k_sigma` × the symbol's own realized vol
(`sigma_ref`) within `horizon_days`, in the rated `direction`". Resolution fills
in `realized_move`, `event_occurred`, and `was_correct`. `p_event` is the
rater's self-reported probability — it is stored verbatim and only trusted once
resolution proves it is calibrated.

Honest caveat: on a fresh or lightly-used install this table can hold thousands
of `open` rows and zero `evaluated` ones — flow ratings only resolve when the
daily evaluation pipeline runs and their horizons have elapsed. An all-open
backlog means "not graded yet", not "all correct".

### 2.5 `edge_metrics`, `attribution`, `regime_perf_tables`, `llm_vetoes`

* **`edge_metrics`** — one row per (scope, snapshot): the rolling statistical
  evidence per scope (`signals:algo_bot:MED`, `options:global`,
  `options:iron_condor`, …). Columns: `n`, `hit_rate`, `avg_return`,
  `information_coef`, `t_stat`, `p_value`, `brier`, `calibration_error`,
  `verdict` (`edge` / `luck` / `insufficient`), plus the deflated triplet
  `sharpe`, `psr`, `dsr` and `min_trl`. Rows are **snapshots**: always read the
  latest row per scope (the Data Center does this for you) and check its `ts` —
  an old snapshot can predate methodology fixes.
* **`attribution`** — for each resolved outcome, the signed contribution of each
  thesis factor. Lets you ask "which factors were present when we were right vs
  wrong".
* **`regime_perf_tables`** — persisted snapshots of the SR | PSR | DSR(N) table
  per (source : horizon : market regime), built with leak-safe, forward-only
  regime labels. Stored as JSON in `table_json`; cells with too few resolved
  trades are honestly marked unavailable rather than given a fabricated triplet.
* **`llm_vetoes`** — the local LLM's conservative trade-review verdicts
  (`veto` / `flag` / `approve`) over candidate trades proposed by the bots, plus
  the graded quality of each call once the judged trade's outcome is known
  (`trade_was_winner`, `conservative_correct`). A veto that avoided a loss was a
  good veto; a veto that blocked a winner was a bad one.

### 2.6 What `status` means

| Value | brain_signals / flow_ratings | options_recommendations |
|---|---|---|
| `open` | Prediction made; horizon not yet elapsed (or horizon price not yet available). Not counted in any accuracy statistic. | Recommendation standing; still being marked daily. |
| `evaluated` | Outcome graded; `was_correct` / `outcome_return` are set. This is the only status counted in hit rates and edge stats. | Final outcome written (`holds_true` set). Positions also carry `final=1` once expiry has settled them. |
| `superseded` | This row was identified as a per-tick duplicate of another row (the "survivor") by the guarded de-duplication sweep. It is kept for audit but **excluded from all statistics**. | Same. |

`superseded` rows are never deleted by default — the raw tick-by-tick timeline
is preserved for audit, but every statistic reads past them.

### 2.7 How N works — de-duplication in three layers

The single most important honesty property of this ledger: **N counts distinct
trades, not scan ticks.** Background scanners re-emit the same standing position
every tick; naïvely counting rows would multiply one trade into nine
"observations", inflating N, t-statistics, and hit rates. (This was a real bug,
found and fixed: one source showed an apparent edge at N=36 that was really N=8.)
Three layers prevent it:

1. **Write throttle.** When the same standing key — `(symbol, source, action,
   horizon)` for signals; `(symbol, strategy, expiry)` for options recs — is
   re-emitted within a 6-hour window onto a still-`open` row, the ledger
   *updates that row in place* instead of inserting a new one. It refreshes
   `last_seen_at` and bumps a `_emit_count` counter inside the `thesis` JSON
   (so the collapsing is auditable), but the prediction fields stay frozen at
   first emit. A genuine action flip, a resolved prior row, or a re-emit outside
   the window inserts a fresh row.
2. **Read-time collapse.** All statistics (accuracy, hit rate, edge metrics,
   calibration fits) additionally collapse rows sharing
   `(symbol, source, action, horizon, day)` — one observation per standing
   episode per day — before computing anything. This catches legacy duplicates
   written before the throttle existed. The collapse is deliberately
   conservative: two genuinely distinct same-direction episodes within one day
   count as one. Undercounting N is honest; overcounting manufactures
   significance.
3. **Guarded backfill.** A maintenance sweep (`scripts/maintenance/dedup_ledger.py`,
   dry-run by default, backs up the DB before touching anything) marks
   historical duplicate rows `status='superseded'` so even raw SQL that filters
   on `status IN ('open','evaluated')` sees distinct trades.

If you query the raw tables yourself, remember this: `COUNT(*)` over
`brain_signals` counts ticks and tombstones, not trades. The How-To guide gives
dedup-aware example queries.

---

## 3. `brain_knowledge` — what the brain observed and did

Written by the knowledge/learning runs and the background scanners.

| Table | One row is… |
|---|---|
| `observations` | One brain observation of one symbol at one moment: `action` (BUY/HOLD/SELL), `blended_score`, `confidence`, `drivers` and `components` (JSON — which sub-models contributed; LLM votes are tagged here with `llm_conf`), `price` at observation. When the pipeline later resolves it: `resolved=1`, `fwd_return` (realized forward return), `was_correct`. HOLD observations are recorded but have no directional grade. |
| `paper_trades` | One simulated (paper) trade opened/closed by a knowledge run: side, entry/exit, qty, `pnl`, close `reason` and the brain's action/confidence at entry. Paper only — no real money is represented here. |
| `risk_decisions` | One allow/block decision by the risk layer, with its `reason` (e.g. rate limits). Useful for auditing why the system did *not* act. |
| `consumer_messages` | One brain→consumer handoff: what the brain said, what the downstream model said, whether they agreed, and what action was taken. |
| `runs` | One scan/learning run: kind, start/end, minutes, symbol count, market phase. |

Reading tip: `observations.was_correct` uses the same convention as the ledger —
it is NULL until `resolved=1`, so always filter `WHERE resolved = 1` before
computing accuracy.

---

## 4. `trial_ledger` — the N behind every deflated statistic

One table, `trials`. One row = **one strategy evaluation**: a backtest, a
parameter-sweep variant, a retrain, or a replay. Columns: `strategy_family`
(e.g. `algo_bot:SWING`, `tnn_retrain`, `tpt_optimizer`), `source`
(backtest/sweep/retrain/replay), `config_hash`, `n_trades`, `sharpe`
(**annualized** — that is the stored convention, consumers de-annualize for the
DSR math), `psr`, and a generic selection `metric`.

Why it exists: if you try 400 strategy variants and show off the best one, that
best result is mostly selection luck. The Deflated Sharpe Ratio corrects for
this, but only if the number of trials N is *actually recorded*. Every backtest
and retrain the platform runs registers itself here, and every DSR you see looks
up its family's count in this ledger. When a family has no recorded history, the
platform deliberately falls back to the **global** trial count — inheriting the
platform's entire search breadth. Over-counting N only makes the test stricter;
under-counting would flatter the result.

Practical consequence you should expect: **the more experiments you run, the
harder your DSR gets to impress.** That is the design working, not a bug. See
[Reading the Stats](READING_THE_STATS.md).

---

## 5. `algo_trades` — the algo bot's own ledger

| Table | Contents |
|---|---|
| `trades` | One row per bot trade: timestamp, ticker, action, price, shares, stop/target, `signal_score`, `confidence`, `regime`, `strategy`, `timeframe`, and — once closed — `pnl`, `pnl_pct`, `exit_reason`, `hold_time_hours`, `max_favorable`/`max_adverse` excursion, `is_closed`. |
| `agent_weights` | Snapshots of the ensemble blend weights per timeframe (trend / momentum / mean-revert / volume / regime / sentiment, plus BRAIN/LLM seats when present) with the training stats that produced them. |
| `strategy_stats` | Per-(strategy, regime, timeframe) rollups: trades, wins, total/avg PnL, win rate. |
| `trade_log`, `regime_history`, `training_history` | Auxiliary logs: raw trade events, market-regime transitions, training runs. |

Honest note: on a fresh install (or if the bot has never been enabled) **all of
these tables are empty** — the Data Center's Algo dashboard will show zeros and
"no trades". That is correct behavior, not missing data. Numbers appear only
when the bot actually trades.

---

## 6. Quick reference — the questions each database answers

| You want to know… | Look in |
|---|---|
| "Was the brain's BUY call on NVDA last week right?" | `feedback_loop_ledger.brain_signals` (resolved rows) |
| "Do the options recommendations actually hit their predicted POP?" | `feedback_loop_ledger.options_recommendations` + the calibration view in the Data Center |
| "Is this source's performance edge or luck?" | `feedback_loop_ledger.edge_metrics` (latest row per scope) |
| "How many things did we try before this looked good?" | `trial_ledger.trials` (N per strategy_family) |
| "What has the bot actually traded and earned?" | `algo_trades.trades` |
| "Why didn't the system act on X?" | `brain_knowledge.risk_decisions` |
| "How did a specific options position evolve day by day?" | `feedback_loop_ledger.options_eval_log` |
| "Which factors were present when we were wrong?" | `feedback_loop_ledger.attribution` |

All of it is browsable — and queryable with plain SQL — in the read-only
[Data Center](DATA_CENTER_HOWTO.md).