# Reading the Stats — SR | PSR | DSR, verdicts, and calibration Every performance figure on this platform ships as a triplet — **raw SR | PSR | DSR** — with the trial count **N** visible, e.g.: ``` SR 1.31 | PSR 74% | DSR 12% (N=592) ``` This guide explains what each number means, where the platform gets it from, what the `edge / luck / insufficient` verdicts are, how to read calibration and POP bands, and — most importantly — **why honest numbers are smaller than the numbers you are used to seeing.** Companion guides: [Data Guide](DATA_GUIDE.md) (what the underlying tables contain) and [Data Center How-To](DATA_CENTER_HOWTO.md) (querying them yourself). --- ## 1. Why one Sharpe ratio is not enough Run 100 random parameter sets over the same history and a handful will post a great Sharpe ratio purely by chance. Pick the best one and you have "found" a strategy — except you selected it *for* its luck, and out of sample the luck evaporates. A Sharpe of 2.0 from one honest test and a Sharpe of 2.0 cherry-picked from 500 attempts look identical when displayed as one number. The platform therefore never shows a Sharpe alone. Each figure is accompanied by two probabilities that price in the two things a raw Sharpe hides: **how lumpy and short the track record is** (PSR) and **how many things were tried before this one looked good** (DSR with N). The math follows Bailey & López de Prado: *"The Sharpe Ratio Efficient Frontier"* (2012) for PSR/MinTRL and *"The Deflated Sharpe Ratio"* (2014) for DSR, with the Mertens (2002) non-normality correction. The implementations live in `core/quant_stats/probabilistic_sharpe.py` and `core/quant_stats/deflated_sharpe.py`; the triplet assembly (and the N lookup) in `core/trial_ledger.py::sr_triplet`. ## 2. The three numbers ### Raw SR — the naive figure Mean return divided by the standard deviation of returns. Displayed **annualized** (per-period SR × √252 for daily series). Useful for continuity with every other tool you have used, but it ignores: * how many observations produced it (20 trades or 2,000), * skew and fat tails (a strategy that collects pennies and occasionally loses a dollar can show a fine SR right up until it doesn't), * how many strategy variants were tried before this one was picked. On very small samples the annualized SR can be absurdly large in either direction (a ×√252 multiplier does that). Never read it without the other two. ### PSR(0) — "is the true Sharpe even above zero?" The **Probabilistic Sharpe Ratio** is the probability that the *true* (population) Sharpe exceeds a benchmark — here 0 — given the observed SR, the record length n, and the skew/kurtosis of the returns. It is a confidence level, not a performance level: * **PSR 97%** — the record is long and clean enough that a true SR > 0 is about as close to proven as market data gets. * **PSR 55%** — barely better than a coin toss; the record is too short or too lumpy to say anything. A high raw SR with a low PSR means "great number, not enough evidence". Fat left tails and negative skew push PSR *down* for the same raw SR — that is the Mertens correction doing its job. ### DSR(N) — "does it survive the selection tax?" The **Deflated Sharpe Ratio** repeats the PSR test, but raises the benchmark from 0 to the Sharpe you would expect the *best of N random tries* to show — `E[max SR | N trials]`, the extreme-value formula from Bailey & López de Prado (2014). The more variants, sweeps, and retrains were run (bigger N), the higher the hurdle, and the smaller the DSR probability. **Where N comes from — and this matters.** N is not an assumption. Every backtest, parameter-sweep variant, retrain, and replay the platform runs registers itself in the **Trial Ledger** (`data/trial_ledger.db`; see the [Data Guide §4](DATA_GUIDE.md)). `sr_triplet` looks up the count for the strategy family being evaluated; a family with no recorded history deliberately inherits the **global** trial count (on this machine: several hundred), because over-counting N only makes the test stricter while under-counting would flatter the result. When ≥ 2 same-family Sharpes are recorded, the cross-trial variance of those Sharpes (de-annualized to per-period units) replaces the standard V = 1 convention in the benchmark — the deflation adapts to how noisy your own search actually was. Practical readings: * **DSR ≥ 95%** — the result stands even after pricing in the full recorded search. Rare, and meant to be. * **DSR ~ 50%** — indistinguishable from the best-of-N luck benchmark. * **DSR ~ 0–20%** — the raw SR is *smaller* than what N random tries would produce; treat the strategy as unproven regardless of how the equity curve looks. ### MinTRL — the fourth number you'll sometimes see **Minimum Track Record Length**: the number of observations required before PSR could reach 95% at the observed SR/skew/kurtosis. If a scope shows `min_trl = 209,903`, that is the platform telling you honestly that at the current edge size you would need ~210k resolved observations to prove it — i.e. the measured edge is tiny. `min_trl = ∞` means the observed SR is at or below the benchmark, so no record length could ever prove it. ## 3. What return series the triplet runs on For **signal scopes** the series is the **direction-signed PnL** of resolved, de-duplicated signals: BUY → +forward return, SELL → −forward return, HOLD → excluded (no position, no PnL). This was a real fix: computing statistics on the raw underlying return once graded a losing SELL-heavy scope as SR 13.45 / DSR 0.99999 while a trader following the signals lost 2.96% per trade. The number you see now is "what following the signals earned". For **options scopes** the series is `outcome_pnl_pct` — realized P&L as a percent of the capital at risk (|max_loss|), clamped to [−100%, +500%]. See [Data Guide §2.2](DATA_GUIDE.md). Both series are counted **after de-duplication** — one observation per standing prediction per day, never one per scan tick (Data Guide §2.7). ## 4. Verdicts: `insufficient`, `luck`, `edge` Each scope in `edge_metrics` carries a verdict computed by `brain/loop_evaluator.py` from the resolved, de-duplicated sample: | Verdict | Rule (all in code, no discretion) | |---|---| | `insufficient` | Fewer than **20** resolved distinct observations. No triplet is shown either — a handful of trades never gets a Sharpe. | | `edge` | n ≥ 20 **and** p-value < 0.05 **and** t-stat > 0 **and** IC > 0 — the mean signed PnL is positive, unlikely to be chance, *and* the predictions actually correlate with realized returns. | | `luck` | Everything else. | Notes on the ingredients: * **t-stat / p-value** — a one-sample test of "mean signed PnL > 0", with an autocorrelation correction: overlapping multi-day forward returns emitted across a correlated universe are not independent, so the effective sample size is shrunk (`n_eff = n·(1−ρ₁)/(1+ρ₁)` for positive lag-1 autocorrelation ρ₁ — never inflated). Uncorrected, this exact bug once produced a bogus t = 8.59 on what was really an N=8 sample. * **IC (information coefficient)** — the correlation between a *signed* prediction (expected return, or confidence signed by BUY/SELL) and the realized return. All three conditions must hold: a scope can post a large positive t-stat and still be stamped `luck` because its IC is ≤ 0 (the predictions carry no ordering information; the mean was direction-agnostic drift). * **Verdicts are per scope** — per (source : horizon) for signals (e.g. `signals:algo_bot:MED`) and per strategy for options (e.g. `options:iron_condor`, plus `options:global`) — so a strong swing model and a weak day-trade model are never averaged into one flattering number. **Reading tip:** `edge_metrics` rows are snapshots. Always read the latest row per scope and check its `ts` — an old snapshot can predate methodology fixes and show numbers the current code would never produce. ## 5. Calibration: Brier, calibration error, and POP bands A model that says "80%" should be right about 80% of the time. Three measures track this: * **Brier score** — mean squared error between stated probability and the 0/1 outcome. Lower is better; 0.25 is what a coin-toss forecaster scores on a 50/50 event. Meaningful mainly in comparison over time and between sources. * **Calibration error** — outcomes are bucketed into 5 probability bins; the error is the count-weighted average |stated probability − realized frequency|. 0.05 means statements are, on average, within 5 points of reality. * **POP reliability curve (options)** — the Data Center's Options dashboard buckets `predicted_pop` into deciles and plots realized `holds_true` rate per bucket, with each bucket's n. A calibrated recommender hugs the diagonal. Only **resolved, distinct** recommendations count — open interim marks and per-tick duplicates are excluded (letting them in once made 396 of 430 "outcomes" fake). The same philosophy is applied to stated confidences elsewhere: the AI analyst's "calibrated confidence" is fit only from resolved outcomes, and when there are not enough resolved votes the UI says exactly that — *"not enough resolved votes yet — raw conviction shown above is uncalibrated"* — instead of dressing a raw softmax up as a probability. Flow ratings' self-reported `p_event` is stored verbatim and only trusted once resolution proves it calibrated. ## 6. Why the numbers get smaller — and why that is the product working Every honesty mechanism in the pipeline pushes displayed numbers *down* relative to naive reporting: 1. **De-duplication shrinks N.** One standing position per day, not one row per scan tick. A famous internal example: an apparent edge at N=36 (t = 8.59) was really N=8 and vanished. 2. **The autocorrelation correction shrinks N again** (`n_eff`), because overlapping forward windows are not independent bets. 3. **PSR discounts short and lumpy records.** Same SR, fewer/fatter-tailed observations → lower probability. 4. **DSR taxes the search.** Every recorded backtest and retrain raises the best-of-N benchmark. *The more experiments you run, the harder your DSR gets to impress.* That is by design — the alternative is selection bias. 5. **Options P&L is risk-normalized.** Percent of capital at risk, not percent of a tiny net debit, clamped so a degenerate denominator cannot print −9,539%. 6. **Grading is horizon-anchored.** A 5-day signal is graded against the price at day 5, from the price frozen at first emit — never against "today's" price, never against a revised entry. A tool that hides any of these steps will show you bigger numbers. They will also be wrong more often, in the direction that costs money. ## 7. Quick decision table | You see… | Read it as… | |---|---| | `insufficient`, no triplet | Not graded yet or too few resolved outcomes. Wait; do not extrapolate from the hit rate. | | `luck`, DSR < 50% | No demonstrated edge. The raw SR is noise or selection. | | `luck` with a big positive t-stat | Check IC — the mean was positive but the predictions did not rank outcomes. Also check the snapshot `ts`. | | `edge`, DSR 50–90% | Promising, unproven. MinTRL tells you how much more evidence is needed. | | `edge`, DSR ≥ 95%, sensible MinTRL | The strongest claim this platform will ever make. | | Huge SR with tiny n or contradictory hit-rate | Suspect a stale snapshot from before a methodology fix; the current evaluator gates triplets to n ≥ 20 and signs returns by direction. | ## 8. References * Bailey, D. H., & López de Prado, M. (2012). *The Sharpe Ratio Efficient Frontier.* Journal of Risk, 15(2), 3–44. — PSR and MinTRL. * Bailey, D. H., & López de Prado, M. (2014). *The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality.* — DSR and the E[max SR of N] benchmark. * Mertens, E. (2002). — variance of the Sharpe estimator under non-normality (the skew/kurtosis term inside PSR/DSR). Implementation: `core/quant_stats/probabilistic_sharpe.py`, `core/quant_stats/deflated_sharpe.py`, `core/trial_ledger.py` (N and the triplet), `brain/loop_evaluator.py` (verdicts, n_eff, calibration).