Holdout Labs

Method

Published statistics, explained plainly

Every number in an audit comes from the peer-reviewed literature. Here is what each one measures, why it matters, and the rule that turns it into PASS, CAUTION or FAIL. The rules are frozen in a versioned rubric that is sealed before it is used. The current version is v1.1.

The checks

Eight tests, each with its source

By default, a strategy is graded against simply holding the asset it trades (buy-and-hold). The selection, significance, holdout and stability checks below are computed on the excess return over that benchmark, so a rule that makes money but trails buy-and-hold does not pass. Cost headroom and the parameter plateau are measured on the strategy’s own returns.

Selection

Deflated Sharpe ratio

If you try fifty versions of a strategy and keep the best, its Sharpe ratio is inflated by the search itself. The deflated Sharpe ratio asks: given how many variants were tried, how uneven the data are, and how long the sample is, how likely is it that the true excess Sharpe ratio over the benchmark beats what the best of that many skill-less variants would show by luck? You tell us how many variants you tried; we show how the answer changes if the real number was higher.

PASS DSR ≥ 0.95 · CAUTION ≥ 0.80 · else FAIL (on excess returns)

Bailey, D. H. & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5), 94–107. SSRN 2460551

Overfitting

Probability of backtest overfitting (PBO), via CSCV

We cut the history into blocks and form every balanced split into a “training” half and a “testing” half. For each split we pick the best variant on training and see where it ranks on testing. PBO is the share of splits where the training winner lands in the bottom half out of sample. A PBO near 0.5 means picking the best backtest is no better than picking at random.

PASS PBO ≤ 0.20 · CAUTION ≤ 0.50, or above 0.50 when the training winner trails the benchmark out of sample no more than 10% of the time · else FAIL

Bailey, D. H., Borwein, J. M., López de Prado, M. & Zhu, Q. J. (2016). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4), 39–69. SSRN 2326253

Multiple testing

Harvey–Liu haircut

A t-statistic of 2 is persuasive for one test and weak for a hundred. The Harvey–Liu method adjusts the strategy’s p-value for the number of tests behind it (Bonferroni, Holm and BHY adjustments) and translates the result into a “haircut”: how much of the backtested excess Sharpe ratio is left once the search is accounted for. Since rubric v1.1 the p-value is one-sided: only evidence for the strategy counts. (Under v1.0’s two-sided p-value, a strategy trailing its benchmark by a significant margin could have passed this check. We found that defect after sealing v1.0, logged it, and fixed it in v1.1.)

PASS one-sided adjusted p ≤ 0.05 · CAUTION ≤ 0.10 · else FAIL

Harvey, C. R. & Liu, Y. (2015). Backtesting. Journal of Portfolio Management 42(1), 13–28. SSRN 2345489

Data snooping

White’s Reality Check and Hansen’s SPA test

These tests ask whether the best variant in your search really has an edge over a benchmark, or whether the best of many random-looking variants would have looked that good anyway. They resample the whole family of variants together, so the answer accounts for the search. Hansen’s version is less distorted by poor variants in the family.

PASS p ≤ 0.05 · CAUTION ≤ 0.10 · else FAIL

White, H. (2000). A Reality Check for Data Snooping. Econometrica 68(5), 1097–1126. Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics 23(4), 365–380.

Out of sample

Holdout degradation

We keep the most recent part of the history back, untouched, and compare the excess Sharpe ratio over the benchmark before and after the split. Some decay is normal. A strategy that keeps less than half its in-sample Sharpe ratio, or turns negative, was probably fitted to noise.

PASS holdout and in-sample excess SR > 0, holdout ≥ 50% of in-sample · CAUTION holdout excess SR > 0 · else FAIL

Standard practice; see Bailey et al. (2016) above on why a single split is necessary but not sufficient.

Stability

Period and regime stability

Does the result come from one lucky year or one kind of market? We break the history into calendar years and into calm, normal and turbulent volatility regimes, and check that the strategy stays ahead of its benchmark in each.

PASS ≥ 60% of years ahead of the benchmark and excess SR > 0 in every regime · CAUTION ≥ 50% · else FAIL

Descriptive check. Regimes are volatility terciles computed from the data, fixed before results are looked at.

Robustness

Parameter and cost sensitivity

A real effect usually sits on a plateau: nearby parameter values work almost as well. A fragile one sits on a spike. We map the neighborhood of your chosen parameters, and raise transaction costs until the edge disappears to find its break-even cost.

Plateau: PASS neighbors ≥ 70% of chosen SR · Costs: PASS break-even ≥ 20 bps

Descriptive checks. Break-even cost is reported in basis points per unit traded.

Risk

Bootstrap drawdowns

The single drawdown in your backtest is one draw from many that could have happened. We resample the history in blocks, which keeps streaks and volatility clusters intact, and report the range of maximum drawdowns you should be prepared for.

Politis, D. N. & Romano, J. P. (1994). The Stationary Bootstrap. Journal of the American Statistical Association 89(428), 1303–1313.

Grading

The A–F fragility grade

The grade describes how fragile the historical evidence is. It is not a forecast and not a recommendation. The rubric is printed in every report, so you can recompute your grade by hand.

Your objective, declared first

Before an audit starts, you choose what success means, and you put that choice in writing:

  • (a) Beat the benchmark (the default): the eight checks above, on excess returns over buy-and-hold.
  • (b) Reduce drawdown at acceptable cost:
    • the primary checks are the reduction in maximum drawdown and in daily tail loss (CVaR95) against the benchmark, with bootstrap intervals, plus a return shortfall that stays within a tolerance you declare in advance (for example, “up to 2% a year less than buy-and-hold”);
    • the same overfitting corrections apply: a deflated statistic, PBO, multiple testing, holdout, stability and the plateau.

The declaration must be dated before your data arrive, its SHA-256 is printed in the report, and you can seal it with RFC 3161 timestamps. Choosing the objective after seeing the results is the forking path an audit exists to catch, so we refuse a declaration dated after the data.

Scoring

Each check scores PASS = 2, CAUTION = 1, FAIL = 0. A check that cannot be run (for example PBO without a variant matrix) is marked N/A and left out. The grade is the share of available points:

GradeShare of pointsMeaning
A≥ 85%Robust on every test we ran. The evidence survives selection, holdout and cost stress.
B≥ 70%Mostly robust. One or two weaker spots worth testing further.
C≥ 55%Mixed. Part of the historical edge looks explainable by selection, period or cost effects.
D≥ 40%Fragile. Most of the historical edge does not survive the stress tests.
F< 40%Very fragile. The backtest shows no robust edge over its benchmark once selection, holdout and stability are accounted for.

Hard caps

Some failures matter more than a points total can show, so four caps apply after scoring. The most severe one wins:

  1. Objective cap. Under (a), a strategy that does not beat its benchmark (SPA check FAIL, or no positive excess return) gets at most C. Under (b), a drawdown reduction that is not robust, or a return shortfall above your declared tolerance, gets at most C.
  2. If the deflated check fails, the grade is at most C.
  3. If the PBO check fails, the grade is at most D.
  4. If you do not declare how many variants you tried and supply no variant matrix, the grade is at most B. We cannot rule out a search we cannot see.

Every FAIL or CAUTION comes with a specific item on the fix list: what to change, and which check it addresses.

Sealed before use

Each rubric version is frozen, hashed and timestamped by DigiCert and FreeTSA (RFC 3161) before any audit is graded under it. Reports print the version and the seal hash:

  • v1.1 (current). Sealed 2026-09-25 14:20:10 UTC.
    Document SHA-256 2311b126336bab9f426a84f9c9cd9cb937e346b403a62f60e3c96ed8b5868e46
    Seal record SHA-256 ac882c422df5154ec7e6e928904c9099d6b286dc9e5369fece4b85df57044138
  • v1.0 (the sample audits). Sealed 2026-09-25 14:02:23 UTC.
    Document SHA-256 0c0420617f2f460d452d862b5e522171bef5e91a259a845d77449bfa115a7292
    Seal record SHA-256 efad782bc139e091f5eb6573e2aa6764f2bcfa5340c455593298152fbd5009ac

What changed in v1.1: the haircut became one-sided, and the declared objective was added. The unsealed v0 drafts were tuned while we looked at the sample audits. The change log records that, and nothing graded under them is published.

Discipline

The holdout lock

A holdout only works if nobody looks at it. Once a period of data has been used to choose anything, even a chart glance that nudges a parameter, it is no longer out of sample.

So we fix the split before any analysis starts: which dates are held back, which statistics will be computed on them, and what counts as a pass. The split is written down and hashed before the data are touched. The held-out period is scored once, at the end.

If you have already looked at your recent data, tell us. We will say so in the report and move the holdout to data you have not used, or recommend a sealed forward trial, which creates a holdout that cannot have been seen: the future.

Sealed forward trial

Timestamps anyone can check

In a sealed forward trial the protocol, code and data manifest are frozen and hashed (SHA-256). The hash is sent to two independent RFC 3161 timestamp authorities, DigiCert and FreeTSA, which sign it together with the current time. Each day’s hypothetical signals are sealed the same way, before the outcome exists.

Nothing here depends on trusting us. A timestamp token proves that exact file existed at that exact time. Change one byte and verification fails.

RFC 3161: Adams, C. et al. (2001). Internet X.509 Public Key Infrastructure Time-Stamp Protocol (TSP). IETF.

Verify it yourself

sha256sum manifest.json
openssl ts -verify -in manifest.digicert.tsr \
  -data manifest.json -CAfile digicert-root.pem
openssl ts -verify -in manifest.freetsa.tsr \
  -data manifest.json -CAfile freetsa-cacert.pem \
  -untrusted freetsa-tsa.crt
# Verification: OK

Results from a sealed trial remain hypothetical performance: they show what a rule would have signalled, not what any account earned.