Holdout Labs

Holdout Labs · strategy audits

Is your backtest real?

Scientific audits for trading strategies. We measure how much of your backtest survives the tests that catch overfitting, and tell you what to fix.

Request a Quick Audit — $149 See the published audits
Illustration: a backtest before and after the holdout split A schematic line rises steadily through the in-sample period, then flattens and wanders after the split into the held-out period. IN-SAMPLE: WHAT YOU OPTIMIZED ON HELD OUT NEVER SEEN
Schematic, not data. The question an audit answers: does the curve you optimized keep its shape on data the search never touched?

How it works

Three steps, fixed price, written report

  1. Send your strategy

    Your rules or code, the data or a data recipe, and an honest count of how many variants you tried before this one. That count matters more than most people expect.

  2. We test it for overfitting

    Deflated Sharpe, probability of backtest overfitting, multiple-testing haircuts, a reality check against data snooping, holdout degradation, and stress tests on costs, parameters and regimes. Every method is published and cited.

  3. You get a grade and a fix list

    An A–F fragility grade computed by a rubric printed in the report, the numbers behind it, and a ranked list of what to change before you trust the backtest.

Published audits

We audit famous rules in public, and publish every result

Each batch is preregistered and sealed with RFC 3161 timestamps before we fetch the data, then run once, pass or fail.

F

Batch 2 · stocks and bonds · 14 audits

Classic market rules: Faber, Siegel’s 200-day, Halloween, dual momentum, 60/40 bands and more

Ten published rules on SPY, QQQ, IWM, EFA, sector and Treasury ETFs. 0 of 14 beat their benchmark with statistical support.

Read the Batch 2 report
F

Batch 1 · US equities

Ten TradingView built-in strategies on SPY

MACD, RSI, Bollinger Bands, Supertrend and six more, as they ship. None beat holding SPY.

Read the Batch 1 report
F

Sample audit

The SPY golden cross

Sharpe 0.77, but 1.8% a year behind holding SPY, and further behind in the holdout.

Read sample audit 1

All audits, including multi-asset and crypto.

Services

Three levels of scrutiny

Quick Audit

$149

3–5 business days

  • Holdout split and walk-forward check
  • Deflated Sharpe for the variants you tried
  • Parameter and cost sensitivity
  • A–F fragility grade and fix list
Request a Quick Audit — $149

Send a high-level description. We’ll confirm scope, then send payment and secure-upload instructions.

Deep Audit

$490

5–10 business days

  • Everything in the Quick Audit
  • PBO via CSCV, Harvey–Liu haircut, White / Hansen SPA
  • Regime splits, placebo strategies, bootstrap drawdowns
  • Report hash timestamped under RFC 3161
Request a Deep Audit — $490

Send a high-level description. We’ll confirm scope, then send payment and secure-upload instructions.

Sealed Forward Trial

$1,500 + $99/mo

60–90 day live registry

  • Preregistered protocol, frozen before the trial
  • Daily hypothetical signals sealed with two independent timestamp authorities
  • A certificate anyone can verify with openssl
Request a Sealed Trial

Send a high-level description. We’ll confirm scope, then send payment and secure-upload instructions.

Why trust the grade

We hold ourselves to the same tests

  • Built by a practitionerFounded by Raghuram Vadapally, former Managing Director and Head of Software Engineering at Carlson Capital, a multi-strategy hedge fund, with earlier trading and risk technology roles at Eton Park, Sculptor and J.P. Morgan. About the founder.
  • Published methodsReports cite established statistical methods and print our own grading rubric, thresholds and exceptions included. No proprietary score, no black box. Read the method.
  • Public toolkitThe public toolkit is being prepared. For now, inspect the published methods, sealed rubrics and complete sample reports here. What it will include.
  • Positive controlsThe test can pass real edges: in 200 sealed synthetic trials each, a real excess Sharpe of 1.0 got an A or B 98% of the time, and pure noise 2%. How we checked.
  • Preregistered case studyWe tested a widely circulated lunar-cycle claim about Bitcoin volatility on ourselves: two preregistrations, timestamped seals, ten rounds of red-team review. The retrospective test found no useful effect (not prospectively tested), and we published it. Read the case study.
  • No signals, no promisesWe never tell you what to buy or sell and never promise performance. The grade describes the evidence, not the future.

Find out before the market does the audit for you.

A Quick Audit is $149, fixed price, delivered in 3–5 business days. If it turns out your backtest is overfit, you get a list of what to fix.

Request a Quick Audit — $149 Ask a question first

Send a high-level description. We’ll confirm scope, then send payment and secure-upload instructions.