Holdout Labs

Open source

The engine is open. The judgment is the service.

You should be able to check our arithmetic. The statistics behind every audit are in a public toolkit you can install and run.

holdout-audit on GitHub

What is open

  • Deflated Sharpe ratio, PBO via CSCV, Harvey–Liu haircut, and Hansen SPA with the stationary bootstrap
  • Holdout, stability, parameter-plateau and cost-headroom checks
  • The A–F grade rubric, exactly as applied in client reports
  • The HTML report generator
  • Preregistration sealing with RFC 3161 timestamps (DigiCert and FreeTSA), and verification
  • The code behind the public sample audits, so you can reproduce them

What the service adds

  • Reproducing your backtest from your rules, code or platform export, and finding where it differs
  • Setting the holdout, variant count and benchmark honestly, which is where most self-audits go wrong
  • Reading the results: which failures matter for your strategy, and a specific fix list
  • Running and maintaining a sealed forward trial
  • An independent third party’s name on the report

Run it yourself

# TODO(owner): confirm package name and CLI once the toolkit is published
pip install holdout-audit
holdout-audit --help

License and contribution guidelines are in the repository.