Holdout Labs

Open source

The engine is open. The judgment is the service.

You should be able to check our arithmetic. The public toolkit is being prepared. For now, inspect the published methods, sealed rubrics and complete sample reports here.

What the public toolkit includes

  • Deflated Sharpe ratio, PBO via CSCV, Harvey–Liu haircut, and Hansen SPA with the stationary bootstrap
  • Holdout, stability, parameter-plateau and cost-headroom checks
  • The A–F grade rubric, exactly as applied in client reports
  • The HTML report generator
  • Preregistration sealing with RFC 3161 timestamps (DigiCert and FreeTSA), and verification
  • The code behind the public sample audits, so you can reproduce them

What the service adds

  • Reproducing your backtest from your rules, code or platform export, and finding where it differs
  • Setting the holdout, variant count and benchmark honestly, which is where most self-audits go wrong
  • Reading the results: which failures matter for your strategy, and a specific fix list
  • Running and maintaining a sealed forward trial
  • An independent third party’s name on the report

Run it yourself

Install from GitHub with pip install git+https://github.com/holdoutlabs-io/holdout-audit (Python 3.11+), then run holdout-audit --help. The samples/ folder has the code behind every public audit, with fetch scripts and SHA-256 manifests so you can rebuild the same data and check our numbers yourself. A PyPI package will follow.

Apache-2.0 licensed. Source, tests and the code behind every public sample audit: github.com/holdoutlabs-io/holdout-audit.