SMARTBETA-1: USMV (iShares MSCI USA Min Vol Factor ETF)
Buy-and-hold USMV (iShares MSCI USA Min Vol Factor ETF) against buy-and-hold SPY, 2011-10-20 to 2026-09-25. Claim tested (sealed before any return data were loaded): MSCI (2013): 'the MSCI Minimum Volatility Indexes exhibited sizable risk reduction' and (Exhibit 11, which includes the Minimum Volatility index) 'All the MSCI World Factor Indexes shown in Exhibit 11 outperform the MSCI World Index'. iShares today: 'The fund offers a way to reduce risk while maintaining equity exposure'. Part of case study SMARTBETA-1 (6 ETFs, Holm across the six). Non-blind (prices seen before the preregistration).
Executive summary
Mixed. Part of the historical edge looks explainable by selection, period or cost effects. The grade measures how fragile the historical evidence is. It is not a forecast and not a recommendation.
Declared objective: reduce drawdown at acceptable cost, declared by the client on 2026-09-26T15:59:15Z (declaration SHA-256 34236ca9ddfba940…). Declared return-cost tolerance: 1.0% a year.
Benchmark: buy-and-hold of the traded asset. This is the rubric v1.2 default. The drawdown reduction was not robust or exceeded the declared cost, so the grade is at most C.
The second row is context on the beat-the-benchmark view; it is not graded under the declared objective.
Checks that failed: Primary: return cost within the declared tolerance.
Scorecard
Swipe sideways to see the whole table →
| Check | Result | Value | Rule |
|---|---|---|---|
| Selection-adjusted tail-loss reduction (deflated dCVaR) | PASS | D 1.000 (N = 1) | PASS deflated dCVaR >= 0.95; CAUTION >= 0.80; else FAIL (DSR construction on the CVaR reduction) |
| Probability of backtest overfitting (CSCV, dCVaR) | N/A | not computed | PASS PBO <= 0.20; CAUTION <= 0.50, or PBO > 0.50 with P(OOS dCVaR < 0) <= 0.10; else FAIL (variants ranked by dCVaR) |
| Multiple-testing adjusted significance of the tail-loss reduction | PASS | adj. p 0.0000 (Bonferroni) | PASS one-sided adjusted p <= 0.05; CAUTION <= 0.10; else FAIL (bootstrap z of dCVaR) |
| Primary: drawdown and tail-loss reduction vs benchmark | CAUTION | dMDD 0.6% (5th pct -2.6%) | PASS 5th-percentile bootstrap dMDD > 0 and dCVaR > 0; CAUTION both point estimates > 0; else FAIL |
| Primary: return cost within the declared tolerance | FAIL | shortfall 3.7%/yr vs 1.0% | PASS 95th-percentile bootstrap return shortfall <= declared tolerance; CAUTION point shortfall <= tolerance; else FAIL |
| Holdout: tail-loss and drawdown reduction | PASS | dCVaR 48.4 bps -> 72.1 bps | PASS holdout and in-sample dCVaR > 0, holdout >= 50% of in-sample, and holdout dMDD > 0; CAUTION holdout dCVaR > 0; else FAIL |
| Stability of the drawdown reduction | PASS | 88% of years shallower | PASS >= 60% of years with a shallower drawdown than the benchmark and dCVaR > 0 in every volatility tercile; CAUTION >= 50%; else FAIL |
| Transaction-cost headroom (absolute) | PASS | break-even 17684.6 bps | PASS break-even cost >= 20 bps per unit traded; CAUTION >= 5 bps; else FAIL (absolute) |
| Parameter plateau (dCVaR) | N/A | no grid | PASS neighbour dCVaR >= 70% of chosen dCVaR; CAUTION >= 40%; else FAIL, or FAIL if chosen dCVaR <= 0 |
How the grade is set (Holdout Labs Fragility Rubric v1.2, sealed before this report was produced). PASS = 2 points, CAUTION = 1, FAIL = 0; N/A checks are excluded. Share of available points: A ≥ 85%, B ≥ 70%, C ≥ 55%, D ≥ 40%, otherwise F. Hard caps: Objective cap (beat the benchmark): if the strategy does not beat its benchmark (SPA check FAIL, or annualised mean excess return <= 0), the grade is at most C. Objective cap (reduce drawdown): if the drawdown reduction is not robust (drawdown check FAIL) or the declared return-cost tolerance is breached (tolerance check FAIL), the grade is at most C. Objective cap (improve Sharpe): if the Sharpe-ratio improvement is not robust (Sharpe check FAIL), the grade is at most C. Selection cap: if the deflated check FAILS, the grade is at most C. Overfitting cap: if the PBO check FAILS, the grade is at most D. Disclosure cap: if the number of variants tried was not declared and no variant matrix was supplied, the grade is at most B.
What was audited
| Sample | 2011-10-21 to 2026-09-25 (3753 periods, 252 per year) |
| Variants tried (N) | 1 |
| Base-case cost | 5 bps per unit traded (one way) |
| Holdout split | 2022-03-31 |
| Compound annual return, strategy (net) | 11.6% |
| Buy-and-hold of the underlying, same period | compound annual return 15.2%; annualised Sharpe 0.93; max drawdown 33.7% |
Results in detail
1. Sharpe ratio and its uncertainty
Rubric checks use the excess series (strategy minus buy-and-hold of the traded asset); the absolute series is shown for reference.
| Excess over benchmark | Absolute | |
|---|---|---|
| Annualised Sharpe (sqrt-time scaling) | -0.470 | 0.888 |
| Annualised Sharpe (Lo 2002 autocorrelation-adjusted) | -0.516 | 1.042 |
| Standard error, annualised (non-normal) | 0.259 | 0.265 |
| Standard error per period: IID-normal / non-normal | 0.0163 / 0.0163 | 0.0163 / 0.0167 |
| Skewness / kurtosis | -0.11 / 10.40 | -0.57 / 23.35 |
| Probabilistic Sharpe ratio vs 0 | 0.0347 | 0.9996 |
| Minimum track record for 95% confidence | ∞ years | 3.6 years |
| Annualised mean / volatility | -3.7% / 7.8% | 11.9% / 13.4% |
2. The variant-count effect (Deflated Sharpe)
| N tried | Noise hurdle (annual excess SR) | DSR |
|---|---|---|
| 1 | 0.000 | 0.0347 |
| 2 | 0.135 | 0.0097 |
| 5 | 0.309 | 0.0013 |
| 10 | 0.408 | 0.0003 |
| 20 | 0.493 | 0.0001 |
| 50 | 0.590 | <0.0001 |
| 100 | 0.656 | <0.0001 |
| 200 | 0.717 | <0.0001 |
| 500 | 0.791 | <0.0001 |
| 1000 | 0.843 | <0.0001 |
Dispersion of trial Sharpe ratios used: 0.259 (annualised standard deviation), set to the null sampling error 1/sqrt(T).
3. Probability of backtest overfitting (CSCV)
Not computed: no variant matrix supplied.
4. Multiple-testing haircut
| t-statistic (excess SR x sqrt(years)) | -1.82 |
| p-value, single test | 0.96526 |
| Bonferroni p (1 tests) | 0.96526 |
| Sidak p | 0.96526 |
| Holm p | n/a |
| BHY p | n/a |
| Haircut excess Sharpe (Bonferroni) | -0.000 (100% haircut) |
| Haircut excess Sharpe (BHY) | n/a |
5. Reality Check and SPA
| Benchmark | buy-and-hold of the traded asset |
| Variants in the test | 1 |
| Best mean excess return (annualised) | -3.68% |
| White Reality Check p | 0.972 |
| Hansen SPA p (consistent / lower / upper) | 1.000 / 1.000 / 1.000 |
| Bootstrap | stationary, 1000 draws, mean block 16 |
6. Holdout degradation
| Periods | Annualised excess Sharpe | |
|---|---|---|
| In-sample (before 2022-03-31) | 2627 | -0.349 |
| Holdout | 1126 | -0.681 |
| Holdout / in-sample | n/a | |
| Consistency p-value | 0.483 |
7. Period and regime stability
| Volatility regime (trailing 21-day, lagged) | Periods | Annualised mean excess | Annualised excess Sharpe |
|---|---|---|---|
| low vol | 1244 | -1.1% | -0.19 |
| mid vol | 1244 | -4.3% | -0.60 |
| high vol | 1244 | -5.7% | -0.59 |
8. Transaction-cost sensitivity
| One-way cost (bps per unit traded) | 0 | 1 | 2 | 5 | 10 | 20 | 50 |
|---|---|---|---|---|---|---|---|
| Annualised Sharpe | 0.89 | 0.89 | 0.89 | 0.89 | 0.89 | 0.89 | 0.89 |
| Annualised mean return | 11.9% | 11.9% | 11.9% | 11.9% | 11.9% | 11.9% | 11.8% |
Turnover 0.1 units per year; break-even cost 17684.6 bps.
9. Parameter sensitivity
Not computed: no parameter grid supplied.
10. Drawdown distribution
11. Declared objective: drawdown reduction at acceptable cost
| Strategy | Benchmark | Reduction | Bootstrap bound | |
|---|---|---|---|---|
| Maximum drawdown | 33.1% | 33.7% | 0.6% | 5th pct -2.6% |
| Daily CVaR95 (bps) | 194.9 | 250.5 | 55.6 | 5th pct 46.7 |
| Return shortfall a year (tolerance 1.0%) | 3.7% | 95th pct 6.9% |
Deflated dCVaR 1.000 against a best-of-1 hurdle of 0.0 bps; adjusted one-sided p <0.0001. Holdout: dCVaR 48.4 bps before 2022-03-31, 72.1 bps after; holdout drawdown reduction 5.4%. Years with a shallower drawdown than the benchmark: 14 of 16.
What is fragile, and how to test the fix
N/A Probability of backtest overfitting (CSCV, dCVaR)
No variant return matrix was supplied, so PBO could not be computed.
CAUTION Primary: drawdown and tail-loss reduction vs benchmark
Maximum drawdown 33.1% against 33.7% for the benchmark (reduction 0.6%; bootstrap 5th percentile -2.6%). Daily CVaR95 reduction 55.6 bps (5th percentile 46.7 bps). Paired stationary bootstrap, 1000 draws.
How to test a fix: A reduction that only shows up in one or two crashes is fragile: test it on a period with different crashes, or in a sealed forward trial.
FAIL Primary: return cost within the declared tolerance
The strategy gave up 3.7% a year of arithmetic return against the benchmark (bootstrap 95th percentile 6.9%). The client declared a tolerance of 1.0% a year on 2026-09-26T15:59:15Z.
How to test a fix: The tolerance was fixed in advance and cannot be revised for this audit; a changed rule is a new variant and needs a new declaration.
N/A Parameter plateau (dCVaR)
No parameter grid was supplied.
These are suggestions for further statistical testing, not suggestions to trade. Any change to the rules creates a new variant: count it in N and confirm it on data it was not designed on.
What held up
- Selection-adjusted tail-loss reduction (deflated dCVaR): Daily CVaR95 falls from 250.5 bps (benchmark) to 194.9 bps (strategy): a reduction of 55.6 bps (bootstrap SE 5.2 bps). The best-of-1 noise hurdle is 0.0 bps; the probability that the true reduction exceeds it is 1.000.
- Multiple-testing adjusted significance of the tail-loss reduction: One-sided p that the tail-loss reduction is zero or worse: 0.0000 for the audited strategy; adjusted for 1 tests (Bonferroni): 0.0000.
- Holdout: tail-loss and drawdown reduction: Tail-loss reduction 48.4 bps before 2022-03-31 and 72.1 bps after; holdout drawdown reduction 5.4%.
- Stability of the drawdown reduction: The strategy's within-year drawdown was shallower than the benchmark's in 14 of 16 years. dCVaR by volatility tercile: low vol 49.2 bps, mid vol 37.4 bps, high vol 63.5 bps.
- Transaction-cost headroom (absolute): The gross mean return falls to zero at 17684.6 bps per unit traded.
Methods appendix
Sharpe ratio and its standard error
Per-period mean over standard deviation of the audited return series (risk-free rate taken as zero), annualised by the square root of periods per year. The standard error uses the IID-normal formula of Lo (2002) and the non-normal correction of Mertens (2002), which widens the error for negative skew and fat tails. Lo's autocorrelation-adjusted annualisation is reported alongside. [1], [2]
Probabilistic Sharpe ratio (PSR) and minimum track record
The probability that the true Sharpe ratio exceeds a benchmark (here zero), given the sample length, skewness and kurtosis; and the minimum sample length for 95% confidence. [3]
Deflated Sharpe ratio (DSR)
The PSR measured against the Sharpe ratio one would expect from the best of N skill-less trials, where N is the number of variants tried and the dispersion of trial Sharpe ratios is estimated from the variant matrix (or, without one, set to the null sampling variance 1/T). It corrects for selection and non-normality at once. [4]
Probability of backtest overfitting (PBO) via CSCV
The variant matrix is cut into 16 time blocks; for each of the 12,870 ways of picking half of them as in-sample, the in-sample best variant is ranked out of sample. PBO is the share of splits where it falls to or below the median. It assumes blocks long enough to preserve serial dependence and a variant set that represents the real search. [5]
Multiple-testing haircut
The Sharpe ratio is turned into a t-statistic and p-value, the p-value is adjusted for the number of tests (Bonferroni always; Holm and BHY when every variant's returns are supplied), and the adjusted p-value is mapped back to a haircut Sharpe ratio. [6], [7]
White's Reality Check and Hansen's SPA test
Tests whether the best of all variants beats the benchmark in mean return once the search over variants is accounted for. Uses the stationary bootstrap with mean block length T^(1/3) (at least 5) to keep short-range dependence. SPA studentises and recentres, so poor variants do not dilute the power. [8], [9], [10]
Holdout degradation
In-sample versus holdout Sharpe ratio at a fixed split date (by default the last 30% of the sample). The consistency p-value asks how surprising the holdout Sharpe would be if the in-sample Sharpe were the truth. A holdout only counts if it was not used to design the rule. [11]
Period and regime stability
Returns and Sharpe ratio by calendar year and by tercile of the underlying market's trailing 21-day volatility (lagged one day). Descriptive: the tercile cut points use the full sample. [12]
Parameter-sensitivity surface
Sharpe ratio across the supplied parameter grid. The neighbour ratio compares the chosen cell with its one-step neighbours: a plateau (ratio near 1) is less fragile than an isolated peak. [13]
Transaction-cost sensitivity
Net return = gross return minus turnover times a one-way cost per unit traded, over a grid of costs; the break-even cost is where the mean net return reaches zero. Market impact beyond a flat cost is not modelled. [12]
Drawdown distribution by bootstrap
The maximum drawdown is recomputed on 1,000 stationary-bootstrap resamples of the return series, showing how much deeper (or shallower) the worst loss could plausibly have been with the same return distribution in a different order. [10]
References
- Lo, A. W. (2002). The Statistics of Sharpe Ratios. Financial Analysts Journal 58(4), 36-52.
- Mertens, E. (2002). Comments on Variance of the IID Estimator in Lo (2002). Working paper, University of Basel.
- Bailey, D. H. & Lopez de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk 15(2), 3-44.
- Bailey, D. H. & Lopez de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5), 94-107.
- Bailey, D. H., Borwein, J. M., Lopez de Prado, M. & Zhu, Q. J. (2016). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4), 39-69.
- Harvey, C. R. & Liu, Y. (2015). Backtesting. Journal of Portfolio Management 42(1), 13-28.
- Harvey, C. R., Liu, Y. & Zhu, H. (2016). ... and the Cross-Section of Expected Returns. Review of Financial Studies 29(1), 5-68.
- White, H. (2000). A Reality Check for Data Snooping. Econometrica 68(5), 1097-1126.
- Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics 23(4), 365-380.
- Politis, D. N. & Romano, J. P. (1994). The Stationary Bootstrap. Journal of the American Statistical Association 89(428), 1303-1313.
- Bailey, D. H., Borwein, J. M., Lopez de Prado, M. & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the AMS 61(5), 458-471.
- Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.
- Pardo, R. (2008). The Evaluation and Optimization of Trading Strategies, 2nd ed. Wiley.
Data and hash appendix
- Prices from Yahoo Finance's public chart endpoint (split- and dividend-adjusted close = total return, fund expenses already deducted); we publish derived statistics only, never raw prices; no redistribution. Yahoo caveats: the adjusted close back-adjusts dividends multiplicatively and can be revised retroactively; rounding affects daily returns slightly; Yahoo is not an audited source.
- yahoo_usmv.csv: SHA-256 cd34f034bb9e00962a33b3ea79f169a78497191866303ecc685b4d4bc420692b, downloaded 2026-09-26T17:11:18Z (after the preregistration seal)
- yahoo_spy.csv: SHA-256 eff3de4ee1361624a3943cdbebf2995b50870790c4459a327fe93e307b52379b, downloaded 2026-09-26T17:11:19Z (after the preregistration seal)
- Audit period: first Yahoo close on or after the inception date 2011-10-18 (2011-10-20) to 2026-09-25; first return is day 2 over day 1; ETF and SPY inner-joined on dates; 3753 daily returns.
- Cost: 5 bps one-way on the day-1 purchase only; fund expenses are inside the adjusted close.
- Declared number of variants N = 1 (one product, run once); multiplicity across the six funds is handled by Holm (PREREG section 7). Holdout: rubric default (start of the last 30%).
- SMARTBETA-1 preregistration SHA-256 42bfad1372704447674cde6efdf17064dd0b290e6be351659783c78d958b7c54, sealed 2026-09-26T16:00:46Z (DigiCert, FreeTSA); OSF registration https://osf.io/2tpn5. The objective declaration is hashed inside the sealed preregistration (section 9) rather than sealed separately.
- NON-BLIND: USMV and SPLV prices, and their Batch 3 result on objective (c), had been seen before this preregistration was written (PREREG section 0).
| Item | SHA-256 / value |
|---|---|
| package | holdout-audit 0.4.0 |
| yahoo_usmv.csv | cd34f034bb9e00962a33b3ea79f169a78497191866303ecc685b4d4bc420692b |
| yahoo_spy.csv | eff3de4ee1361624a3943cdbebf2995b50870790c4459a327fe93e307b52379b |
| audited_series_sha256 | 674b2c072af8210a9887e06488fdf7e6226fc26bc0b0c3e4ef4e59798e3b9ca0 |
| objective_declaration_sha256 | 34236ca9ddfba9402f30ce7f0f35bc3e5686abf093ae57a07b175caf12078cd2 |
| preregistration_sha256 | 42bfad1372704447674cde6efdf17064dd0b290e6be351659783c78d958b7c54 |
| config_sha256 | e699357bd5377f0ea6ce5a2c653227b7abe13757729626e9d530b938121c4cae |
| bootstrap seed | 20260925 |
| rubric version | 1.2 (docs/RUBRIC-v1.2.md) |
| declared objective | reduce_drawdown |
| preregistration seal | 2b4e724baf201a87450a5e1bb302a2dfac29c08eb42e9213df5ef1b28d660241 (digicert, freetsa; earliest 2026-09-26T16:00:46Z) |
| rubric document SHA-256 | 81bc60b5bf4520fdfb91c43b37883c30a9a2061903442d1e8a8b05f3d37a71fc |
| rubric seal record SHA-256 | 142cbfb45314faab8a578b62a17532ac323b0f619f74ab008244163d682db4dd |
| rubric sealed by digicert (RFC 3161) | 2026-09-25T20:33:45Z |
| rubric sealed by freetsa (RFC 3161) | 2026-09-25T20:33:46Z |
Anyone with the same inputs and package version can re-run the audit and must obtain the same audited-series hash and the same statistics.