Case study · SMARTBETA-1 · preregistered
Smart beta, backtest vs live
0 of 6 flagship smart-beta ETFs delivered on their launch-era claims after launch. Momentum finished ahead of SPY, but not by enough to tell from chance. Quality, value and size went the other way. The two low-volatility funds cut day-to-day risk, but did not reliably reduce their worst drawdown, and gave up far more return than the 1% a year allowed in advance.
Method: the test was fixed, sealed with RFC 3161 timestamps and registered on OSF before any return data were loaded. It was run once, and every result is published (proof).
The question
The first wave of single-factor ETFs launched in 2011–2013 on index histories that were largely back-tested, and on academic factor premia measured over decades. The years since launch are a natural holdout: data no backtest could have seen.
After launch, did each fund deliver what was claimed for it, on the total returns investors actually received, against buy-and-hold SPY? We took the six flagship US single-factor ETFs of that wave: iShares’ MSCI USA momentum, quality, value, size and minimum-volatility funds, and SPLV on the S&P 500 Low Volatility Index. Each is tested as sold, index changes included, from its first close to 2026-09-25.
How it was done
- Claim fixed first. Each fund’s objective follows its launch-era claim, quoted from MSCI, iShares or S&P Dow Jones Indices. Return claims are tested as “beat SPY”; risk-reduction claims as “reduce drawdown, giving up no more than 1.0% a year”.
- Sealed and registered before data. Funds, objectives, tests, costs, the multiple-testing correction and the verdict wording were sealed and registered on OSF before a single price or factor return was loaded.
- Run once. One test per fund under our sealed rubric v1.2, Holm’s correction across the six, family-wise α = 0.05. “Delivered on its claim” needs a Holm p ≤ 0.05 and no failed objective check.
- Publish all. Every fund, every check and every descriptive table, whatever it showed.
Result
None of the six delivered on its claim
Primary results: one confirmatory test per fund
Full live sample of each fund against SPY on the same dates, net of a 5 bps entry cost; fund expenses are already in the prices. Intervals are 90% paired stationary-bootstrap intervals. The verdict wording was fixed in the preregistration (§7). The grade links to the full audit report for that fund.
| Fund | Launch-era claim | Objective tested | Key statistics vs SPY | Holm p | Verdict | Grade | Why? |
|---|---|---|---|---|---|---|---|
| MTUMiShares MSCI USA Momentum Factor ETF, from 2013-04-18 | “The MSCI Momentum and MSCI Quality Indexes historically exhibited the strongest historical returns”MSCI, Foundations of Factor Investing, Dec 2013 | (a) beat SPY | Average excess +1.7 pts/yr [−2.1, +5.5] CAGR 16.0% vs 14.7% | 1.00 | Not shown to deliver | D | Why? MTUMMTUM: momentum What happened. MTUM finished ahead of SPY by 1.7 points a year on average. Compounded, that was 16.0% vs 14.7% a year: $1 became $7.32 against $6.29. It was ahead in 8 of 14 calendar years, with higher volatility (20.2% vs 16.9%). Why “not shown to deliver”. The lead is small next to MTUM’s year-to-year swings against SPY (+15.8 points in 2017, −15.4 in 2021, −17.0 in 2023). A lead this size shows up by chance about 1 time in 4 (p = 0.25, Holm 1.00). The 90% interval, −2.1 to +5.5 points a year, includes zero. Why D. The selection, significance and benchmark (SPA) checks fail; the holdout check passes. Descriptive. The lead came under the original index: +3.1 points a year to November 2020, then −0.1 after the switch to the MSCI USA Momentum SR Variant Index. |
| QUALiShares MSCI USA Quality Factor ETF, from 2013-07-18 | “The MSCI Momentum and MSCI Quality Indexes historically exhibited the strongest historical returns”MSCI, Dec 2013 | (a) beat SPY | Average excess −0.4 pts/yr [−1.8, +0.9] CAGR 13.6% vs 14.1% | 1.00 | Went the other way | F | Why? QUALQUAL: quality What happened. QUAL trailed SPY by 0.4 points a year on average (compound 13.6% vs 14.1%; $1 became $5.38 against $5.71). It was ahead in 5 of 14 calendar years, with almost the same volatility (17.0% vs 16.9%). Why “went the other way”. The claim was a higher return; the point estimate is below SPY. The 90% interval (−1.8 to +0.9 points a year) is narrow: a live premium of more than about 1 point a year is hard to reconcile with this record. Why F. The selection, significance, benchmark, holdout and stability checks all fail. Descriptive. +1.7 points a year under the original MSCI USA Quality Index (2013–2015), −0.8 after the switch to the Sector Neutral Quality Index in August 2015. With two distributions missing from the price data added back, the average excess is −0.37 instead of −0.43; nothing changes (DEVIATION-001). |
| VLUEiShares MSCI USA Value Factor ETF, from 2013-04-18 | “Tilts towards these factors such as Value, Low Size, and Momentum historically produced excess long-term returns”MSCI, Dec 2013 | (a) beat SPY | Average excess −0.5 pts/yr [−4.4, +3.5] CAGR 13.8% vs 14.7% | 1.00 | Went the other way | F | Why? VLUEVLUE: value What happened. VLUE trailed SPY by 0.5 points a year on average (compound 13.8% vs 14.7%; $1 became $5.64 against $6.29). It was ahead in 7 of 14 calendar years, with more volatility (18.7% vs 16.9%) and a deeper worst fall (39.5% vs 33.7%). Why “went the other way”. The point estimate is below SPY. But the swings were large (−18.6 points in 2020, −17.6 in 2024, +35.3 so far in 2026), so the 90% interval is wide, −4.4 to +3.5 points a year: these data cannot rule out a modest premium in either direction. Why F. The selection, significance and benchmark checks fail; holdout and stability are CAUTION. Data note. With a distribution missing from the price data added back, the average excess is −0.43 instead of −0.48; nothing changes (DEVIATION-001). |
| SIZEiShares MSCI USA Size Factor ETF, from 2013-04-18 | “Tilts towards these factors such as Value, Low Size, and Momentum historically produced excess long-term returns”MSCI, Dec 2013 | (a) beat SPY | Average excess −2.6 pts/yr [−4.9, −0.2] CAGR 11.7% vs 14.7% | 1.00 | Went the other way | F | Why? SIZESIZE: size What happened. SIZE trailed SPY by 2.6 points a year on average (compound 11.7% vs 14.7%; $1 became $4.41 against $6.29). It was ahead in only 3 of 14 calendar years. Why “went the other way”. The claim was an excess return; the whole 90% interval, −4.9 to −0.2 points a year, sits below zero. Why F. Every check except cost headroom fails, including the holdout: its excess Sharpe ratio fell from −0.14 before the holdout to −0.86 in it. Descriptive. −1.1 points a year under the launch index (MSCI USA Risk Weighted), −3.6 under the MSCI USA Low Size Index from November 2018. With a missing distribution added back, the average excess is −2.53 instead of −2.56; nothing changes (DEVIATION-001). |
| USMViShares MSCI USA Min Vol Factor ETF, from 2011-10-20. Non-blind. | “sizable risk reduction”MSCI, Dec 2013“a way to reduce risk while maintaining equity exposure”iShares | (b) reduce drawdown, giving up ≤ 1.0%/yr | Max drawdown 33.1% vs 33.7% Volatility 13.4% vs 16.7% Daily tail-loss cut 56 bps Return cost 3.7%/yr vs 1.0% tolerance | <0.0001 tail-loss cut only | Not shown to deliver (non-blind) | C | Why? USMVUSMV: minimum volatility What held up. Lower risk day to day: volatility 13.4% against SPY’s 16.7%, and on its worst 5% of days it lost 56 bps less on average (p < 0.0001; the cut was larger in the holdout). What didn’t. Its worst peak-to-trough fall, 33.1%, was barely shallower than SPY’s 33.7%, and not reliably so (drawdown check CAUTION). It also gave up 3.7 points a year of average return, far above the 1.0% tolerance declared in advance (FAIL). $1 became $5.13 against $8.23. Why C, not “delivered”. The claim was less risk at an acceptable cost; both halves must hold. The failed tolerance check caps the grade at C. Descriptive. Beta to SPY 0.71; CAPM alpha +0.4% a year (90% interval −2.3% to +3.0%). A low-volatility fund can show a positive alpha and a negative raw excess return at the same time. Non-blind: its prices and an earlier Sharpe-ratio result (Batch 3) had been seen before this test was written. |
| SPLVInvesco S&P 500 Low Volatility ETF, from 2011-05-05. Non-blind. | “in back-tested results, has out-performed the S&P 500 over the near, medium, and long term, with less volatility”S&P Dow Jones Indices, Dec 2011, quoted in an SEC filing | (b) reduce drawdown, giving up ≤ 1.0%/yr | Max drawdown 36.3% vs 33.7% Volatility 14.4% vs 17.1% Daily tail-loss cut 51 bps Return cost 4.4%/yr vs 1.0% tolerance | <0.0001 tail-loss cut only | Not shown to deliver (non-blind) | C | Why? SPLVSPLV: low volatility What held up. Lower risk day to day: volatility 14.4% against SPY’s 17.1%, and on its worst 5% of days it lost 51 bps less on average (p < 0.0001). What didn’t. Its worst peak-to-trough fall was deeper than SPY’s, 36.3% vs 33.7% (drawdown check FAIL). It gave up 4.4 points a year of average return against the 1.0% tolerance (FAIL). $1 became $4.10 against $7.57. The “out-performed” claim. Tested as a secondary objective: SPLV trailed SPY by 4.4 points a year, the opposite direction (Holm p 1.00). Why C. The failed drawdown and tolerance checks cap the grade at C. Descriptively, beta to SPY 0.68 and CAPM alpha −0.2% a year (90% interval −3.9% to +3.5%). Non-blind: its prices had been seen in earlier audits. |
“Average excess” is the arithmetic statistic the rubric tests; CAGR is what a holder compounded. The two agree in sign for all six funds. For USMV and SPLV the Holm p is for the daily tail-loss cut alone; the verdict also needs the drawdown and cost checks. Open Why? on any row for a plain-English explanation.
SPY’s figures differ slightly by row because each fund is compared on its own dates. Quotations are brief extracts cited from public documents, with no backtest figures reproduced; full quotations and sources are in the preregistration (§3, §10). Delivered: 0 of 6.
Secondary results
Declared in advance, corrected together (Holm across 8), and never able to change a verdict above. 0 of 8 significant.
- Better risk-adjusted return (all six). Every fund had a lower Sharpe ratio than SPY over its own dates: MTUM 0.84 vs 0.90, QUAL 0.84 vs 0.87, VLUE 0.78 vs 0.90, SIZE 0.73 vs 0.90, USMV 0.89 vs 0.93, SPLV 0.71 vs 0.86. Holm p 1.00 for each.
- Beat SPY (USMV and SPLV). USMV trailed by 3.7 points a year and SPLV by 4.4 (Holm p 1.00 for each).
Descriptive context
No inference claimed. On a common window (2013-07-19 to 2026-09-25) the ordering is the same: MTUM +2.0 points a year, QUAL −0.4, VLUE −0.6, SIZE −2.3, USMV −3.7, SPLV −5.5. Against the broad US market (French data) instead of SPY, the monthly results are close to those against SPY.
Every descriptive table, including calendar years, index-regime subperiods and CAPM, is in the per-fund reports and the study summary.
Before vs after launch
How large was each factor’s academic premium before the fund launched, and after? This is a description, not a test: no p-values, no grades, and the sealed results above are unchanged. The index providers’ own backtests are licensed and were not used. The “before” evidence is the long-short factor premium on Kenneth French’s public data, a proxy that differs from the funds in universe (all US stocks) and construction.
The academic premium was smaller after launch than before for all six mappings.
Before launch, four of the premia (momentum, profitability, value, size) had t-statistics of 2 or more over their full history. After launch, none did.
| Fund | Long-short factor (French) | Pre-launch, full history | Pre-launch, 1990 on | Post-launch | Fund minus SPY, post-launch |
|---|---|---|---|---|---|
| MTUM | Momentum (Mom) | +8.1 (t 4.5) | +6.9 (t 1.9) | +2.6 (t 0.7) | +1.2 (t 0.5) |
| QUAL | Profitability (RMW) | +3.2 (t 2.9) | +4.6 (t 2.3) | +2.4 (t 1.0) | −0.4 (t −0.5) |
| VLUE | Value (HML) | +5.0 (t 3.7) | +3.3 (t 1.5) | −0.6 (t −0.2) | −0.1 (t −0.0) |
| SIZE | Size (SMB) | +2.6 (t 2.1) | +1.8 (t 0.8) | −1.5 (t −0.6) | −2.2 (t −1.6) |
| USMV | Low minus high variance quintile | +3.0 (t 0.9) | +1.8 (t 0.3) | −3.0 (t −0.5) | −3.3 (t −1.8) |
| SPLV | Low minus high variance quintile | +2.5 (t 0.8) | +0.6 (t 0.1) | −2.0 (t −0.3) | −4.1 (t −1.7) |
Annualised means of monthly returns, points a year, with i.i.d. t-statistics. Pre-launch windows end the month before each fund’s inception; “1990 on” roughly matches the providers’ back-tested era. Post-launch runs to 2026-08, the last month of French data.
Read with care
- A proxy, not the product. The factors are long-short portfolios over all US stocks; the funds are long-only tilts on large and mid caps. A smaller factor premium after launch is not, by itself, evidence about any fund.
- Premium or capture? A simple long-only version of each factor (top-quintile or top-decile portfolios minus the market) did at least as well as the fund after launch for 5 of 5 funds with such a proxy. Because the universes differ, that gap is not evidence that a product failed to capture its factor.
- A familiar pattern. Rules and premia tend to look strongest on the history in which they were found. The six mappings here fit that pattern, but a pre-vs-post difference this noisy is also what chance can produce.
Full tables, windows and long-only proxies: BEFORE-AFTER.md. Source: Kenneth R. French Data Library; derived statistics only.
Read with care
Low power: “not shown to deliver” is not proof of absence. With 13 to 15 years of live data, 80% power needs an information ratio of about 0.63–0.69 at the last Holm step and 0.83–0.89 at the first. Long-only factor tilts typically have information ratios well below that. The intervals show what the data do rule out: live average premia above +5.5 points a year for MTUM, +0.9 for QUAL, +3.5 for VLUE and −0.2 for SIZE are hard to reconcile with the record; smaller ones cannot be ruled out.
Survivorship. All six funds still trade. We chose them as the flagship first-wave US single-factor ETFs, not on performance, but a study of surviving flagships can flatter the category.
Not fully blind. SPY prices had been seen in earlier audits. USMV and SPLV prices, and USMV’s earlier Batch 3 result (a Sharpe ratio of 0.88 vs 0.92, claim not supported), were seen before this preregistration was written. They are tested here on a different objective, and labelled non-blind throughout. MTUM, QUAL, VLUE, SIZE and the French factor data had not been loaded by any Holdout Labs study.
Price data and DEVIATION-001. Returns are from Yahoo Finance’s adjusted close, which is not an audited source and can be revised. The issuer cross-check, fixed in advance to run after the confirmatory test, found four distributions that the price data omit: QUAL (ex-dates 2014-12-24 and 2016-03-23), VLUE and SIZE (2016-03-23). The as-run results therefore understate those three funds slightly. Under a rule sealed before any corrected number was computed (DEVIATION-001), the confirmatory results stand as run, and a one-off sensitivity added the distributions back. No verdict or grade changed: QUAL −0.43 → −0.37 points a year, VLUE −0.48 → −0.43, SIZE −2.56 → −2.53; every Holm p stays 1.00. The largest calendar-year difference against the issuers’ own series, across all six, was 1.66 points (SIZE, 2016).
Not advice. These are statistical findings about past data. They are not investment advice and not a recommendation to buy, sell or hold any fund, and a result is not a forecast. A grade measures how fragile the evidence for a claim is; it is not a rating of a fund.
Prices from Yahoo Finance’s public chart endpoint (split- and dividend-adjusted close, fund expenses already deducted); factor data from the Kenneth R. French Data Library. We publish derived statistics only, never raw prices or factor series. No MSCI, S&P or AQR data were used (licence decision). Fund and index names are those of their owners; no affiliation or endorsement is implied.
Seal and registration
The proof that the test was fixed before any data were loaded: hashes, timestamp tokens and the order of events. Back to top
Preregistration. PREREG.md, SHA-256 42bfad1372704447674cde6efdf17064dd0b290e6be351659783c78d958b7c54, timestamped by DigiCert (RFC 3161) on 2026-09-26 at 16:00:46 UTC and FreeTSA at 16:00:47 UTC, before any return series was loaded. Tokens: DigiCert, FreeTSA, seal record.
OSF registration. osf.io/2tpn5, created 2026-09-26 at 17:04:20 UTC, public with no embargo (pending OSF’s standard admin approval); project osf.io/8wpez. The RFC 3161 seal is the binding pre-data timestamp; OSF is the public archive.
Analysis notes. ANALYSIS-NOTES.md records how open details of the text were implemented. It was written after the download and before any return statistic was computed, and sealed at 17:20:06 UTC. SHA-256 716137eb3acd56144f61d5981a330755eb9385149054c576aab2bd0e6dfe9930. Tokens: DigiCert, FreeTSA, seal record.
Deviation. DEVIATION-001 (distributions missing from the price data; details above), sealed at 17:25:44 UTC before its sensitivity analysis was run. SHA-256 7b2ee0538a1979343bcfb43cd21b20a34494ed487e1b80f39965f30abedc12bd. Tokens: DigiCert, FreeTSA, seal record.
To check a token: openssl ts -reply -in <token>.tsr -text shows the time and the message imprint, which is the file’s SHA-256. Companion files hashed inside the preregistration (§9): preregistration.json, the six objective declarations (MTUM, QUAL, VLUE, SIZE, USMV, SPLV) and the data-licence decision. Engine: holdout-audit 0.4.0, rubric v1.2 (sealed 2026-09-25), seed 20260925, 1,000 stationary-bootstrap draws.
Order of events, 2026-09-26 (UTC)
- 16:00:46
Preregistration sealed (DigiCert, FreeTSA). Only metadata had been fetched: fund names, inception dates, index names and changes, and the claim wording.
- 17:04:20
Registered on OSF (2tpn5).
- 17:11
Price and factor data downloaded, every file hashed.
- 17:20:06
Analysis notes sealed, before any return statistic was computed.
- 17:20:21
The single confirmatory run.
- After the run
Issuer cross-check (fixed in the preregistration to follow the run) found four distributions missing from the price data. DEVIATION-001 sealed at 17:25:44, then its one-off sensitivity analysis: no verdict or grade changed.
Have a backtest or a factor product? Get it audited
The same discipline, applied to your strategy or product claim: the objective fixed before the test, the test sealed before the data, and a written report with every check.
Send a high-level description. We’ll confirm scope, then send payment and secure-upload instructions.
Out of Sample · newsletter
Get new audits by email
Out of Sample, free, no spam. Every result published, including the failures.
You sign up on our newsletter page (hosted by Beehiiv); unsubscribe any time. Want a specific indicator tested? Request an audit.