# SMARTBETA-1: Smart beta, backtest vs live. Did the factors deliver after the ETFs launched?

**Status:** written 2026-09-26 (UTC) and sealed with RFC 3161 timestamps (`holdout-audit seal-doc`, DigiCert and FreeTSA) **before any return series was loaded by this study**: no ETF price, no index level and no factor return. Only metadata were fetched before the seal: fund names, inception dates, index names, index-change dates and the wording of the launch-era claims (sources in §10).

**Scope:** statistical analysis of past data only (`docs/SCOPE-POLICY.md`). Nothing here or in the results is investment advice or a recommendation to buy, sell or hold any fund. Asset guard (Celestial DEV-031, `tests/test_policy.py`): US equity ETFs only; no commodities, metals or commodity ETFs; no astronomical features.

**Publish all:** every ETF, every primary and secondary result and every descriptive table will be published at `samples/case-smartbeta/` and on holdoutlabs.io **whatever they show**, including if the factors did deliver. No ETF will be dropped, no test re-run with other settings, no objective switched after the data are seen.

## 0. What we have already seen (read this first)

This is a preregistered test, but it is **not fully blind**, and every published summary will say so.

- **SPY** (the benchmark) daily prices have been seen in earlier Holdout Labs audits.
- **USMV and SPLV** prices have been seen, and so have their Batch 3 results (`samples/batch3/`, sealed 2026-09-25): judged on objective (c) against SPY from 2011-11-01, USMV's Sharpe ratio was 0.88 vs SPY's 0.92, and the claim was not supported. The two low-volatility tests below are therefore **non-blind** and are labelled so in every table. They stay in the primary family (dropping them would itself be a result-dependent choice) and they use objective (b), which Batch 3 did not test.
- **MTUM, QUAL, VLUE and SIZE** prices have **not** been loaded by any Holdout Labs study. Kenneth French factor returns have not been loaded by any Holdout Labs study.
- **General knowledge.** The author has the general background knowledge any finance reader has of how factor investing is commonly said to have fared since 2013 (for example the widely discussed weakness of value in the late 2010s). No return statistic for the six ETFs over their live periods has been computed or looked up for this study. The design choices below follow fixed rules stated in advance (the flagship list; objectives from the launch-era claim text; the Batch 3 tolerance precedent), not expected results.

## 1. Question

Single-factor "smart beta" ETFs were launched in 2011–2013 on the strength of index histories that were largely back-tested, and on academic factor premia measured over decades before launch. **After launch, did the ETFs deliver what was claimed, measured on the total returns investors actually received, against a cap-weighted US equity benchmark?**

## 2. Universe (fixed; no additions or removals)

| ETF | Fund | Inception (issuer) | Underlying index at launch → changes | Factor |
|---|---|---|---|---|
| **MTUM** | iShares MSCI USA Momentum Factor ETF | 2013-04-16 | MSCI USA Momentum Index → MSCI USA Momentum SR Variant Index on or about 2020-12-01 (same parent; rebalances spread over several days) | Momentum |
| **QUAL** | iShares MSCI USA Quality Factor ETF | 2013-07-16 | MSCI USA Quality Index → MSCI USA Sector Neutral Quality Index on or about 2015-08-31 | Quality |
| **VLUE** | iShares MSCI USA Value Factor ETF | 2013-04-16 | MSCI USA Value Weighted Index → MSCI USA Enhanced Value Index on or about 2015-08-31 | Value |
| **SIZE** | iShares MSCI USA Size Factor ETF | 2013-04-16 | MSCI USA Risk Weighted Index → MSCI USA Low Size Index on or about 2018-11-30 | Size |
| **USMV** | iShares MSCI USA Min Vol Factor ETF | 2011-10-18 | MSCI USA Minimum Volatility Index (no change found) | Low volatility |
| **SPLV** | Invesco (PowerShares at launch) S&P 500 Low Volatility ETF | 2011-05-05 (operations commenced 2011-05-02) | S&P 500 Low Volatility Index (index inception 2011-04-20; earlier values back-tested) | Low volatility |

**Why these six.** They are the flagship US single-factor ETFs of the first smart-beta wave: iShares' five MSCI USA factor funds (momentum, quality, value, size, minimum volatility) and SPLV, the first large low-volatility ETF, launched on an S&P index whose pre-2011 history is explicitly back-tested. Each has a launch-era claim we can quote (§3) and at least 13 years of live history. All six were still trading on 2026-09-26 (issuer pages). Excluded: multi-factor funds (a different claim), funds launched after 2015 (short live histories), sector and dividend funds, and any commodity product (policy).

**Survivorship note.** All six survived to today. We did not select them on performance, but a study restricted to surviving flagship funds can be flattering; this is stated in the results.

**We test each ETF as sold**, including its index changes, because that is what investors held. Subperiods split at each index change are reported descriptively (§6.3).

## 3. The claims under test (quoted, with sources; no backtest figures reproduced)

- **MSCI factor indexes (MTUM, QUAL, VLUE, SIZE, USMV).** MSCI's launch-era research (Bender, Briand, Melas & Subramanian, *Foundations of Factor Investing*, MSCI Research Insight, December 2013): "Certain factors have historically earned a long-term risk premium"; "All the MSCI World Factor Indexes shown in Exhibit 11 outperform the MSCI World Index over the period shown, June 1988 to June 2013" (Exhibit 11 covers the Momentum, Quality, Value Weighted, Risk Weighted and Minimum Volatility indexes, among others); "The MSCI Momentum and MSCI Quality Indexes historically exhibited the strongest historical returns while the MSCI Minimum Volatility Indexes exhibited sizable risk reduction"; "Tilts towards these factors such as Value, Low Size, and Momentum historically produced excess long-term returns". iShares today, USMV: "The fund offers a way to reduce risk while maintaining equity exposure in a diversified portfolio."
- **S&P 500 Low Volatility (SPLV).** S&P Dow Jones Indices marketing text of December 2011, reproduced in an HSBC free-writing prospectus (SEC EDGAR accession 0001144204-12-005653, filed 2012-02-03): "a diversified equity index that, in back-tested results, has out-performed the S&P 500 over the near, medium, and long term, with less volatility", introduced with "how can I potentially reduce equity risk while preserving the long term return advantages that stocks generally provide?"

**Objective per ETF (rule: the objective follows the claim's own wording).**

| ETF | Primary objective (rubric v1.2) | Why |
|---|---|---|
| MTUM, QUAL, VLUE, SIZE | **(a) beat the benchmark** | The claim is a return premium ("outperform", "excess long-term returns"). |
| USMV, SPLV | **(b) reduce drawdown at acceptable cost**, declared return-cost tolerance **1.0% a year** | The claim is risk reduction while keeping equity-like returns. The 1.0% tolerance is the value Holdout Labs already declared for an "equity-like returns" claim in Batch 3 (Faber, sealed 2026-09-25), reused unchanged rather than chosen for this study. S&P's stronger "out-performed ... with less volatility" is tested too, as a secondary objective (a). |

Secondary objectives: **(c) improve Sharpe** for all six ("better risk-adjusted return historically", MSCI 2013), and **(a) beat the benchmark** for USMV and SPLV.

## 4. Benchmark

**Buy-and-hold SPY** total return (Yahoo adjusted close), on the same dates as each ETF. Reasons: each MSCI fund's parent is the cap-weighted MSCI USA Index (large and mid caps, roughly 85% of US market cap), and we found no ETF tracking that index over the whole 2011–2026 window; the S&P 500 is the standard cap-weighted proxy and is itself SPLV's parent; SPY is the most liquid cap-weighted US ETF with a full live history over the window; and it is the benchmark used in earlier Holdout Labs audits. iShares names the MSCI USA Index as USMV's secondary benchmark, which supports a cap-weighted US large/mid-cap comparison.

**Sensitivity (descriptive only):** the French market return (Mkt-RF + RF, monthly, all US stocks) as a broad-market benchmark for each ETF's monthly excess return.

## 5. Data (loaded only after this document is sealed)

- **ETF and benchmark returns.** Yahoo Finance public chart endpoint, daily, split- and dividend-adjusted close (total return, fund expenses already deducted). Tickers MTUM, QUAL, VLUE, SIZE, USMV, SPLV and SPY. Files get SHA-256 hashes in `MANIFEST.json` at download. SPY, USMV and SPLV may be re-downloaded fresh so that all seven series share one download date; if so, the new files are the ones used.
- **Yahoo caveats (stated in every report).** The adjusted close back-adjusts dividends multiplicatively and can be revised retroactively. Rounding in the adjusted series affects daily returns slightly. Yahoo is not an audited source.
- **Data-quality checks, fixed now.** (i) Drop any row dated before the fund's inception. (ii) Flag any daily |return| > 25%, and any gap of more than 5 consecutive NYSE trading days. (iii) After the run, cross-check each ETF's Yahoo-derived calendar-year total return against the issuer's published calendar-year total return (NAV or market). Only the maximum absolute difference is published (licensing). If a flagged problem cannot be resolved from the issuer's own data, the affected ETF is still reported, with the problem stated; nothing is dropped silently.
- **"Before" evidence.** Kenneth R. French Data Library, monthly: Fama/French 3 factors (Mkt-RF, SMB, HML, RF); 5 factors 2x3 (RMW); Momentum factor (Mom); univariate sorts: 10 portfolios formed on momentum (prior 12-2), portfolios formed on book-to-market, on operating profitability, and on variance. The versions are those current at download, with file hashes recorded. Derived statistics only (see `../LICENCE-DECISION.md`).
- **Not used:** any MSCI or S&P index series, AQR data, or issuer backtest figures (licensing).
- **End date:** returns run through the close of **2026-09-25**. French data run through the last month available at download, stated in the results.

## 6. Analysis plan

### 6.1 Primary tests (one per ETF, confirmatory)

For each ETF, `holdout-audit` 0.4.0 under **rubric v1.2 exactly as sealed** (`docs/RUBRIC-v1.2.md`, SHA-256 `81bc60b5bf4520fdfb91c43b37883c30a9a2061903442d1e8a8b05f3d37a71fc`, sealed 2026-09-25T20:33:45Z):

- **Strategy:** buy-and-hold the ETF. Daily total returns from the first Yahoo close on or after the inception date to 2026-09-25. The first return is day 2 over day 1.
- **Benchmark:** buy-and-hold SPY on the same dates (inner join on dates).
- **Costs:** 5 bps one-way on the day-1 purchase only (turnover 1.0 on day 1, 0 afterwards), as in Batch 3's low-volatility ETF audit.
- **Declared number of variants N = 1** per ETF (each fund is one product, run once), passed as `n_trials=1`. Multiplicity across the six funds is handled by Holm (§7).
- **Objective declaration:** `declaration-<ticker>.json` (hashes in §9). Primary objective as in §3; for (b) `max_return_shortfall_annual = 0.01`.
- **Holdout split:** the rubric default, the start of the last 30% of each ETF's sample.
- **Settings:** 1,000 stationary-bootstrap draws, mean block length max(5, T^⅓), seed 20260925, 252 periods a year. CSCV/PBO and the parameter-plateau check are N/A (no variant matrix).
- **Primary statistic:** the one-sided p-value of the rubric's multiple-testing ("haircut") check with N = 1. Under (a) that is P(Z ≥ annualised excess Sharpe × √years); under (b) it is the paired-bootstrap z of dCVaR.

### 6.2 Metrics reported for every ETF (all six, whatever the objective)

These metrics use the full live sample and the same dates for the ETF and SPY. Each comes with a paired stationary-bootstrap 90% interval (same settings as §6.1).

1. **Average excess return**, arithmetic, annualised: (mean(r) − mean(b)) × 252.
2. **Compound excess:** CAGR(ETF) − CAGR(SPY), and terminal wealth of $1 in each. If the arithmetic and compound measures differ in sign, this is said explicitly (`docs/NOTE-arithmetic-vs-compound-excess.md`).
3. **Sharpe difference** dSR = SR(ETF) − SR(SPY), annualised, computed as rubric v1.2 computes it. A version net of the French RF (monthly) is reported descriptively.
4. **Volatility difference:** annualised σ(ETF) − σ(SPY), and the ratio.
5. **Maximum drawdown difference:** dMDD = MDD(SPY) − MDD(ETF); also dCVaR95.
6. **Beta to SPY and CAPM alpha** (OLS on daily returns, Newey-West standard errors, 5 lags), descriptive. A low-volatility fund can have a positive alpha but a negative raw excess; both are reported.
7. **Rubric v1.2 letter grade** on the primary objective, with every check.

### 6.3 Descriptive analyses (labelled DESCRIPTIVE; no inference claimed)

- **Common window:** all six from 2013-07-17 (the day after QUAL's inception) to 2026-09-25, so the funds can be compared on the same dates.
- **Index-regime subperiods:** each ETF split at its index-change date(s) (§2).
- **Calendar-year table:** ETF minus SPY total return by year.
- **Broad-market sensitivity:** monthly ETF excess over the French market return.

### 6.4 Before vs after (DESCRIPTIVE ONLY)

We cannot use the MSCI or S&P backtests (licensing), so the "before" evidence is the academic factor premium on French data. The mapping is fixed now:

| ETF | Long-short factor | Long-only proxy (minus market = Mkt-RF + RF) |
|---|---|---|
| MTUM | Mom | Top decile, value-weighted, 10 portfolios on prior 12-2 return |
| QUAL | RMW (from 1963-07) | Highest quintile, value-weighted, portfolios on operating profitability |
| VLUE | HML | Highest quintile, value-weighted, portfolios on book-to-market |
| SIZE | SMB | none (the MSCI Low Size index reweights large and mid caps, and French's small-cap quintiles are mostly micro-caps, so no honest long-only match exists) |
| USMV, SPLV | Lowest minus highest variance quintile, value-weighted | Lowest-variance quintile, value-weighted |

Windows, monthly:

- **Pre-launch, full:** from the first month available to the month before the ETF's inception month.
- **Pre-launch, index-backtest era:** 1990-01 to the month before the inception month. This roughly matches the providers' back-tested histories (MSCI 2013 used June 1988 to June 2013).
- **Post-launch:** the first full month after inception to the last month available.

Statistics per window:

- annualised arithmetic mean and its i.i.d. t-statistic;
- annualised volatility;
- for the long-only proxies, compound excess over the market;
- for the low-volatility proxies, the volatility ratio to the market and the maximum-drawdown difference.

Side by side, the table shows the ETF's own post-launch monthly excess over SPY, for the same months. A Welch t for the pre vs post difference in the long-short factor mean is shown as descriptive context, not as a test.

This comparison separates two readings: "the factor premium itself shrank after launch" (French post < pre) and "the product did not capture the factor" (French post ≥ ETF post). The mapping is a proxy, it differs in universe and construction, and it is never described as evidence.

## 7. Inference criteria

- **Primary family:** the 6 primary tests. **Holm** step-down correction across the 6 at family-wise α = 0.05.
- **"Delivered on its claim"** is used for an ETF only if both hold:
  - its Holm-adjusted p ≤ 0.05;
  - its primary objective check is not FAIL. Under (a): the SPA check is not FAIL and the annualised mean excess is > 0 (the rubric's objective cap is not triggered). Under (b): the drawdown check and the tolerance check are both not FAIL.
- **Otherwise the wording is fixed:**
  - "**not shown to deliver**" (never "failed" or "did not work") when the point estimate is in the claimed direction but does not pass;
  - "**went the other way**" when the point estimate is opposite to the claim.

  USMV and SPLV always carry "(non-blind)".
- **Secondary family:** 8 tests, namely (c) improve Sharpe for all six, and (a) beat the benchmark for USMV and SPLV. Holm across the 8. These are reported in full but never change the "delivered" wording.
- **Power (stated now).** Each fund has about 13 to 15 years of live data. Under (a), 80% power at one-sided α needs an annualised excess Sharpe (information ratio) of about (z_α + 0.84) / √13.4. That is ≈ 0.68 at α = 0.05 (the last Holm step) and ≈ 0.88 at α = 0.0083 (the first Holm step). Published long-only factor tilts typically have information ratios well below that, so **a "not shown to deliver" result is weak evidence that the premium was absent**. The confidence intervals in §6.2 will be shown so that readers can see what the data rule out.
- The rubric letter grade is reported for every ETF regardless.

## 8. Exclusions, missing data and deviations

- **No exclusions.** All six ETFs are analysed and published. If a fund were delisted before 2026-09-25 (none is), its window would end at its last trading day.
- **Missing days.** Inner join of ETF and SPY dates. No forward-filling of returns. Gaps over 5 trading days are flagged and reported (§5).
- **Source failure.** If the Yahoo endpoint is unavailable or fails the checks in §5, the fallback is the issuer's own downloadable daily NAV history, used under the same "derived statistics only" rule, and recorded as a deviation.
- **French library.** If a listed dataset is renamed, the closest-named successor is used and recorded as a deviation. If one is withdrawn, that row of §6.4 is reported as missing.
- **Any other deviation** from this document is written, dated and listed on the results page and in the affected reports before those results are published.

## 9. Companion files (hashed here; sealed with this document)

| File | SHA-256 |
|---|---|
| `prereg/preregistration.json` (machine-readable companion, written 2026-09-26T15:59:15Z) | `b7287d367edafc5854ff3b4167aeacdf483bf972e3e530a1eecb645d59c8a6fa` |
| `prereg/declaration-mtum.json` | `572c1ddd0f7e1eb95e060f8a3c0f7b0668e5cf766a5b5e4fdaa2d4307a59ee71` |
| `prereg/declaration-qual.json` | `68d11b3f242aff87a999fe7ce9dc30339e985ba1f0bf962cb8da7f30cc5a0f0a` |
| `prereg/declaration-vlue.json` | `41e684ecf975dd040e60ebd05d1f0a38a1c913965aac737be5fc3052d939c6a1` |
| `prereg/declaration-size.json` | `995bdae7321af44821b09a472cbe36c9fc35c624da40f472bac143fcc7463cd8` |
| `prereg/declaration-usmv.json` | `34236ca9ddfba9402f30ce7f0f35bc3e5686abf093ae57a07b175caf12078cd2` |
| `prereg/declaration-splv.json` | `9269fabe4c034d9446bc22e95e46a44226b3a8635c1371e38d168ba85079dd26` |
| `LICENCE-DECISION.md` | `d75bb6b107c249a4293ce66485522af08e29ac8236a1e09831614b7dcc0a5ac1` |
| `docs/RUBRIC-v1.2.md` (already sealed) | `81bc60b5bf4520fdfb91c43b37883c30a9a2061903442d1e8a8b05f3d37a71fc` |

If this document and a companion file disagree, **this document governs**.

## 10. Sources for the metadata (fetched 2026-09-26, before the seal)

- iShares product pages (fund name, inception date, current index, objective text): MTUM `ishares.com/us/products/251614`, QUAL `/256101`, USMV `/239695`, VLUE `/251616`, SIZE `/251465`. Only metadata were read; our fetcher was told not to return any performance figure.
- Index-change dates (SEC EDGAR, iShares Trust CIK 1100663):
  - MTUM: prospectus supplement of 2020-09-18 (0001193125-20-249191);
  - QUAL and VLUE: prospectus supplement of 2015-06-26 (0001193125-15-237101);
  - SIZE: prospectus supplement of 2018-10-01 (0001193125-18-289698);
  - SIZE's launch index: 485BPOS of 2013-04-12 (0001193125-13-153074).
- SPLV inception: PowerShares Exchange-Traded Fund Trust II annual report N-CSR filed 2012-01-09 (0001104659-12-001190): "May 5, 2011 (Fund Inception)"; operations from 2011-05-02. S&P 500 Low Volatility index inception date (2011-04-20) and "back-tested" wording: HSBC FWP 0001144204-12-005653.
- MSCI (2013), *Foundations of Factor Investing*, `msci.com/documents/1296102/1336482/Foundations_of_Factor_Investing.pdf`: claim sentences only. We noticed one back-test figure in passing (an MSCI World index, not US, pre-2013); it is not used.
- Not verified: the launch dates of the MSCI factor indexes themselves (MSCI's index pages did not load for our fetcher). They do not enter the design, because the before/after boundary is the ETF inception date.