Batch 1: MACD (TradingView built-in strategy) on SPY
TradingView's built-in 'MACD Strategy' with its default settings, long and short as it ships, on SPY daily bars. Benchmark: buy-and-hold SPY. Part of Indicator Audit Batch 1 (20 audits, one sealed preregistration, Holm across the batch).
Executive summary
Very fragile. The backtest shows no robust edge over its benchmark once selection, holdout and stability are accounted for. The grade measures how fragile the historical evidence is. It is not a forecast and not a recommendation.
Declared objective: beat the benchmark, declared by the client on 2026-09-25T18:21:27Z (declaration SHA-256 35b4d77e284614c1…).
Benchmark: buy-and-hold of the traded asset. This is the rubric v1.1 default. The strategy did not beat its benchmark under the rubric definition, so the grade is at most C.
Checks that failed: Selection-adjusted significance vs benchmark (Deflated Sharpe); Multiple-testing haircut vs benchmark (Harvey-Liu); Beats the benchmark after data snooping (Hansen SPA); Holdout degradation (excess Sharpe); Period and regime stability (vs benchmark); Transaction-cost headroom (absolute); Parameter plateau (absolute).
Scorecard
| Check | Result | Value | Rule |
|---|---|---|---|
| Selection-adjusted significance vs benchmark (Deflated Sharpe) | FAIL | DSR 0.000 (N = 48) | PASS DSR >= 0.95; CAUTION >= 0.80; else FAIL (on excess returns) |
| Probability of backtest overfitting (CSCV) | CAUTION | PBO 0.320 (12870 splits) | PASS PBO <= 0.20; CAUTION <= 0.50, or PBO > 0.50 with P(OOS excess SR < 0) <= 0.10; else FAIL |
| Multiple-testing haircut vs benchmark (Harvey-Liu) | FAIL | adj. p 1.0000 (BHY) | PASS one-sided adjusted p <= 0.05; CAUTION <= 0.10; else FAIL (excess-return t-statistic; evidence for the strategy only) |
| Beats the benchmark after data snooping (Hansen SPA) | FAIL | SPA p 1.000; RC p 1.000 | PASS p <= 0.05; CAUTION <= 0.10; else FAIL |
| Holdout degradation (excess Sharpe) | FAIL | excess SR -0.45 -> -0.55 | PASS holdout and in-sample excess SR > 0 and holdout >= 50% of in-sample; CAUTION holdout excess SR > 0; else FAIL |
| Period and regime stability (vs benchmark) | FAIL | 18% of years ahead | PASS >= 60% of years ahead of benchmark and excess SR > 0 in every volatility tercile; CAUTION >= 50%; else FAIL |
| Transaction-cost headroom (absolute) | FAIL | break-even -3.6 bps | PASS break-even cost >= 20 bps per unit traded; CAUTION >= 5 bps; else FAIL (absolute) |
| Parameter plateau (absolute) | FAIL | neighbours n/a% of chosen | PASS neighbour SR >= 70% of chosen SR; CAUTION >= 40%; else FAIL, or FAIL if chosen SR <= 0 (absolute) |
How the grade is set (Holdout Labs Fragility Rubric v1.1, sealed before this report was produced). PASS = 2 points, CAUTION = 1, FAIL = 0; N/A checks are excluded. Share of available points: A ≥ 85%, B ≥ 70%, C ≥ 55%, D ≥ 40%, otherwise F. Hard caps: Objective cap (beat the benchmark): if the strategy does not beat its benchmark (SPA check FAIL, or annualised mean excess return <= 0), the grade is at most C. Objective cap (reduce drawdown): if the drawdown reduction is not robust (drawdown check FAIL) or the declared return-cost tolerance is breached (tolerance check FAIL), the grade is at most C. Selection cap: if the deflated check FAILS, the grade is at most C. Overfitting cap: if the PBO check FAILS, the grade is at most D. Disclosure cap: if the number of variants tried was not declared and no variant matrix was supplied, the grade is at most B.
What was audited
| Sample | 1994-01-03 to 2026-09-24 (8237 periods, 252 per year) |
| Variants tried (N) | 48 · variant matrix of 48 columns supplied |
| Base-case cost | 2 bps per unit traded (one way) |
| Holdout split | 2017-01-03 |
| Compound annual return, strategy (net) | -4.1% |
| Buy-and-hold of the underlying, same period | compound annual return 10.9%; annualised Sharpe 0.65; max drawdown 55.2% |
Results in detail
1. Sharpe ratio and its uncertainty
Rubric checks use the excess series (strategy minus buy-and-hold of the traded asset); the absolute series is shown for reference.
| Excess over benchmark | Absolute | |
|---|---|---|
| Annualised Sharpe (sqrt-time scaling) | -0.481 | -0.130 |
| Annualised Sharpe (Lo 2002 autocorrelation-adjusted) | -0.666 | -0.173 |
| Standard error, annualised (non-normal) | 0.175 | 0.175 |
| Standard error per period: IID-normal / non-normal | 0.0110 / 0.0110 | 0.0110 / 0.0110 |
| Skewness / kurtosis | -0.13 / 29.00 | -0.10 / 14.59 |
| Probabilistic Sharpe ratio vs 0 | 0.0030 | 0.2279 |
| Minimum track record for 95% confidence | ∞ years | ∞ years |
| Annualised mean / volatility | -14.5% / 30.2% | -2.4% / 18.7% |
2. The variant-count effect (Deflated Sharpe)
| N tried | Noise hurdle (annual excess SR) | DSR |
|---|---|---|
| 1 | 0.000 | 0.0030 |
| 2 | 0.030 | 0.0018 |
| 5 | 0.069 | 0.0009 |
| 10 | 0.091 | 0.0006 |
| 20 | 0.109 | 0.0004 |
| 48 | 0.130 | 0.0002 |
| 50 | 0.131 | 0.0002 |
| 100 | 0.145 | 0.0002 |
| 200 | 0.159 | 0.0001 |
| 500 | 0.175 | <0.0001 |
| 1000 | 0.187 | <0.0001 |
Dispersion of trial Sharpe ratios used: 0.057 (annualised standard deviation), estimated from the variant matrix.
3. Probability of backtest overfitting (CSCV)
| PBO | 0.320 |
| Splits (blocks) | 12870 (16) |
| Variants | 48 |
| P(IS winner trails benchmark out of sample) | 1.000 |
| Median OOS excess Sharpe of IS winner (annual) | -0.55 |
| Degradation slope (OOS on IS) | -0.88 |
Histogram of logit ranks: values left of the dashed line are splits where the in-sample winner finished at or below the out-of-sample median (logit 0).
4. Multiple-testing haircut
| t-statistic (excess SR x sqrt(years)) | -2.75 |
| p-value, single test | 0.99701 |
| Bonferroni p (48 tests) | 1.00000 |
| Sidak p | 1.00000 |
| Holm p | 1.00000 |
| BHY p | 1.00000 |
| Haircut excess Sharpe (Bonferroni) | -0.000 (100% haircut) |
| Haircut excess Sharpe (BHY) | -0.000 |
5. Reality Check and SPA
| Benchmark | buy-and-hold of the traded asset |
| Variants in the test | 48 |
| Best mean excess return (annualised) | -7.27% |
| White Reality Check p | 1.000 |
| Hansen SPA p (consistent / lower / upper) | 1.000 / 1.000 / 1.000 |
| Bootstrap | stationary, 1000 draws, mean block 20 |
6. Holdout degradation
| Periods | Annualised excess Sharpe | |
|---|---|---|
| In-sample (before 2017-01-03) | 5792 | -0.454 |
| Holdout | 2445 | -0.545 |
| Holdout / in-sample | n/a | |
| Consistency p-value | 0.778 |
7. Period and regime stability
| Volatility regime (trailing 21-day, lagged) | Periods | Annualised mean excess | Annualised excess Sharpe |
|---|---|---|---|
| low vol | 2739 | -15.5% | -0.91 |
| mid vol | 2738 | -6.4% | -0.26 |
| high vol | 2739 | -21.4% | -0.50 |
8. Transaction-cost sensitivity
| One-way cost (bps per unit traded) | 0 | 1 | 2 | 5 | 10 | 20 | 50 |
|---|---|---|---|---|---|---|---|
| Annualised Sharpe | -0.08 | -0.11 | -0.13 | -0.20 | -0.32 | -0.55 | -1.22 |
| Annualised mean return | -1.6% | -2.0% | -2.4% | -3.8% | -5.9% | -10.3% | -23.4% |
Turnover 43.6 units per year; break-even cost -3.6 bps.
9. Parameter sensitivity
Chosen variant fast=12|slow=26|signal=9|mode=long_short ranks 25 of 48. Grid median Sharpe 0.01, best 0.44; 50% of cells positive; neighbour ratio n/a.
10. Drawdown distribution
What is fragile, and how to test the fix
FAIL Selection-adjusted significance vs benchmark (Deflated Sharpe)
Excess Sharpe over buy-and-hold of the traded asset (total return): -0.48 a year over 32.7 years. After allowing for 48 variant(s) tried, the best-of-N noise hurdle is an annualised excess Sharpe of 0.13; the probability that the true excess Sharpe exceeds it is 0.000.
How to test a fix: Declare every variant you tried (including abandoned ones), then re-test on data you have not looked at: a sealed forward period or a later holdout. A DSR that only passes at N = 1 is not evidence.
CAUTION Probability of backtest overfitting (CSCV)
Across 12870 combinatorial in-sample/out-of-sample splits of 48 variants, ranked by excess Sharpe, the in-sample winner ranked in the bottom half out of sample 32.0% of the time and trailed the benchmark out of sample 100.0% of the time.
How to test a fix: Shrink the search: fewer free parameters, coarser grids, rules fixed from theory before looking at results. Then re-run CSCV on the smaller family; PBO should fall.
FAIL Multiple-testing haircut vs benchmark (Harvey-Liu)
Excess-return t-statistic -2.75 (single-test p 0.9970). Adjusted for 48 tests (BHY), p = 1.0000; the Bonferroni-haircut excess Sharpe is -0.00.
How to test a fix: A longer sample raises the t-statistic without new selection: extend the test to earlier or later data the rule was not designed on, keeping the rule frozen.
FAIL Beats the benchmark after data snooping (Hansen SPA)
Benchmark: buy-and-hold of the traded asset (total return). Testing whether the best of 48 variant(s) beats it in mean return, with a stationary bootstrap (1000 draws, mean block 20 periods): Hansen SPA p = 1.000, White Reality Check p = 1.000.
How to test a fix: If the rule's value is lower risk rather than higher return, state that claim before testing it (e.g. drawdown or volatility against the benchmark) and test it on unseen data.
FAIL Holdout degradation (excess Sharpe)
In-sample (before 2017-01-03) annualised excess Sharpe -0.45; holdout -0.55 over 2445 periods. Consistency p-value 0.778 (low values mean the holdout is unlikely to share the in-sample excess Sharpe).
How to test a fix: Lock a fresh holdout before any further tuning, or run a sealed forward trial with preregistered pass criteria; only data you have not seen can confirm a fix.
FAIL Period and regime stability (vs benchmark)
The strategy's compounded return beat the benchmark's in 6 of 33 calendar years. Annualised excess Sharpe by volatility tercile: low vol -0.91, mid vol -0.26, high vol -0.50.
How to test a fix: Check whether the edge over the benchmark is concentrated in a few years or one regime; if so, state that regime as part of the hypothesis and test it on a period the rule has not seen.
FAIL Transaction-cost headroom (absolute)
Turnover is 43.6 units a year. The gross mean return falls to zero at a one-way cost of -3.6 bps per unit traded. Base case in this report: 2 bps.
How to test a fix: Measure your real fills (spread, slippage, fees) over a few weeks and re-run with that cost; reduce turnover with wider entry/exit bands and check the Sharpe survives.
FAIL Parameter plateau (absolute)
The chosen variant ranks 25 of 48 (annualised Sharpe -0.13; grid median 0.01; best 0.44). Its one-step grid neighbours average n/a% of its Sharpe; 50% of the grid has a positive Sharpe.
How to test a fix: Prefer the centre of a broad plateau over the single best cell, and report the grid median as the honest expectation.
These are suggestions for further statistical testing, not suggestions to trade. Any change to the rules creates a new variant: count it in N and confirm it on data it was not designed on.
Methods appendix
Sharpe ratio and its standard error
Per-period mean over standard deviation of the audited return series (risk-free rate taken as zero), annualised by the square root of periods per year. The standard error uses the IID-normal formula of Lo (2002) and the non-normal correction of Mertens (2002), which widens the error for negative skew and fat tails. Lo's autocorrelation-adjusted annualisation is reported alongside. [1], [2]
Probabilistic Sharpe ratio (PSR) and minimum track record
The probability that the true Sharpe ratio exceeds a benchmark (here zero), given the sample length, skewness and kurtosis; and the minimum sample length for 95% confidence. [3]
Deflated Sharpe ratio (DSR)
The PSR measured against the Sharpe ratio one would expect from the best of N skill-less trials, where N is the number of variants tried and the dispersion of trial Sharpe ratios is estimated from the variant matrix (or, without one, set to the null sampling variance 1/T). It corrects for selection and non-normality at once. [4]
Probability of backtest overfitting (PBO) via CSCV
The variant matrix is cut into 16 time blocks; for each of the 12,870 ways of picking half of them as in-sample, the in-sample best variant is ranked out of sample. PBO is the share of splits where it falls to or below the median. It assumes blocks long enough to preserve serial dependence and a variant set that represents the real search. [5]
Multiple-testing haircut
The Sharpe ratio is turned into a t-statistic and p-value, the p-value is adjusted for the number of tests (Bonferroni always; Holm and BHY when every variant's returns are supplied), and the adjusted p-value is mapped back to a haircut Sharpe ratio. [6], [7]
White's Reality Check and Hansen's SPA test
Tests whether the best of all variants beats the benchmark in mean return once the search over variants is accounted for. Uses the stationary bootstrap with mean block length T^(1/3) (at least 5) to keep short-range dependence. SPA studentises and recentres, so poor variants do not dilute the power. [8], [9], [10]
Holdout degradation
In-sample versus holdout Sharpe ratio at a fixed split date (by default the last 30% of the sample). The consistency p-value asks how surprising the holdout Sharpe would be if the in-sample Sharpe were the truth. A holdout only counts if it was not used to design the rule. [11]
Period and regime stability
Returns and Sharpe ratio by calendar year and by tercile of the underlying market's trailing 21-day volatility (lagged one day). Descriptive: the tercile cut points use the full sample. [12]
Parameter-sensitivity surface
Sharpe ratio across the supplied parameter grid. The neighbour ratio compares the chosen cell with its one-step neighbours: a plateau (ratio near 1) is less fragile than an isolated peak. [13]
Transaction-cost sensitivity
Net return = gross return minus turnover times a one-way cost per unit traded, over a grid of costs; the break-even cost is where the mean net return reaches zero. Market impact beyond a flat cost is not modelled. [12]
Drawdown distribution by bootstrap
The maximum drawdown is recomputed on 1,000 stationary-bootstrap resamples of the return series, showing how much deeper (or shallower) the worst loss could plausibly have been with the same return distribution in a different order. [10]
References
- Lo, A. W. (2002). The Statistics of Sharpe Ratios. Financial Analysts Journal 58(4), 36-52.
- Mertens, E. (2002). Comments on Variance of the IID Estimator in Lo (2002). Working paper, University of Basel.
- Bailey, D. H. & Lopez de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk 15(2), 3-44.
- Bailey, D. H. & Lopez de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management 40(5), 94-107.
- Bailey, D. H., Borwein, J. M., Lopez de Prado, M. & Zhu, Q. J. (2016). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4), 39-69.
- Harvey, C. R. & Liu, Y. (2015). Backtesting. Journal of Portfolio Management 42(1), 13-28.
- Harvey, C. R., Liu, Y. & Zhu, H. (2016). ... and the Cross-Section of Expected Returns. Review of Financial Studies 29(1), 5-68.
- White, H. (2000). A Reality Check for Data Snooping. Econometrica 68(5), 1097-1126.
- Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics 23(4), 365-380.
- Politis, D. N. & Romano, J. P. (1994). The Stationary Bootstrap. Journal of the American Statistical Association 89(428), 1303-1313.
- Bailey, D. H., Borwein, J. M., Lopez de Prado, M. & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the AMS 61(5), 458-471.
- Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.
- Pardo, R. (2008). The Evaluation and Optimization of Trading Strategies, 2nd ed. Wiley.
Data and hash appendix
- Prices from Yahoo Finance's public chart endpoint; we publish derived statistics only, never raw prices; no redistribution. SPY daily OHLC (unadjusted) and dividends; total return computed locally.
- Normalised file spy_ohlc_daily_yahoo.csv: 8471 rows ending 2026-09-24, SHA-256 f05d93e20e2ad5f36cf078de7ecabcaa613a3cb8f6a9ddbb2d19f0db44ed3867.
- Audit period 1994-01-03 to 2026-09-24; holdout from 2017-01-03 (fixed in the sealed batch preregistration).
- Execution convention: signal at the close, held close to close; stop entries treated as filled at the signal close; short-side funding not modelled (differs from TradingView's broker emulator).
| Item | SHA-256 / value |
|---|---|
| package | holdout-audit 0.3.0 |
| spy_ohlc_daily_yahoo.csv | f05d93e20e2ad5f36cf078de7ecabcaa613a3cb8f6a9ddbb2d19f0db44ed3867 |
| audited_series_sha256 | f193eb82731fd01ffa4daad0de79ef3db64d074fe5836efbf51993b82302bc92 |
| objective_declaration_sha256 | 35b4d77e284614c15fafdacaf91e122b600e520c1ea61e33627a64c3497c765d |
| preregistration_sha256 | 35b4d77e284614c15fafdacaf91e122b600e520c1ea61e33627a64c3497c765d |
| config_sha256 | 831792d60c51520d2e2b3e28d92aef6cbdb34510ed73df1ad3514b5f49db792e |
| bootstrap seed | 20260925 |
| rubric version | 1.1 (docs/RUBRIC-v1.1.md) |
| declared objective | beat_benchmark |
| preregistration seal | 9f25abc24d54753071fc54da00bda52705ea2e7c0b6d2ba3011d7c009fa22d5c (digicert, freetsa; earliest 2026-09-25T18:21:44Z) |
| objective declaration seal | 9f25abc24d54753071fc54da00bda52705ea2e7c0b6d2ba3011d7c009fa22d5c (digicert, freetsa; earliest 2026-09-25T18:21:44Z) |
| rubric document SHA-256 | 2311b126336bab9f426a84f9c9cd9cb937e346b403a62f60e3c96ed8b5868e46 |
| rubric seal record SHA-256 | ac882c422df5154ec7e6e928904c9099d6b286dc9e5369fece4b85df57044138 |
| rubric sealed by digicert (RFC 3161) | 2026-09-25T14:20:10Z |
| rubric sealed by freetsa (RFC 3161) | 2026-09-25T14:20:11Z |
Anyone with the same inputs and package version can re-run the audit and must obtain the same audited-series hash and the same statistics.