# BACKTEST - Plain poll average vs bias-correcting "Tensor" rules, leave-one-cycle-out

Reproduce: `python -I backtest.py && python -I bootstrap.py && python -I late_shift.py` in this folder (needs numpy, pandas, scipy, openpyxl). Raw outputs: `out/results.txt`, `out/bootstrap.txt`, `out/late_shift.txt`, per-race CSVs `out/races_minpolls{1,2,3}.csv`, per-cycle LOCO `out/loco_by_cycle_minpolls*.csv`. Data: `data/sb_rawpolls_021325.xlsx` (Silver Bulletin, see DATA.md); SHA-256 in `data/SHA256SUMS.txt`.

## Design

- Unit: a polled general-election race (Pres by state, Sen, Gov, House district) in an even-year cycle. Poll average = mean over distinct non-partisan pollsters of the pollster's mean DEM-REP margin fielded 0-21 days before the election; at least 2 pollsters (sensitivity: 1 and 3).
- Target: certified margin as recorded in the file (not independently audited).
- Held-out: each cycle 2002-2024 in turn (12 cycles; 2000 and 1998 lack a prior presidential margin for the lean rule, so only M0/M1 numbers on those are in the cycle tables). Each model is fit on all _other_ cycles, **including future ones**, which is the standard leave-one-cycle-out; it is more generous than a strict forward test and I flag this.
- Rules compared (all symmetric by construction; none hard-codes a party):
  - M0 plain average.
  - M1 subtract mean historical bias (global shift).
  - M2 M1 plus a term for which party holds the White House (tests "polls lean against or for the incumbent party").
  - M3 shrink the poll average toward the state's last presidential margin (one fitted weight).
  - M4 M3 plus intercept plus White House term.
  - M5 persistence: subtract half of the previous cycle's mean bias.
- Metrics: MAE, mean signed error (bias), RMSE, calibration of normal intervals using each fold's training residual SD (nominal 80% and 90%), wrong-call share (sign of predicted margin vs result), width. Paired cluster bootstrap by cycle for MAE differences.

## Results (min 2 pollsters, 946 races, 12 held-out cycles)

| Model                      | MAE      | Mean error | RMSE     | Cov 80% | Cov 90% | Wrong-call | 90% width |
| -------------------------- | -------- | ---------- | -------- | ------- | ------- | ---------- | --------- |
| M0 plain average           | 4.18     | +1.19      | 5.39     | 0.796   | 0.886   | 11.2%      | 17.3      |
| M1 global shift            | 4.18     | -0.01      | 5.35     | 0.802   | 0.890   | 11.7%      | 17.3      |
| M2 shift + WH party        | 4.30     | -0.03      | 5.47     | 0.785   | 0.881   | 12.7%      | 17.3      |
| M3 shrink to state lean    | **4.07** | +1.06      | **5.27** | 0.801   | 0.884   | **11.0%**  | 16.9      |
| M4 shrink + intercept + WH | 4.18     | -0.06      | 5.35     | 0.793   | 0.879   | 12.3%      | 16.8      |
| M5 persistence             | 4.24     | +0.57      | 5.45     | 0.802   | 0.881   | 12.4%      | 17.3      |

Naive sampling-error-only 90% interval (what a poll's stated margin of error implies): covers **42%** of outcomes (80% interval: 34%). Calibrated intervals using total error cover 88-89% (nominal 90%) and 80% (nominal 80%).

Paired cluster bootstrap of MAE difference vs plain average (negative = better), 12 cycles:
| Model | MAE diff | 95% CI | Cycles better |
|---|---|---|---|
| M1 global shift | +0.006 | [-0.26, +0.28] | 6 / 12 |
| M2 shift + WH party | +0.123 | [-0.16, +0.42] | 6 / 12 |
| M3 shrink to lean | **-0.112** | [-0.25, +0.03] | 10 / 12 |
| M4 | +0.001 | [-0.34, +0.36] | 8 / 12 |
| M5 persistence | +0.092 | [-0.16, +0.33] | 4 / 11 |

Sensitivity: with min 1 pollster (1,536 races) M3 MAE 4.86 vs M0 4.93; with min 3 (660 races) 3.81 vs 3.87. Same ordering: M3 slightly better, nothing else better.

## What the results say (including the negatives)

1. **No bias correction beats the plain average.** The constant "correct for the usual miss" shift (M1) gives identical MAE (diff +0.006, CI straddles 0) and wins in only 6 of 12 held-out cycles. It removes the average bias in pooled terms (+1.19 -> -0.01) but at the cost of making cycles like 2006, 2008, 2010, 2012, 2018, 2022 worse. This is the key negative result: **a correction that would have helped in 2014-2024 hurts in other years.**
2. The White House-party terms (M2, M4) and persistence (M5) do not help; M2 and M5 are worse on average.
3. The **only improvement** is shrinking toward last presidential margin (M3): -0.11 pts MAE (-2.7%), 10 of 12 cycles, but the bootstrap interval includes 0. Treat as weak evidence of a small gain. It does not correct party bias (mean error stays +1.06).
4. **Calibration: the real gain.** Stated poll margins of error (sampling only) cover 42% of outcomes at nominal 90%. Total-error intervals with about +/-8.5 pts half-width at 90% are well calibrated out-of-sample. The "tensor" benefit is honest uncertainty, not a better point estimate.
5. Correlated misses: in competitive races (|avg| <= 5) the number of wrong calls per cycle was consistent with independence in most cycles; the exceptions were mild (2016: 13 observed vs 9.3 expected, P=0.095 under independence; 2018: 9 vs 13.1, P(<=)=0.098). So **even though mean bias is cycle-level, the wrong-call count test lacks power to confirm or refute correlation** with 14 cycles and 18-35 competitive races per cycle. (The ICC of 0.20 from the variance components is the stronger evidence.)
6. Late-shift proxy: see LATE_NEWS.md; momentum terms are not supported.

## Limits of this backtest

- Polled races are a selected, mostly competitive sample; results for unpolled races are not covered.
- LOCO uses future cycles in training. A strict forward test would be even less favourable to any learned correction.
- No mode, house-effect, or pollster-quality terms were tested (the SB file lacks mode); no fundamentals (generic ballot, approval, economy) were tested because I did not obtain them. So the full Tensor in MODEL.md is **not backtested**; only the simplified rules above were.
- `margin_actual` is as recorded in the source file, not independently verified.
- 2000 and 1998 are absent from the held-out rows because the lean rule needs a prior presidential result.
- 14 cycles. Differences under about 0.3 pts MAE are not detectable.