# Findings: polling error and its causes

Convention: margin = DEM minus REP; error = poll minus result; positive error means polls too favourable to Democrats.
Tags: [RUN] computed here by `backtest.py` (`out\results.txt`); [WEB] read this session; [MEM] background knowledge, UNVERIFIED.

_Written by the research subagent (2026-10-15); saved verbatim by the main session because the subagent could not write this file._

## 1a. Computed error by cycle [RUN]

Source: https://www.natesilver.net/p/pollster-ratings-archive-2025. Filter: head-to-head DEM v REP, nonpartisan sponsors, fielded 0-21 days out, one value per pollster per race, at least 2 pollsters. Offices: President (state level), Senate, Governor, House (polled districts only). This is polled, mostly competitive races.

| Cycle | Races | Mean err | Mean abs err |
| ----- | ----- | -------- | ------------ |
| 1998  | 41    | -4.29    | 5.66         |
| 2000  | 88    | -1.27    | 3.49         |
| 2002  | 66    | +2.44    | 4.53         |
| 2004  | 83    | +1.06    | 2.98         |
| 2006  | 94    | -0.45    | 3.73         |
| 2008  | 105   | +0.73    | 3.99         |
| 2010  | 92    | -0.40    | 4.82         |
| 2012  | 82    | -2.68    | 3.85         |
| 2014  | 70    | +3.14    | 4.40         |
| 2016  | 79    | +4.10    | 5.03         |
| 2018  | 82    | -0.42    | 3.39         |
| 2020  | 81    | +5.16    | 5.58         |
| 2022  | 56    | +0.14    | 4.25         |
| 2024  | 60    | +3.27    | 3.98         |

- By office (see `out\results.txt`): 2020 governor, House, Senate and President were all +4 to +9; 2016 all +3.4 to +7.1; 2012 all -2.3 to -3.4. 2014 was a midterm with polls too Democratic: Senate +2.4, Governor +2.9, House +5.1.
- Pooled: mean of 14 cycle means +0.75, SD 2.65, t=1.06. Between-cycle SD 2.43, within-cycle race SD 4.84, ICC 0.20. Lag-1 and lag-2 correlations are about 0. Presidential years average +1.33 and midterms +0.20, which is within noise.

Independent summaries (definitions differ, so numbers won't match):

- Silver Bulletin: 2022 bias about D+0.8; 2018 about R+0.5; polls overstated Democrats by 3, 4.7 and 2.9 in 2016, 2020 and 2024. [WEB] https://www.natesilver.net/p/actually-sometimes-polls-underestimate
- AAPOR 2024: 611 final-two-week polls, average absolute two-party-margin error 3.3 vs 5.3 (2020) and 5.2 (2016); polls underestimated Republicans for the third straight presidential cycle. [WEB] https://aapor.org/publications/2024-pre-election-polling-report/
- AAPOR 2020: largest national error in about 40 years, largest state error in at least 20; Democratic margin overstated nearly everywhere; Democratic and Republican pollsters were off by similar amounts; cause unknown, nonresponse plausible; no shy-Trump evidence. [WEB] https://aapor.org/wp-content/uploads/2022/11/AAPOR-Task-Force-on-2020-Pre-Election-Polling_Report-FNL.pdf , https://www.pewresearch.org/short-reads/2021/07/21/a-conversation-about-u-s-election-polling-problems-in-2020/
- AAPOR 2016: national polls fairly accurate (aggregate Clinton +3.2 vs +2.1 actual). State misses were attributed to missing education weighting and late deciders; in Wisconsin, 14% decided in the last week and broke for Trump nearly 2:1. [WEB, secondary] https://aapor.org/wp-content/uploads/2023/01/AAPOR-2016-Election-Polling-Report.pdf , https://www.pewresearch.org/short-reads/2017/06/16/qa-pew-research-centers-president-on-key-issues-in-u-s-polling/
- Shirani-Mehr, Rothschild, Goel, Gelman, JASA 2018: 4,221 polls, 608 races, 1998-2014; total RMSE about 3.5 pts, about twice the stated margin; election-level bias about 2 pts; errors are shared within an election. [WEB] https://ideas.repec.org/a/taf/jnlasa/v113y2018i522p607-614.html
- 2022: several Senate races missed by more than 4 pts per RealClearPolitics averages. [WEB, secondary] https://www.niemanlab.org/2022/11/some-midterm-polls-were-on-target-but-finding-which-pollsters-to-believe-can-be-tough/

## 1b. Generic ballot / House

Generic-ballot error was not computed (UNVERIFIED). The 538 file has a `House-G-US` type (883 rows, no 2024) that was not analysed. A claim that the generic-ballot average missed the House popular vote by 4.1 pts on average is quoted at https://fivethirtyeight.com/features/some-early-clues-about-how-the-midterms-will-go and is unverified.

## 2. Mechanisms: established vs contested

- **Established:**
  - Sampling error understates total error. Sampling-only 90% intervals cover about 42% of races [RUN].
  - Errors are correlated within a cycle (ICC 0.20) [RUN, WEB].
  - Shared state errors are standard practice. The Economist/Columbia model has a "correlated state errors" section: Heidemanns, Gelman, Morris, HDSR 2020, https://hdsr.mitpress.mit.edu/pub/nw1dzd02 (section title only seen).
- **Education under-weighting:** established for 2016, not sufficient for 2020 or 2024.
- **Partisan differential nonresponse:** the leading suspect, but contested and not measurable from toplines.
- **Shy Trump voters:** mostly rejected (AAPOR 2020).
- **Late deciders:** cycle-specific. Found in 2016, not in 2020.
- **Turnout / likely-voter modelling:** plausible midterm driver. AAPOR 2020 found mail and early voting handled reasonably.
- **Mode effects:** untested here.
- **Third parties and undecided:** handle with two-party share plus an explicit variance term. Size UNVERIFIED.
- **Herding:** contested, UNVERIFIED.

## 3. Implications

1. "Polls understate Republicans" is a recent-cycle regularity (2014, 2016, 2020, 2024), not a law. The sign flipped in 2012 and 2018, so a symmetric model must leave the sign unknown.
2. The forecast-relevant quantity is a cycle-level shared bias with mean near 0 and SD about 2.4-2.7 pts. Its width matters more than its centre.
3. No method has been shown to predict the sign in advance (see BACKTEST.md).