# MODEL - Vugol Tensor: specification

Status: **specification plus a tested simplification.** Only the parts marked [DATA] are supported by our backtest (BACKTEST.md). Everything else is proposed structure, not validated.

## 0. Notation

Races r = 1..R in cycle c; pollsters p; modes m (live phone, IVR, online panel, text, mixed); days-to-election t = 0..T (t=0 is election day). Margin = DEM minus REP in points of two-party share unless stated. i indexes polls.

## 1. Latent state (what we want)

- `mu_r` = true final two-party margin of race r (election-day outcome).
- Poll-implied latent margin on day t: `theta_{r,t}`, a random walk that ends at `mu_r + delta_r`:
  `theta_{r,t-1} = theta_{r,t} + eps_{r,t}`, `eps ~ N(0, s_walk^2)` per day, with the walk anchored by `theta_{r,0} = mu_r` plus the cycle bias term below.

## 2. Poll likelihood

For poll i in race r, pollster p, mode m, field-day t_i, sample size n_i, with raw margin `y_i`:

```
y_i = theta_{r,t_i} + h_p + g_m + b_c(r)*[bias shared] + u_i
u_i ~ N(0, sigma_s,i^2 + sigma_x^2),   sigma_s,i^2 = (100^2 * 0.95) / n_eff,i   (sampling variance of a margin, approx.)
h_p ~ N(0, tau_h^2)        pollster house effect (sum-to-zero over pollsters, so no net party tilt is built in)
g_m ~ N(0, tau_g^2)        mode effect (also sum-to-zero)
sigma_x                    extra non-sampling poll-level noise
```

n_eff,i = n_i / design-effect; design effect is a hyperparameter, because published n overstates information.

## 3. The "tensor": shared error across races

The object that carries correlation is the error of the polling system relative to the outcome:
`e_{r} = (poll-implied margin) - mu_r`, collected as a tensor with modes (cycle or race) x pollster x days-to-election x mode. Decompose:

```
e_{c,r,p,m,t} =  B_c                      shared cycle bias (scalar for the whole country)
              +  S_{c,state(r)}           state/regional component
              +  O_{c,office(r)}          office component (Pres/Sen/Gov/House)
              +  E_r                      race-specific residual
              +  h_p + g_m                house and mode terms (observed per-poll in the likelihood)
              +  W_{r,t}                  late-drift term (sd grows with t)
```

Priors (all zero-mean, symmetric in party):

```
B_c ~ N(0, sigma_B^2)               [DATA] sigma_B about 2.4-2.7 pts (between-cycle SD in 14 cycles; 90% CI wide)
S_{c,s} ~ N(0, sigma_S^2) with spatial/demographic correlation Omega_S (e.g., by region or by state-similarity kernel)   [NOT estimated; the file has no state demographic covariates]
O_{c,o} ~ N(0, sigma_O^2)           [DATA, rough] office means within cycle differ by 1-3 pts
E_r ~ N(0, sigma_E^2)               [DATA] about 4.5-4.8 pts within-cycle SD
W_{r,t} ~ N(0, s_walk^2 * t)        late drift
```

A mean-zero `B_c` is the symmetric choice. It lets the model say polls are too D or too R depending on evidence, but the **evidence available before election day is polls vs fundamentals**, see section 5. The tensor is a _covariance structure_; a rank-1 or low-rank factorisation `Sigma = sigma_B^2 11' + Omega_S + diag(...)` is all that 14 cycles can support. Nothing richer should be fit.

## 4. Pooled prediction (election day)

Posterior for `mu_r` combines the poll signal and a prior:

```
mu_r ~ N( m_r,  v_r )            prior from section 5
poll average A_r  with  A_r | mu_r ~ N( mu_r + B_c + S + O + E_r + W,  total variance )
```

Total error variance for a single race's poll average, from data: about (2.4^2 + 4.7^2) = 5.3^2 pts^2 at the 0-21 day window [DATA]. Reported sampling-only intervals cover only about 42% of 90% targets [DATA], so **the variance, not the mean, is the main correction**.

## 5. Fundamentals prior (what is and is not justified)

- `m_r = lean_r + beta_1 * national_swing_c + beta_2 * incumbency_r + ...`
- [DATA, tested] A prior of "last presidential margin in the same state" with a single shrinkage weight reduced LOCO MAE from 4.18 to 4.07 pts (-0.11 pts, bootstrap 95% CI -0.25 to +0.03, better in 10 of 12 held-out cycles). That is a small, borderline gain, and it is the only bias-free improvement we found.
- Generic ballot, approval, economic indicators, incumbency: **proposed, not tested** (no data pulled; effectively 7 midterms with national-level covariates). Midterm "penalty" for the president's party is well known [WEB, secondary: https://sites.middlebury.edu/presidentialpower/tag/forecast-models], but at the poll-race level a WH-party term did not help (M2/M4 in BACKTEST.md).

## 6. Late-news term

`W_{r,t}` is a variance-only random walk. A signed GDELT term `lambda * (tone_shock_D - tone_shock_R)` is **not included** because it cannot be identified from about 5 cycles (LATE_NEWS.md). Keep a switch with `lambda = 0`.

## 7. Undecided allocation

Work in two-party share: `y_two-party = D/(D+R)`. Undecided/third-party share `U`: allocate proportionally to the two-party split by default; allocation toward the challenger/out-party is a commonly-asserted rule [MEM] UNVERIFIED and **not applied** since a one-sided rule breaks symmetry. Add `U` as variance: `sigma_U^2 = (kappa * U)^2` with kappa about 0.3 as a pre-registered hyperparameter, to be checked against the data when the file's third-party columns are cleaned (not done).

## 8. Fitting options

1. **Stan/PyMC** (full model): hierarchical state-space with the terms above. Fit time minutes. Needs poll-level data with pollster and mode (the 538 file has mode; the Silver Bulletin file does not). Not run: no time before election, and the backtest says extra complexity has not shown predictive gain.
2. **Cheap TypeScript approximation for `packages/core`** (recommended v0; no MCMC):
   - `A_r` = mean over pollsters of per-pollster mean margin in the last 21 days (each pollster counts once, so house effects average out; require at least 2 pollsters).
   - Point estimate: `mu_hat_r = lean_r + b * (A_r - lean_r)` with `b` about 0.9 (fit on 2002-2024, to be refit with the final set), or simply `A_r` if `lean_r` is unavailable. Do NOT subtract a global bias.
   - Interval: `mu_hat_r +/- z_q * sigma_tot`, with `sigma_tot` from the LOCO training residual SD (about 5.0-5.4 pts, which gave 80/90% coverage of 0.80/0.88-0.89 in BACKTEST.md).
   - Joint statements: simulate one national draw `B ~ N(0, 2.5^2)` plus race residuals `N(0, 4.6^2)` and report seat/race counts from the simulation. This is what makes the multi-race summaries coherent.
   - Versioned pure function; store input poll IDs and weights per CLAUDE.md rule.
3. Optional later: shrink `h_p` per pollster, estimate `g_m` from the 538 mode column, add the regional term `S` once demographic covariates exist.

## 9. What is justified by data

| Component                                 | Evidence                        | Verdict                               |
| ----------------------------------------- | ------------------------------- | ------------------------------------- |
| Shared cycle error `B_c` with SD ~2.4-2.7 | 14 cycles, ICC 0.20             | **Justified (as variance).**          |
| Mean of `B_c` != 0                        | t = 1.06 pooled; no persistence | **Not justified.** Mean 0.            |
| Sign predictable from last cycle          | corr -0.02                      | **Not justified.**                    |
| State-lean shrinkage                      | -0.11 MAE (CI crosses 0)        | **Weakly justified.**                 |
| Pollster/mode/house effects               | not estimated here              | Plausible; **untested** on this data. |
| Regional covariance `S`                   | no covariates                   | **Untested.**                         |
| Fundamentals (generic ballot etc.)        | none tested                     | **Untested.**                         |
| Signed late-news term                     | 5 cycles, endogenous            | **Not identifiable.**                 |

## 10. Is deep learning / naive Bayes / other ML appropriate?

Plainly: **not for the forecast itself.**

- The effective sample for anything about _systematic_ error is the number of cycles: 14 (2000-2024 even years), and only 7 presidential-year and 7 midterm cycles. Even the race-level count (about 1,000-1,500 polled races) is dependent: ICC 0.20 means cycle-level effects dominate what generalises. A neural network, gradient boosting model, or a high-dimensional "tensor decomposition" of pollster x race x day x mode has orders of magnitude more free parameters than independent cycle observations, and will fit the past and fail on 2026.
- Naive Bayes assumes conditional independence of features, which is exactly what is violated here (all signals share the cycle error). It would be overconfident.
- Our LOCO test: every correction learned from other cycles (global shift, party-of-WH term, persistence) failed to beat the plain average. More flexible learners would be worse, not better.
- **Honest roles for ML** (none of which change the headline number): (1) data engineering: LLM-assisted extraction of poll fields from articles/PDFs with human approval (already CLAUDE.md policy); (2) record linkage / pollster name matching; (3) text classification of GDELT/news topics for display; (4) at the respondent level, with vote-validated microdata (CES, Pew; not used here), multilevel regression and poststratification and nonresponse models. Those are statistical tools applied to millions of respondent rows, not to 14 election outcomes.
- Prefer simple hierarchical shrinkage, which is what the section 8.2 formula is.