# DATA - What exists, what we can use before 2026-11-03 All downloads below were done with `curl` into `data/` from first-party or well-known sources; nothing was executed. Files read as data only (CSV via pandas, XLSX via openpyxl). ## 1. Used in this project | Dataset | URL | Coverage | Fields | Licence | Status | | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Silver Bulletin raw polls** (`data/sb_rawpolls_021325.xlsx`, 1.75 MB, 12,306 rows) | page: https://www.natesilver.net/p/pollster-ratings-archive-2025 ; file https://www.natesilver.net/api/v1/file/433d805a-6eb9-4c19-a5a7-4c883725d59b.xlsx | 1998-2024 general-election (Pres, Sen, Gov, House) plus presidential primaries, updated Feb 13 2025 | race id, location, pollster, polldate, electiondate, sample size, candidate shares, margin_poll, margin_actual, partisan flag, type, `bias`, `error`, `rightcall` | "free of charge for any purpose, please attribute to Silver Bulletin" [WEB] | **Used for the backtest.** Clean. No `methodology`/mode column in this file; no days-to-election column (we derive it). The file URL is an opaque UUID and could change. | | **FiveThirtyEight pollster-ratings `raw_polls.csv`** (`data/raw_polls.csv`, 4.9 MB, 20,466 rows) | https://raw.githubusercontent.com/fivethirtyeight/data/master/pollster-ratings/raw_polls.csv (README https://github.com/fivethirtyeight/data/tree/master/pollster-ratings) | 1998-2022 polls (a few 2023 rows), incl. `methodology`, `time_to_election`, `aapor_roper`, `transparency_score`, generic ballot (`House-G-US`) | similar plus mode | Repo README fetch showed no licence statement (UNVERIFIED; 538 data repos historically used CC BY 4.0 [MEM]) | Downloaded successfully on 2026-10-15; used only for cross-checking and for mode inventory. **No 2024.** | Cleanliness notes: both files contain multiple rows per poll (pollster versions/questions), partisan-sponsored polls (about 9% of rows), runoffs and special elections with their own dates, and third-party/other-candidate pairings. We filtered to DEM v REP head-to-head, nonpartisan, deduplicated by pollster-race, and used each file's own certified `margin_actual`. We did not independently audit `margin_actual` against official certified results (UNVERIFIED), and a few rows have implausible values (e.g. 538 file House special elections with +/-20 pt "errors" in 2009, 2013, 2015). ## 2. 538's archive status - ABC/Disney shut FiveThirtyEight in March 2025; staff laid off [WEB: https://en.wikipedia.org/wiki/FiveThirtyEight]. The archive was reported removed/redirecting in May 2026 [WEB, secondary: https://flowingdata.com/2026/05/19/fivethirtyeight-scrubbed-from-the-internet/ , https://www.lawyersgunsmoneyblog.com/2026/05/disney-kills-the-538-archives-without-explanation]. - As of 2026-10-15 the **GitHub raw file still downloaded** (so the GitHub repo content is live today, but could vanish). Recommendation: **keep our local copy and its SHA-256, do not rely on re-fetching.** Hash is in `data/SHA256SUMS.txt`. - NYT picked up a poll-tracking database after the shutdown [WEB, secondary: https://niemanlab.org/2025/03/the-new-york-times-picks-up-the-shuttered-fivethirtyeights-poll-tracking-database]; terms and access UNVERIFIED. ## 3. Other public sources (inventory; not downloaded) | Source | What | Notes | | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | MIT Election Data + Science Lab (electionlab.mit.edu / Harvard Dataverse) [MEM] | Certified returns: president by state 1976+, Senate, House, governors | URLs not verified this session; licences are typically CC0/CC BY on Dataverse (UNVERIFIED). Best candidate for independent re-check of `margin_actual`. | | FEC / state certifying authorities | Official certified results | Primary truth; slow. | | Roper Center / iPoll [MEM] | Poll microdata | Paywalled/membership. | | Cooperative Election Study (CES/CCES) and Pew validated-voter studies [MEM] | Respondent-level, vote-validated | Free for research; needed to study nonresponse and turnout properly; not used here. | | RealClearPolling, 270toWin, Wikipedia poll tables | Scraped poll lists | Terms and quality vary; RCP terms restrictive (UNVERIFIED). Not recommended as a training source. | | AAPOR task force reports (2016, 2020, 2024) | Aggregate tables | Links in FINDINGS.md. Task force poll-level data availability UNVERIFIED. | | Macro fundamentals (approval, generic ballot, GDP, income) [MEM] | Gallup (paywalled series), BEA/FRED (free), Roper | Not downloaded. FRED/BEA are public; approval history from Gallup/UCSB American Presidency Project (UCSB data free for scholarly use, UNVERIFIED). | | GDELT | https://api.gdeltproject.org (DOC 2.0, ~3-month window) and raw file/BigQuery archives | See LATE_NEWS.md. Public, no key; historical coverage UNVERIFIED. | | Silver Bulletin pollster ratings XLSX | https://www.natesilver.net/api/v1/file/9d6ee14f-1cbd-4b37-b80c-1dee208c9994.xlsx | Pollster-level stats; not downloaded (not needed for v0). | ## 4. What we can actually use before Nov 3, 2026 Usable now: - Historical calibration set: 14 even-year cycles of race-level polls and results (1998-2024) from the Silver Bulletin file; that is all the training data a model could honestly use. - Mode inventory from the 538 file (to 2022). **Not available to me** (and therefore not in the backtest): (a) 2026 poll-level data (Vugol's own ingestion is the source; the Silver Bulletin and 538 files stop in 2024/2022); (b) a national generic-ballot series, presidential approval, economic fundamentals, and 2026 candidate data (none downloaded this session, so the fundamentals prior is **untested**); (c) respondent-level data for nonresponse modelling; (d) GDELT history. If Vugol's pipeline already stores poll rows with pollster, mode, dates, sample, and the original artifact, a 2026 feed can be scored against the same schema within 19 days; the model must therefore be shipped as a simple, frozen, documented calculation, not something to train on new data before election day.