######## min distinct pollsters per race = 2; races (with lean available)=946 ######## MAE mean_error RMSE cov80 cov90 wrongcall width90 races cycles model M0 plain average 4.177 1.187 5.388 0.796 0.886 0.112 17.286 946.0 12.0 M1 constant shift (global bias) 4.182 -0.013 5.345 0.802 0.890 0.117 17.286 946.0 12.0 M2 shift + WH-party term 4.300 -0.034 5.465 0.785 0.881 0.127 17.268 946.0 12.0 M3 shrink to state lean 4.065 1.057 5.272 0.801 0.884 0.110 16.865 946.0 12.0 M4 shrink + intercept + WH 4.177 -0.056 5.352 0.793 0.879 0.123 16.813 946.0 12.0 M5 persistence (half of last cycle bias) 4.242 0.570 5.453 0.802 0.881 0.124 17.284 880.0 11.0 Naive sampling-error-only 90% interval coverage (M0): 0.420; 80%: 0.340 == Cycle-level polling error (poll avg minus actual, DEM-REP pts; + = polls too pro-DEM) == races mean_err sd_err mae se_mean cyc 1998 41 -4.29 5.76 5.66 0.90 2000 88 -1.27 4.49 3.49 0.48 2002 66 2.44 5.12 4.53 0.63 2004 83 1.06 3.99 2.98 0.44 2006 94 -0.45 4.86 3.73 0.50 2008 105 0.73 5.38 3.99 0.53 2010 92 -0.40 6.58 4.82 0.69 2012 82 -2.68 3.66 3.85 0.40 2014 70 3.14 4.74 4.40 0.57 2016 79 4.10 4.66 5.03 0.52 2018 82 -0.42 4.39 3.39 0.48 2020 81 5.16 3.96 5.58 0.44 2022 56 0.14 5.64 4.25 0.75 2024 60 3.27 3.61 3.98 0.47 == By cycle x office: mean error == mean count type_simple Gov-G House-G Pres-G Sen-G Gov-G House-G Pres-G Sen-G cyc 1998 -5.48 -3.63 NaN -2.93 21.0 3.0 NaN 17.0 2000 -1.09 0.86 -1.32 -1.62 10.0 5.0 45.0 28.0 2002 3.98 0.81 NaN 1.24 31.0 14.0 NaN 21.0 2004 1.10 1.10 1.24 0.73 6.0 15.0 39.0 23.0 2006 -0.23 0.26 NaN -1.99 30.0 41.0 NaN 23.0 2008 -1.70 2.94 0.54 0.22 8.0 20.0 46.0 31.0 2010 -1.19 1.43 NaN -2.43 26.0 40.0 NaN 26.0 2012 -2.52 -2.59 -2.25 -3.39 7.0 17.0 34.0 24.0 2014 2.91 5.08 NaN 2.40 29.0 14.0 NaN 27.0 2016 4.52 7.11 4.08 3.43 8.0 5.0 40.0 26.0 2018 -0.62 -0.73 NaN 0.22 23.0 35.0 NaN 24.0 2020 8.93 5.81 4.31 6.04 4.0 9.0 47.0 21.0 2022 1.22 -3.45 NaN 0.81 23.0 11.0 NaN 22.0 2024 3.19 4.81 3.14 2.92 4.0 7.0 31.0 18.0 Variance components: between-cycle sd=2.43 pts, within-cycle (race) sd=4.84, ICC=0.20 Mean error pres-year vs midterm: mean std count pres_year False 0.20 5.64 501 True 1.33 5.07 578 Cycle signs: 1998:R, 2000:R, 2002:D, 2004:D, 2006:R, 2008:D, 2010:R, 2012:R, 2014:D, 2016:D, 2018:R, 2020:D, 2022:D, 2024:D Cycles polls too pro-DEM: 8 of 14; mean of cycle means 0.75, sd 2.65, t-stat of mean 1.06 Lag-1 corr of cycle mean error: -0.02; lag-2 (same cycle type): 0.01 == Wrong-call counts in competitive races (|avg|<=5): observed vs expected if race errors were INDEPENDENT == cyc races expected_wrong observed_wrong P(sim>=obs) P(sim<=obs) 1998 10 3.4 4 0.451 0.783 2000 31 10.6 8 0.890 0.208 2002 20 7.0 8 0.399 0.764 2004 26 8.8 7 0.836 0.296 2006 32 10.6 11 0.510 0.642 2008 21 6.9 4 0.957 0.118 2010 28 9.1 6 0.937 0.136 2012 29 9.5 7 0.890 0.209 2014 23 8.8 6 0.926 0.159 2016 28 9.3 13 0.095 0.954 2018 35 13.1 9 0.953 0.098 2020 22 7.8 8 0.540 0.634 2022 22 7.6 6 0.834 0.309 2024 18 6.6 7 0.513 0.674 (Above uses races with a prior presidential margin available, so 2000 is excluded from backtest tables.) All-race cycle table incl. 2000/2002 first cycles: cyc 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022 2024 count 41.00 88.00 66.00 83.00 94.00 105.00 92.0 82.00 70.00 79.0 82.00 81.00 56.00 60.00 mean -4.29 -1.27 2.44 1.06 -0.45 0.73 -0.4 -2.68 3.14 4.1 -0.42 5.16 0.14 3.27 Per-cycle LOCO MAE: model M0 plain average M1 constant shift (global bias) M2 shift + WH-party term M3 shrink to state lean M4 shrink + intercept + WH M5 persistence (half of last cycle bias) cyc 2002 4.53 4.23 4.24 4.25 4.07 NaN 2004 2.98 2.86 2.86 2.84 2.74 2.86 2006 3.73 4.09 4.19 4.07 4.41 3.81 2008 3.80 3.87 3.92 4.01 4.37 3.82 2010 4.82 5.11 5.16 4.59 4.74 4.84 2012 3.85 4.88 5.22 3.69 5.16 3.74 2014 4.40 3.98 4.08 4.32 3.86 5.08 2016 5.03 4.48 4.75 4.95 4.47 4.16 2018 3.39 3.75 3.84 3.00 3.32 4.05 2020 5.58 4.96 5.20 5.09 4.95 5.74 2022 4.25 4.51 4.52 4.10 4.42 5.04 2024 3.98 3.33 3.47 3.94 3.26 3.93 ######## min distinct pollsters per race = 1; races (with lean available)=1536 ######## MAE mean_error RMSE cov80 cov90 wrongcall width90 races cycles model M0 plain average 4.928 1.211 6.402 0.807 0.897 0.119 20.675 1536.0 12.0 M1 constant shift (global bias) 4.923 -0.020 6.377 0.811 0.904 0.124 20.675 1536.0 12.0 M2 shift + WH-party term 5.032 -0.049 6.495 0.803 0.896 0.127 20.657 1536.0 12.0 M3 shrink to state lean 4.858 1.214 6.337 0.803 0.902 0.111 20.324 1536.0 12.0 M4 shrink + intercept + WH 4.903 -0.066 6.385 0.796 0.900 0.123 20.240 1536.0 12.0 M5 persistence (half of last cycle bias) 5.052 0.616 6.504 0.802 0.890 0.129 20.673 1443.0 11.0 Naive sampling-error-only 90% interval coverage (M0): 0.494; 80%: 0.399 ######## min distinct pollsters per race = 3; races (with lean available)=660 ######## MAE mean_error RMSE cov80 cov90 wrongcall width90 races cycles model M0 plain average 3.867 1.191 4.884 0.792 0.885 0.118 15.576 660.0 12.0 M1 constant shift (global bias) 3.895 -0.002 4.860 0.785 0.870 0.120 15.576 660.0 12.0 M2 shift + WH-party term 4.050 -0.006 5.023 0.771 0.864 0.130 15.545 660.0 12.0 M3 shrink to state lean 3.808 1.097 4.816 0.779 0.880 0.114 15.257 660.0 12.0 M4 shrink + intercept + WH 3.973 -0.023 4.951 0.773 0.870 0.133 15.220 660.0 12.0 M5 persistence (half of last cycle bias) 3.912 0.610 4.961 0.785 0.876 0.117 15.565 615.0 11.0 Naive sampling-error-only 90% interval coverage (M0): 0.365; 80%: 0.291