Back out to June and August, and the models’ scores start to spread out, with Leo at or near the top of the pack. Leo lost points over the last few weeks for scoring the Georgia race as closer to a tossup than some of the other models; it was using some more conservative assumptions about what would happen in a runoff between David Perdue and Michelle Nunn. If you rank the forecasts based on their daily logarithm scores, and then average these rankings for every day since Aug. 19 (the earliest date for which we have forecasts from six of the seven models), Leo does fairly well.  Average daily rank for the Senate forecast models, from Aug. 19 through Election Day. A lower value denotes greater relative accuracy.  But all of this is ignoring the fact that things like Brier scores or logarithmic scores, while useful in the long run, contain an irreducible element of chance. In general, putting too much weight on these post hoc evaluations could leave you vulnerable to outcome bias.  Suppose we decide to flip a coin — you say the probability that it lands heads is 50 percent; I say it’s 100 percent. I flip the coin and it lands heads. Whether you use Brier scores or logarithmic scores, my forecast will be judged better than yours. In fact, my forecast will be judged better by almost any reasonable metric. It’s only in the long run, over many repeated flips, that the flaws in my forecast would be revealed, as (eventually) tails would come up, and my logarithmic score would drop to negative infinity.  But we can’t repeat the 2014 election multiple times. The fact that there were 36 Senate races helps somewhat, but the majority of them were not competitive enough to provide useful data, and this still won’t get you past the inherent randomness. It’s only in judging outcomes over a series of elections that these methods are useful. And it’s worth noting that Drew Linzer, the creator of the Daily Kos forecast, did quite well in 2012 as well.  There are other more intangible measures that are also worthwhile — stability and transparency among them. You don’t want a forecast that gets distracted by shiny objects, so to speak, chasing after every new poll that comes in. But nor do you want a forecast so stubborn that it fails to capture meaningful shifts when they do occur. Finding this balance between stability and sensitivity — being able to distinguish signal from noise, which we explored in our comparison of Senate models over time — is part of a good forecast.  In general, the more transparent a forecast is about its methods, the better it serves this goal. We’ve tried to be as up front as we could about the choices underlying Leo and the underlying assumptions, including posting our data and code on GitHub.  But perhaps the truest measure of a forecast is whether it makes you more knowledgeable about the world around you. I hope that Leo achieved this.
The elections are over. What have we learned about the efforts to analyze and predict the outcomes of various Senate races?