How Well Did Our Sports Predictions Hold Up During A Year Of Chaos?
When we first published an interactive checking the accuracy of our various prognostications, back in April 2019, the sports world was relatively normal. The idea of playing in empty stadiums was still novel, game postponements over anything other than weather were rare, and masks were strictly for goalies and catchers (and Rip Hamilton). But a lot has changed since then  and not in a way thats especially conducive to accurate forecasts. Although the championship results from sports pandemic season(s) have been surprisingly predictable, thats a pretty rough gauge of how much more chaotic things have been since early 2020. So with our interactive getting a fresh update this week, lets dive deeper into how well our sports predictions survived the upheaval that defined the past 15 months. NFL  The last big change for our NFL prediction model came in 2019, when we added an adjustment to account for the starting quarterbacks in each game. That upgrade served the model well in its debut season, increasing its predictive accuracy as compared with our old system. On a game-to-game basis, we can see that the model is generally calibrated well  even though it is somewhat overconfident in moderate-to-large favorites and a little underconfident in slight favorites. Some of those trends stayed the same in 2020, as the model was still down on slight favorites: When we said a team had a 55 percent chance to win, the team ended up winning 63 percent of the time. But we were too high on the next tier of favorites  teams with a 60 percent chance actually lost 55 percent of their games  and underconfident on teams in the 70-to-80 percent range. (Then again, all of those differences were well within the bounds of a 95 percent confidence interval around each bins predictions.) Overall, our NFL predictions had a surprisingly good year in 2020, despite the pandemic backdrop. The Brier score for our pregame win probabilities last regular season (0.208) was the most accurate mark in any year since 2015, and teams favored by the model won 68.6 percent of their games  the best record for favorites in a season since 2015 as well. MLB  Baseball is the sport where we traditionally go out on a limb the least in our predictions, since each game is so unpredictable. (For instance, the top-ranked Dodgers would have only a 72 percent chance of beating the bottom-ranked Pirates at a neutral field, and thats about as extreme as mismatches can get in MLB.) So our calibration plots tend to be bunched a lot more tightly around 50-50 matchups in baseball than in sports such as football and basketball. In 2020, our model handled the toss-up games well, though it was somewhat inconsistent with moderate favorites: Teams with a 60 percent chance of winning actually won 63 percent of games, while teams with a 65 percent chance won only 60 percent of the time. But with bigger mismatches, it was on the money  teams with a 69 percent chance to win won exactly 69 percent of the time. In terms of overall accuracy, 2020 wasnt our baseball models worst year  though it wasnt its best, either. The models 2020 Brier score (0.243) was tied with 2017 for the worst showing since 2016, and a 57.1 percent winning percentage for favorites  while better than in 2016 or 2017  was more than 2 percentage points worse than it had been in 2018 and 2019 (59.4 percent). NBA  Our primary NBA model has probably undergone the most radical evolution over the years since we started predicting games, moving from a pure Elo-based forecast to a hybrid version that also used player ratings, and finally to a system based totally on player ratings. (And even within that last form, weve used a few different metrics for the player ratings.) So when we look at the models performance over many years, its synthesizing all the pros and cons of the various different ways weve made predictions over time. Still, there are some overall themes that were true in 2020, deeply strange as that season ended up being. The NBA models tend to be overconfident in favorites, consistently forecasting a higher win probability for teams above 50 percent odds than the rate they actually win at. The most extreme example in the overall sample is for teams with 75 percent odds, which only won at a 66 percent clip. You can see this in the 2020 sample as well, with teams in the 75 percent bin winning only 66 percent of games and teams in the 80 percent bucket winning at only a 69 percent rate:  All of those differences are within their bins respective confidence intervals, but it does seem to be a trend  and perhaps indicative of how hoops favorites operate nowadays, no matter how much we try to account for load management and the like. Either way, in terms of accuracy, 2020 was a season to forget for our NBA model. The regular season was second only to 2017 in terms of the worst Brier scores since 2016, and things got worse in the playoffs. (In response, we made a big change to how we handled our RAPTOR metric before the 2021 season.) With a winning percentage of just 64.9 percent for favorites, 2020 edged out 2017 (65 percent) as our models worst year for predicting winners since 2016. Club soccer  Soccer is somewhat different from the sports listed above, since draws are a very regular occurrence in game results. Still, we can compare win-probability forecasts with actual outcomes using the same format as our other calibration plots:  In general, our soccer forecasts line up really well along the calibration line. They do tend to be slightly underconfident in heavy underdogs and, at the other end of the spectrum, slightly overconfident in massive favorites. That trend played out a bit in 2020  teams with a predicted 80 percent chance of winning won at only a 72 percent clip. But those types of lopsided matchups are rare; the vast majority of soccer match-win probabilities fall into a range between 15 and 50 percent, where the model is best calibrated. Accuracy-wise, our soccer model didnt do quite as well as usual in 2020. Its Brier score (0.161) was the worst its been since 2017, and its winning percentage for favorites (61.6 percent) was its lowest mark in that span. The absence of home-field advantage  the effect of which we had to guess at, with empty stadiums for much of last season  may have had a particular effect on club soccer predictions, as home teams in our sample went from winning 57.8 percent of matches (again, counting ties as half-wins) in 2017, 2018 and 2019 to just 54.6 percent in 2020. March Madness  There were no mens or womens NCAA basketball tournaments in 2020. But we did have them in 2021, and I thought it would be interesting to see how those tournaments  as well as the NCAA models in general over the years  fared. For a sport built on its madness, the college basketball models are generally well calibrated. Both the womens and mens versions are slightly overconfident in teams in the 25 to 30 percent range of win probabilities, as well as for those in the 80 to 85 percent range. All of those differences are within the confidence bounds for each bin, though:  To a certain degree, 2021 was a tale of two very different tournaments. While the mens side settled down a bit after a host of early upsets, it still ended up with its worst Brier score (0.212) since we started predicting March Madness in 2014. After Baylor capped off the tournament with a surprising knockout of favored Gonzaga in the championship game, favorites finished the 2021 mens tourney with just a 62.7 percent win rate  easily the lowest for any year in our sample. But the womens tourney was a bit more predictable. Its Brier score (0.133) wasnt as good as it had been in 2015 (0.107) or 2019 (0.118), but it was much better than in 2018 (0.162), ending up around the overall average for tournaments since 2015. Likewise, favorites won 82.5 percent of their games in 2021, which wasnt quite a six-year high but was better than the historical average for the womens tournament. All in all, the difficulties and disorder of 2020 (and beyond) clearly showed up in our sports forecasts over the past year-plus. Although some leagues were fazed less than others, with the NFL somehow coming out as predictable as ever  because of course it did  many of the sports we track had some of their most chaotic seasons in recent memory. Chalk it up to changes in home advantage, less reliable data on team performance, or simply players and coaches having to worry about real-world problems in addition to on-field ones. Whatever the reason, most sports got a lot harder to predict with the pandemic raging all around them.