To be honest, I thought that it would be a layup to show there is almost no correlation between preseason and regular season performance. It’s a rarity for starters to play more than a half, and the coaching staffs seem to be more interested in player evaluation than winning. But as a statistician, I had to let the data speak.
To begin the investigation, I scraped data from Pro Football Reference for every season since 1982 (as far back as I could find preseason data). I used this data to perform a pretty simple Spearman Correlation test between preseason records and regular season win percentages (since some teams play 5 preseason game). As expected, there is a very weak correlation (r=0.18), and regressing regular season win % by preseason win % accounts for almost none of the variation (R-squared=0.03). Upon plotting the data the relationship is clear; there is nothing about preseason win % that allows us to meaningfully predict regular season win % on its own (there is a jitter effect on the points so there location does not convey their exact value).
There does appear to be some interesting distribution of playoff outcomes in the columns, so I broke the preseason outcomes into thirds based on the underlying distribution of preseason outcomes and aggregated the probability of each post season outcome.
Upon breaking the data into thirds we notice some interesting trends in the postseason outcome probabilities. As shown above, it appears that winning in the preseason is correlated with making the playoffs. There a statistically significant difference between the odds of making the playoffs after winning 0-40% of preseason games vs. winning >60% of preseason games. It appears there is an increase in playoff probability along with preseason record. But then why are 3 of the worst teams ever coming off of perfect preseasons? Is there something that could explain this phenomenon?
To investigate further, I attempted to control for the talent level of each team by incorporating last season’s win total. I again broke these prior regular season performances into thirds (0-7 Wins, 8-10 Wins, and 11-16 Wins) based on 33rd and 66th percentiles in the data. After performing the analysis on these binned data sets, we can see that the correlation is slightly more apparent for teams in the middle third (r=0.24), and non-existent for teams in the upper third (r=0).
Additionally, binning postseason performance shows that preseason success has diminishing or non existent returns in the upper and lower thirds of the teams in the study. There is a clear pattern that mid-tier teams (8-10 wins last season) who perform well in the preseason also have better chances of postseason success, while the pattern is less clear in the other tiers of teams. Shockingly, 55% of the mid tier teams that win >60% of their preseason games make the playoffs. What kind of coach wouldn’t want those playoff odds?
As everyone has no doubt heard hundreds of times, correlation is not causation. This doesn’t mean the Cowboys are doomed (although Sunday might), and it doesn’t mean that teams should be going for it on 4th and long and breaking out all their new offensive packages to try to win games in the preseason. Preseason success is more about missing injuries and assessing new talent/depth. It takes about 2 plays of watching Saquon bust through gaps to determine that the Giants are probably gonna have a more explosive offense this upcoming season. My best understanding of our findings is that preseason success is a proxy for depth and new talent. For teams that are already pretty talented (coming off 11+ wins), it really doesn’t matter what their preseason looks like. Chances are their main contributors barely even stepped on the field in the preseason. On the flip side, for struggling teams, preseason results have diminishing returns. This might mean that they are still trying to figure out who their main contributors even are, thus leading to more snaps for players higher on the depth chart. For teams in the middle that are fringe playoff teams, a good preseason could represent an influx of new talent/depth in areas of need that could bolster the teams performance. So yes, the preseason kinda matters for some teams. Bummer.