Instead of a build up to Super Bowl XLIX, the media has, for over a week now, thrown the book at the New England Patriots over their alleged culpability in “Deflategate.” One such argument involved using the Patriots fumble rate as a data point to support the rabid dogs. Luckily, Inside the Pylon has better minds, including Daryl Sng, who showed that the analysis by Warren Sharp is like the Jets kickoff unit: garbage coming in equals garbage going out.
The following is a cautionary tale about data analytics in football. Short version: It can be done very badly.
A piece of analysis by NFL analyst Warren Sharp that is making the rounds on the internet purports to show that players fumbled at an unfathomably low rate in New England, both compared to teams other than New England and also compared to the same players when they left the Patriots. Sharp offers a compelling narrative given the current news cycle: New England players, according to his data, fumble much less in New England than when they leave for other teams. According to Sharp, players who played for the Patriots between 2007-2014 had a fumble rate of one fumble per 105 touches, and then regressed to 56 touches per fumble upon leaving the team. He then uses that to infer that the Patriots might have done something post-2007 to reduce their fumbling rate, such as playing with deflated balls.
None of these publications, unfortunately, actually studied the data sources being used.
In data analysis, the result you get is only as good as the data you put into the analysis. And the data Sharp uses is, to put it finely, a hot mess. Most crucially, Sharp counted fumbles on special teams plays such as punt returns and kickoff returns in his data. Let’s enumerate the problems with this:
- Kick returners like Brandon Tate and Wes Welker will have vastly inflated fumble rates simply because fumble rates on punt returns and kickoff returns are much higher than on regular plays. (A study from Football Perspective suggests that the fumble rate on kickoff returns is 3.1%, while the fumble rate on punt returns is 3.5%. Another study, conducted by Advanced Football Analytics and covering a similar period, suggests that the fumble rate on all run and pass plays is 1.67%.)
- A second reason special team fumbles screw up the analysis is that it makes the “touches per fumble” stat that Sharp calculated meaningless because he is mixing and matching data from disparate sets. For instance, Sharp considers Welker to have had 690 touches and 12 fumbles for New England, for 58 touches per fumble. However, all 690 touches were on plays from scrimmage (672 receptions, 18 rushes). Yet of those 12 fumbles, 6 were punt return fumbles. It would hardly be accurate to count “touches per fumble” unless Sharp added in the number of times Welker touched the ball on special teams.
- Finally, in relation to Deflategate, it’s worth noting that special teams balls (the so-called K-balls) are handled completely differently and kept separate from the other game balls. The 12 K-balls are sealed in a special box, shipped by the manufacturer, and never handled by the teams prior to the game. Barring some truly nefarious deeds by the Patriots to fix the K-balls involving Jason Bourne-level machinations, any analysis of ball-handling statistics to see if the Patriots did something unusual with the game balls should focus exclusively on the game balls. In effect, Sharp used data involving K-balls to make a claim about non-K-balls. That’s not science.
A proper analysis of fumble rates of regular, non-special teams balls would necessarily have to remove all data involving special team plays. The fact that Tate played many more snaps as a kick returner for Cincinnati than he did in New England, and therefore had many more chances to fumble on his new team, should not lead one to conclude that there was anything special about the way New England treated the ball or ball security.
Another problem with Sharp’s data is that his fumble totals include the postseason, even though the rushing attempt and reception numbers were purely regular season numbers. This makes the “touches per fumble” stat even less meaningful, particularly when looking at players on a team like the Patriots who regularly accrue significant post-season snaps. Let’s look at Welker again. In reality, he only had 5 regular season fumbles during plays from scrimmage; the 6th fumble was a muffed catch in the 2011 postseason game against Baltimore.
I suspect the cause of the bad data is that Sharp collected his figures from the otherwise excellent Pro-Football-Reference.com, which for some reason includes special teams and postseason fumbles under its “Receiving and Rushing” statistics. How does one tell that fumbles on special teams and playoffs were in the stats that Sharp used? The easiest way is to compare Pro-Football-Reference.com’s fumble data with NFL.com’s career stats breakdown. The PFR page for Welker, for instance, “credits” him with 25 fumbles, which is the data that Sharp uses (12 fumbles as a Patriot, 13 elsewhere). Yet his NFL.com career stats page (which goes into much greater detail concerning the situations in which he fumbled) shows that Welker only committed 6 fumbles as a receiver; he amassed another 18 regular-season fumbles ‒ one as a kickoff returner and 17 as a punt returner. Of course, that still leaves us one short of 25. Unfortunately, there’s no easy way of finding out a player’s postseason fumbles besides looking at game logs – we found the muffed catch in the 2011 Baltimore game by doing the tedious work of fact-checking.
(Welker also fumbled in the 2010 Pro Bowl, for what it’s worth. We only mention it because it’s equally relevant to the hypothesis that New England’s low fumble rate is due to deflated balls as is the K-ball fumble data.)
In redoing Sharp’s analysis of how players performed after leaving New England, we arrive at the following data:
Comparing this to Sharp’s original table, the data shows that the Patriots still have an advantage in ball security, even when comparing the same players. However, it’s nowhere near the 88% increase that Sharp claims – it’s closer to a 23% increase. That seems more consistent with the idea that the Patriots under Bill Belichick preach ball security, and Belichick’s predilection for players that do not fumble. I
Statistical analysis has already transformed baseball. Its use in football studies is increasing and will transform that sport as well. But a core principle of researching data is that any analysis is only as good as the data on which it is based. Unfortunately, in this case, bad data led to bad conclusions – conclusions which unfortunately are being amplified and repeated through the media.
EDITOR’S NOTE: The article as initially published was prepared using pre-2007 figures for a number of Patriots players, and included playoffs. An astute reader pointed out that to present an apples-to-apples comparison with Sharp’s numbers, we should look only at post-2007 data, and focus on the regular season. We have revised the numbers accordingly, and also corrected a typographical error that we spotted while doing so. Proving once again that it’s never too late for accuracy — especially with numbers!
Follow Daryl on Twitter @singaporesoxfan.