FROM THE ARCHIVES: Fumbling the Data: The Truth About the Patriots Fumble Rate

Instead of a build up to Super Bowl XLIX, the media has, for over a week now, thrown the book at the New England Patriots over their alleged culpability in “Deflategate.” One such argument involved using the Patriots fumble rate as a data point to support the rabid dogs. Luckily, Inside the Pylon has better minds, including Daryl Sng, who showed that the analysis by Warren Sharp is like the Jets kickoff unit: garbage coming in equals garbage going out.

The following is a cautionary tale about data analytics in football. Short version: It can be done very badly.

A piece of analysis by NFL analyst Warren Sharp that is making the rounds on the internet purports to show that players fumbled at an unfathomably low rate in New England, both compared to teams other than New England and also compared to the same players when they left the Patriots. Sharp offers a compelling narrative given the current news cycle: New England players, according to his data, fumble much less in New England than when they leave for other teams. According to Sharp, players who played for the Patriots between 2007-2014 had a fumble rate of one fumble per 105 touches, and then regressed to 56 touches per fumble upon leaving the team. He then uses that to infer that the Patriots might have done something post-2007 to reduce their fumbling rate, such as playing with deflated balls.

The study went viral and was picked up by such august publications as Slate, Vox, and the Wall Street Journal. Perhaps understandably so: it fits right in with the trend towards data journalism.

None of these publications, unfortunately, actually studied the data sources being used.

In data analysis, the result you get is only as good as the data you put into the analysis. And the data Sharp uses is, to put it finely, a hot mess. Most crucially, Sharp counted fumbles on special teams plays such as punt returns and kickoff returns in his data. Let’s enumerate the problems with this:

  1. Kick returners like Brandon Tate and Wes Welker will have vastly inflated fumble rates simply because fumble rates on punt returns and kickoff returns are much higher than on regular plays. (A study from Football Perspective suggests that the fumble rate on kickoff returns is 3.1%, while the fumble rate on punt returns is 3.5%. Another study, conducted by Advanced Football Analytics and covering a similar period, suggests that the fumble rate on all run and pass plays is 1.67%.)
  2. A second reason special team fumbles screw up the analysis is that it makes the “touches per fumble” stat that Sharp calculated meaningless because he is mixing and matching data from disparate sets. For instance, Sharp considers Welker to have had 690 touches and 12 fumbles for New England, for 58 touches per fumble. However, all 690 touches were on plays from scrimmage (672 receptions, 18 rushes). Yet of those 12 fumbles, 6 were punt return fumbles. It would hardly be accurate to count “touches per fumble” unless Sharp added in the number of times Welker touched the ball on special teams.
  3. Finally, in relation to Deflategate, it’s worth noting that special teams balls (the so-called K-balls) are handled completely differently and kept separate from the other game balls. The 12 K-balls are sealed in a special box, shipped by the manufacturer, and never handled by the teams prior to the game. Barring some truly nefarious deeds by the Patriots to fix the K-balls involving Jason Bourne-level machinations, any analysis of ball-handling statistics to see if the Patriots did something unusual with the game balls should focus exclusively on the game balls. In effect, Sharp used data involving K-balls to make a claim about non-K-balls. That’s not science.

A proper analysis of fumble rates of regular, non-special teams balls would necessarily have to remove all data involving special team plays. The fact that Tate played many more snaps as a kick returner for Cincinnati than he did in New England, and therefore had many more chances to fumble on his new team, should not lead one to conclude that there was anything special about the way New England treated the ball or ball security.

Another problem with Sharp’s data is that his fumble totals include the postseason, even though the rushing attempt and reception numbers were purely regular season numbers. This makes the “touches per fumble” stat even less meaningful, particularly when looking at players on a team like the Patriots who regularly accrue significant post-season snaps. Let’s look at Welker again. In reality, he only had 5 regular season fumbles during plays from scrimmage; the 6th fumble was a muffed catch in the 2011 postseason game against Baltimore.

I suspect the cause of the bad data is that Sharp collected his figures from the otherwise excellent Pro-Football-Reference.com, which for some reason includes special teams and postseason fumbles under its “Receiving and Rushing” statistics. How does one tell that fumbles on special teams and playoffs were in the stats that Sharp used? The easiest way is to compare Pro-Football-Reference.com’s fumble data with NFL.com’s career stats breakdown. The PFR page for Welker, for instance, “credits” him with 25 fumbles, which is the data that Sharp uses (12 fumbles as a Patriot, 13 elsewhere). Yet his NFL.com career stats page (which goes into much greater detail concerning the situations in which he fumbled) shows that Welker only committed 6 fumbles as a receiver; he amassed another 18 regular-season fumbles ‒ one as a kickoff returner and 17 as a punt returner. Of course, that still leaves us one short of 25. Unfortunately, there’s no easy way of finding out a player’s postseason fumbles besides looking at game logs – we found the muffed catch in the 2011 Baltimore game by doing the tedious work of fact-checking.

(Welker also fumbled in the 2010 Pro Bowl, for what it’s worth. We only mention it because it’s equally relevant to the hypothesis that New England’s low fumble rate is due to deflated balls as is the K-ball fumble data.)

In redoing Sharp’s analysis of how players performed after leaving New England, we arrive at the following data:

Patriots Other
Player REC RSH TOU FUM TOU/FUM REC RSH TOU FUM TOU/FUM
Wes Welker 672 18 690 6 115.0 122 1 122 0
Laurence Maroney 18 470 425 4 106.3 4 36 40 1 40.0
BenJarvus Green-Ellis 26 510 536 0 26 498 524 5 104.8
Danny Woodhead 92 250 342 3 114.0 81 121 202 2 101.0
LeGarrette Blount 6 213 219 3 73.0 6 65 71 1 71.0
Ben Watson 87 1 88 2 44.0 154 1 155 1 155.0
Jabar Gaffney 74 0 74 0 191 0 191 2 95.5
Brandon Lloyd 74 0 74 0 14 0 14 0
Donte’ Stallworth 47 1 4 0 41 8 49 1 49.0
Brandon Tate 24 6 30 1 30.0 31 4 35 0
TOTAL 1120 1406 2526 19 132.9 670 733 1403 13 107.9

 

Comparing this to Sharp’s original table, the data shows that the Patriots still have an advantage in ball security, even when comparing the same players. However, it’s nowhere near the 88% increase that Sharp claims – it’s closer to a 23% increase. That seems more consistent with the idea that the Patriots under Bill Belichick preach ball security, and Belichick’s predilection for players that do not fumble. If these players had combined for just four more fumbles over a seven-year span during their Patriots tenure, their overall collective fumble rate be nearly identical to their non-Patriots fumble rate.

Statistical analysis has already transformed baseball. Its use in football studies is increasing and will transform that sport as well. But a core principle of researching data is that any analysis is only as good as the data on which it is based. Unfortunately, in this case, bad data led to bad conclusions – conclusions which unfortunately are being amplified and repeated through the media.

EDITOR’S NOTE: The article as initially published was prepared using pre-2007 figures for a number of Patriots players, and included playoffs. An astute reader pointed out that to present an apples-to-apples comparison with Sharp’s numbers, we should look only at post-2007 data, and focus on the regular season. We have revised the numbers accordingly, and also corrected a typographical error that we spotted while doing so. Proving once again that it’s never too late for accuracy — especially with numbers!

Follow Daryl on Twitter @singaporesoxfan.

Inside The Pylon covers the NFL and college football, reviewing the film, breaking downmatchups, and looking at the issues, on and off the field.

70 thoughts on “FROM THE ARCHIVES: Fumbling the Data: The Truth About the Patriots Fumble Rate

  1. Why did some values go up? I.e. Ben Watson went from 2 NE fumbles in Sharp’s table to 6 here, as well as 88 touches in Sharp’s table to over 300 here. Is that due to the inclusion of postseason touches?

  2. “Unfortunately, there’s no easy way of finding out a player’s postseason fumbles besides looking at game logs – we found the muffed catch in the 2011 Baltimore game by doing the tedious work of fact-checking.”

    I found it in about 30 seconds. It’s not tedious at all. You just have to use PFR’s game finder.

    http://pfref.com/tiny/UdCtC

    Looking at the stat breakdown, you’ll see that Baltimore is the team that the fumble came against. Then sort the plays by team, and look at everything that went against the Ravens. Or to make it easier on yourself, repeat the search with just the Ravens on offense, which gives you this result (http://pfref.com/tiny/Eo14u).

    Then you’ll see that Welker muffed a punt.

  3. You may have touched on this, but there are so many problems with the original “study” that I could have missed it.

    1. The original table includes ST fumbles, but doesn’t include ST touches in the touches column. So that’s not good….

    2. This is how Brandon Tate (according to the table) has 11 fumbles on 35 touches for other teams, making up about half of the total fumbles. That’s probably why they called him Ole Butterhands.

  4. Haha, I love how the author had to change his data from 7% to 23%, but didn’t change his conclusion. 23% may not be as damning as 88%, but it’s still HIGHLY suspicious.

      1. Why is a 23% difference in NE’s fumble rate versus the rest of the league suspicious, given that deflated footballs are easier to secure AND that the difference correlates to when the rules were changed? Gee, I don’t know.

        1. But it’s not a difference in NE’s overall fumble rate vs the rest of the league overall. The study only covers those players who played for New England AND for at least one other team. Daryl’s analysis set out to show that the Sharp analysis was complete nonsense — which it is.

          As Daryl notes, just 4 more fumbles by this group with the Patriots over a SEVEN-YEAR SPAN and the two rates would be identical. That’s 112 games. When such a tiny adjustment in the data produces such enormous changes in the observed results, it’s a sign of small sample sizes with lots of noise. That’s because it only assesses a minuscule percentage of NFL ball touches.

          Using a small data set like this to purport that ANYTHING is suspicious is ignorant. Using the wrong data on top of that (as Sharp did) is laughable. That’s what Daryl proved.

          You must place a lot of faith in Spring Training batting averages.

    1. The change really just shows the extreme sensitivity of the data to the inputs–which you really don’t like to see when you’re trying to draw strong conclusions. And note in this current article: “If these players had combined for just four more fumbles over a seven-year span during their Patriots tenure, their overall collective fumble rate be nearly identical to their non-Patriots fumble rate.”

  5. ^ Another thing: If, as the author suggests, Belichick’s emphasis on not fumbling and preference for players who don’t fumble explains the difference with the rest of the league, then why were his teams right at the league average up until 2007 when the rules changed? Did he suddenly have a revelation that fumbling is bad?

    1. You draw attention to another flaw in the earlier analysis. The rules changed in 2007 to permit the AWAY team to prepare its own balls. Prior to this, the home team prepared ALL the balls. Thus, Advanced Football Analytics’ exclusion of teams that play their home games in a dome makes no sense. If the conclusion you’re trying to prove (good science!) is that the 2007 rule change permitted the Patriots to start cheating, then you should be looking only at fumble rates on the road, and there is no logical reason to exclude teams that play their home teams on the road.

      1. Derek,

        If the home team prepared all the balls, and then both teams shared them, there would have been no advantage to deflating them prior to the rule change in 2007. Your opponent would share the benefits and make the exercise pointless.

        So there was no incentive to take a little air out before 2007, unless the balls were being segregated during the game and deflation was done subtly on the sideline. I have my doubts that was feasible.

        I couldn’t quite decipher the end of your last sentence: “and there is no logical reason to exclude teams that play their home teams on the road.”

  6. A better mind probably should not make the following statement,

    “That seems more consistent with the idea that the Patriots under Bill Belichick preach ball security, and Belichick’s predilection for players that do not fumble.”

    It is basically nonsense that Bill Belichick preaches ball security and/or looks to players with better ball security than other coaches. Considering that it is well known that turnovers are one of the most important factor, if not the most important factor, to winning a game, almost every coach/GM in the league preaches ball security and looks for players that hold onto the ball. Perhaps they do not do as well as Belichick at that, but a better mind should be able to present that evidence rather than relying on media narratives.

    1. Point taken on how all NFL coaches are concerned about turnovers. Check this out though: It’s is an interesting piece on the issue from the New York Times from 2008 that indicates how Belichick is aberrational in his approach to the state of analytics with respect to fumbling. What’s most interesting about this, to me anyway, is that he’s usually on the forefront of the advanced data like with 4th downs and such, but he disagrees with the data on this issue:

      http://www.nytimes.com/2008/11/23/sports/football/23titans.html


      Schwartz, now the defensive coordinator for the Tennessee Titans, had an economics degree from Georgetown University, an abiding fascination with statistics and a preference for watching game film over television. That made him a kindred spirit with his first N.F.L. boss, Bill Belichick. But when Schwartz told Belichick his findings from an early N.F.L. research project almost 15 years ago, Belichick said he did not believe him.

      “Fumbles are a random occurrence,” Schwartz said he told Belichick. “Being able to get interceptions or not throw interceptions has a high correlation with good teams. But over the course of a year, good teams don’t fumble any more or less than bad teams. Bill didn’t agree. He said, ‘No, good teams don’t fumble the ball.’ But actually, they fumble just as often as bad teams.”

    2. “It is basically nonsense that Bill Belichick preaches ball security and/or looks to players with better ball security than other coaches.”

      No, it’s actually not. DeMarco Murray led the league in fumbles from a RB, Belichick, conversely, benched Stevan Ridley for fumbling less than Murray did. And he was a 1200 yd runner.

      From Antwan Smith to Corey Dillion to Ben-Jarvis Green-Ellis to Blount, the runners he clearly prefers are ones the least prone to fumbling. His 3rd down backs, people like Faulk, Vereen and Woodhead, rarely saw too many carries.

      So, sure, everyone might “preach” ball security, but the Pats actually install runners who are good at it.

  7. Sharp’s numbers are flawed, but not “meaningless.”

    Rather than focussing on a smallish sample of players who’ve played elsewhere (which includes kick-returners), let’s think about how far off Sharp’s overall team-by-team numbers were. Which is something that you do not do here. [Note: I’m NOT implying it’s because you’re a Pats fan. That would be unkind!]

    To correct Sharps numbers you need to take out fumbles on kickoff returns (and make some other adjustments). When you do, is that going to make the Pats look less sure-handed, or more so?

    I don’t know, but I suspect that the results won’t be all that much different. Because although kick returns are roughly twice as likely to be fumbled, plays from scrimmage outnumber kick returns by a factor of at least ten to one. Teams today average about 4-6 total returns per game. And they average about 60 plays from scrimmage per game.

    So even if the Pats’ punt-return-fumble percentage is grossly different from other teams’, which I imagine it’s not, a lot of your (very good) objection will get eaten up by the sheer volume of non-kick-return data.

    So, again, Sharp’s numbers are skewed, as you correctly point out. But it’s not clear in which direction they are skewed as far as the team-by-team comparison is concerned, and they are far from “meaningless.” Qualitatively, they probably mean something, as I suspect you’d see if you ran a corrected formula.

  8. So Harvard put their best guy on the case, and the lowest he can get the number down to is 23%, which still shows that Patriots fumbled far less frequently for New England since 2007 than the same players did when playing for other teams.

    Now put this on your abacus: Tom Brady, the guy who touches the ball more than anyone else for the Pats, saw his fumbling rate drop by about 50% since 2007 compared to his first six years in the league. Quick, somebody call M.I.T.!

    1. Check out Peyton Manning’s decline over that same exact period. It’s just as extreme, and from 2006-2012 dropped 55%. Both Brady and Manning decided to stop cheating in 2013 and 2014 it appears, as they fumbled 16 and 17 times, respectively.

        1. For what it’s worth, could we stop calling it a 2007 rule change? It was changed prior to the 2006 season, which raises the question just how much the people circling the wagons have even researched the topic in the rush to find stats that paint the Patriots as non-fumbling cheaters.

  9. Myrna,

    The data is skewed, badly. Not just the individual, for reasons concluded here, but the team as well. It’s a snapshot in time chosen strategically to mask all other variables. Trend analysis of the same kind shows that their fumble rate isn’t all that odd. A regression model shows that they were projected to fumble less over time based on their play mix % changing over time. The Colts the year prior had a better 5 yr fumble rate, which disproves the “anomaly” or “1 in 16000” nonsense.

    As for the individual data. If an advantage was had by ne it stands to reason the same would be true for individuals. The fact he used special teams fumbles without adjusting based on fumble type or not even having the decency to use the special teams touches shows he had an agenda or isn’t good at this. Neither is forgivable when the insinuation is this obvious. If you remove fumbles from special teams the advantage is negligent and traced almost solely to one player, an outlier. Meaning on a player by player basis, large majority showed nothing to indicate an advantage to the patriots.

    A positional analysis shows that Tom Brady is the main beneficiary while the other skill positions are hardly different over time. This, and the Colts data mentioned earlier, leads to the obvious reason his handpicked time looked so weird. The patriots have one of the greatest qbs in the history of football, as did the Colts. Over time the patriots recognized and built accordingly. In 2007 there was a huge shift in their play mix, hence the fumble declines. Interestingly in the regression model the patriots are fumbling more often than they “should” over the last 3 years. Where did the advantage go? Hard truth, there never was an unfair advantage, just an amazing qb who controls majority of the offensive plays.

  10. Just curious why you left off LaGarrette Blount’s stats from Tampa, where he was fumbling like a fumblebee. And did Danny Amendola get abducted by aliens? I don’t even see him listed on your chart. Sharp wasn’t just comparing how frequently players fumbled AFTER leaving New England, but also how they did before coming there, which is a more honest look at the discrepancy than this concoction.

  11. Mike,

    If I may, those are very good callouts but both have legitimate answers. Amendola was used mostly as kr/pr specialist w rams, majority of touches from that avenue which has a much higher fumble rate than the way the pats used him (as a wr). Special teams fumbles happen at 2x rate of rb/wr touches. Unless this is adjusted for can’t compare the two, it’s oranges to apples.

    Blount was a fumble machine his first two seasons with the bucs. But his last season with the bucs and his time with the Steelers he actually has a better fumble rate than his time with the pats. In other words, his fumbling issues were “fixed” prior to the patriots or he simply can’t handle the rigor of being #1 back (only time he fumbled often was with bigger workloads).

    I did a similar analysis removing tate and amendola, bc of the return issues (all 11 of tates fumbles with cinci were on special teams) and I saw a less dramatic difference where 18 players with over 3000+ touches with patriots and with other teams since 2007 had a fumble rate of 106 w new England and 99 everywhere else. Statistically irrelevant.

  12. Unsurprisingly, another NE based person trying to disprove the theories that the Pats take license with the rules. And once again, using smokescreen to distract from the real issue. The real issue is not that the Patriots fumble significantly less than the rest of the league. The real issue is that, prior to the change in the handling of game balls, they did not. Prior to the 2007 rule change, the Patriots rate of fumbling was on par with that of other teams in the league. Once the Pats were given the ability to prep their own game balls, that rate was slashed to a small fraction of it’s prior rate. Did the Patriots suddenly learn discipline in 2007, keeping in mind that all their titles came prior to that? Or, as is more likely, did the ability to prep their own balls, legally or illegally, directly contribute to a drastic reduction in fumbles by the Pats?

    In this there is no lie, no misleading stats. It is fact that 2000-2006, the Pats were an average team when it came to holding on to the football, and 2007-present they have been maybe the best team in history at holding on to the football. And none of the Pats apologists will try to explain this drastic difference, because they can’t. So they obfuscate by going after the trees and ignoring the forest.

  13. Chris,

    Wow. Really? I don’t think you can just arbitrarily decide not to count Blount’s fumbles in Tampa because you have decided his problems were “fixed” and therefore … well, just shouldn’t count. That’s a pretty amazing rationalization to ignore hard data. This is rigorous statistical analysis?

    And we’re just going to pretend guys like Amendola and Moss never existed for similarly bizarre reasons? Oh, man,

  14. Patriots fans, gotta love em. Picking and choosing their facts and data to hide the fact that their team has been constantly and consistently cheating since the arrival of Bill Belichick.

    3 Superbowl wins, none of them legitimate. And if they win on Sunday, it means nothing, other than a 4th illegitimate title.

    1. Hey Eric, I will admit right off the bat, i am a HUGE Pats fan, so hopefully I can be critical of myself and not let that get me got. Know what I mean? Anyways, I sincerely wanted to know why you think the 3 Super Bowl wins were illegitimate, because I’ve seen it a lot and I’m not sure why everyone thinks that. While I think the deflated footballs story right now is beyond stupid, the reason I was interested was to see what made the statistical analysis suspect. My brother has his master’s in statistical analysis, so maybe I’ll forward both to him. But seriously, I’m interested to know why you think the Super bowls were illegitimate, the 2001, 2003, and 2004 seasons I’m assuming you mean. Thanks!

      1. They are illegitimate because the Patriots cheated in those Super Bowls. They were caught video taping the opposing teams closed off walk throughs (the practices where new wrinkles and trick plays and plays that they haven’t previously put on tape are installed). These practices are closed to the public and to the media due to the sensitive nature of what’s being installed during these sessions. The Patriots illegally taped these practices and had an unfair advantage of being able to prepare prior to the games for any new wrinkles or trick plays.

        Furthermore, the Patriots would tape the opposing teams defensive signals and then spend the inordinately long Super Bowl half time matching the signals up with the play calls so they could radio in to Tom Brady exactly what play the defense was going to run and let him know precisely which players were blitzing and what coverage they were running.

        How much easier is it to play QB when you know what the defense is going to do on every play before the snap? Tom Brady never had to read a defense in the second half of those games. He knew what was coming before the ball was even snapped. In addition, the coaches routinely would call the PERFECT play on every third down blitz. They would always call the absolute PERFECT blitz beating screen pass every single time a defense blitzed in the second quarter.

        Just go back and watch the 2004 Super Bowl. The Patriots could barely get a 1st down in the first half against the Eagles. The second half comes around and every single time they blitzed they would run the PERFECT screen pass right behind the blitz and gain 20-30 yards. Over and over again.

        Leading up to the discovery of these methods and the cheating that was going on, every single one of these teams would remark after the game, how it was uncanny how the Patriots always knew the defensive play before the snap, how Tom would call out the blitzers and alert his teamates to the types of coverages that they were going to see, before the defense even lined up. Everyone thought him and Belichick were geniuses and then we all found out they were just cheating.

        When you’re winning Super Bowls by 1-3 points, that’s the type of advantage that wins you the game. All those late game drives to secure a win look a lot less impressive when you realize that Tom Brady had the answers to the test before he took it.

  15. Mike, you seem to enjoy cherry picking. First of all, he addressed Amendola. So even if you take into account Blunts numbers, he is one player, who has played less than 2 full seasons with the Patriots, too small of a sample size to mean anything. Chris understands statistics, which is why he explained that regression analysis and trend analysis proves this whole attempt to prove the Pats cheated with data is false, if you do it properly. But I doubt you will be persuaded because your responses have not shown reading comprehension.

    1. Trevor,

      Amendola had 4 fumbles AS A RECEIVER for St. Louis, so no, the explanation for leaving him out doesn’t hold up. Blount has 219 touches for New England, considerably more than Watson, Gaffney, Lloyd, Stallworth or Tate (who are all included). Leaving out Blount’s 9 fumbles in Tampa from 2010-11 is preposterous and totally skews these results. And the logic for erasing the existence of Randy Moss (who actually fumbled slightly more often as a Patriot)? Was his sample size also too small? It looks like it was just a cover for dumping a bunch of other player’s legitimate stats that don’t support the desired outcome.

      1. Moss had 1007 touches from scrimmage and 13 fumbles or 1.29%. He had 252 of those touches in NE and 5 fumbles. a 1.98% rate.
        Blount has had 737 touches from scrimmage to date and 13 fumbles or 1.76%. He has had 219 touches in NE and 3 fumbles for 1.37%.

        There is less difference between Blount’s drop-off and Moss’ increase.

        Danny Amendola has had 4 fumbles in 290 touches or 1.38% but hasn’t had one in his 82 touches in New England. Stevan Ridley didn’t fumble this year in 98 touches but he did fumble 9 times prior to this year. In 672 touches he fumbles at 1.34%. These things do tend to average out and statistically to look at one guy and his 2 injury limited years in New England is not particularly relevant.

    1. Ben Jarvis Green was an outlier. He didn’t fumble for a couple years and then he went to the Bengals and had a bunch. If you take him out of the equation, players actually fumble a little more on average with the Patriots. Also, the most important person to look at is Tom Brady, who fumbled on average 12 times a year in his first 3 seasons (11, 12 and 13) – that was the outlier. He became less fumble prone starting in 2004 – 3 years before the rule change. So you could say Brady fumbled twice as much before 2007 (between 2001-2006) That’s true. Sounds fishy. Until you realize he fumbled 2.5 times as much his first 3 years in the league. Sometimes QBs mature and work on their craft and then plateau. The 2 years leading up to the rule changed (playing with league footballs) he had only 4 fumbles per season (his career average, except for his first 3 years, is 5 fumbles per season.)

      1. Dan,

        Actually, taking BJGE out of the equation (which you can’t do) still doesn’t do the trick, not if you include Blount’s 2010-11 fumbles in Tampa, and not if you can find the ditch where this analyst dumped Amendola and Sammy Morris. As for Brady, regardless of how you want to spin it, his fumbling rate dropped by almost 50% starting in 2007. He still had 12 fumbles in 2006, tied for 2nd-highest in his career, so no it doesn’t look like he got it all figured out after 2003. It was 2007, when he got the rule change he wanted, when his fumbling suddenly took a nosedive.

        1. Mike, Blount fumbled once every 7 times he caught a pass in Tampa. He only has 4 catches in NE. Is Blount’s fumble per rushing attempt any different? Maybe NE picked up on a tendency or are utilizing him better.
          I guess the Brady data I have is different than yours. The only outlier was Brady’s first 3 years in the league where he fumbled 12 times per season (total fumbles). This is where I found the Brady fumble stats – http://www.teamrankings.com/nfl/player-stat/fumbles and below are the total fumbles per season.
          2014 – 6
          2013 – 9
          2012 – 2
          2011 – 6
          2010 – 3
          2009 – 4
          2008 – DNP
          2007 – 6
          2006 – 4
          2005 – 4
          2004 – 7
          2003 – 13
          2002 – 11
          2001 – 12

  16. The notion that “t’s closer to a 23% increase. That seems more consistent with the idea that the Patriots under Bill Belichick preach ball security, and Belichick’s predilection for players that do not fumble” would make sense if not for the fact that numbers seem to change sharply in 2007. For example, look at Brady’s interception percentage pre-2007 and post 2007. There is a rather stark change. Pre-2007 2.6%, 2007 and beyond-1.6%. Sample sizes are very similar and this cannot be explained away by the K-ball. It clearly does have to do with having perhaps better receivers (although even in 2013 with a depleted receiving group his int % was below 2%), changes to the rules on contact to receivers (although it is unclear how that would strongly change int%-perhaps a league-wide study would tell) and maturation as a QB (although he is now 37, and these were his years of age 30-37. It is clear his arm is a far cry now from what it was in his younger days). In all, again proof of nothing, but food for thought.

  17. chris-

    Not sure where your “regression model” is, and the “trend analysis” is also MIA. Anyway, both, if they exist, don’t mean a ton unless they’re correlated with the real world. And by that I mean with those of every other team in the league.

    But I do know that this isn’t worth much:

    “As for the individual data. If an advantage was had by ne it stands to reason the same would be true for individuals. The fact he used special teams fumbles without adjusting based on fumble type or not even having the decency to use the special teams touches shows he had an agenda or isn’t good at this. ”

    Those two sentences are muddy at best. For one, the above data shows that it IS true for individuals, just not to the same degree as Sharp’s analysis. Moreover, when you’re looking at an entire team over several 5-year periods, for a total of 8 seasons, no, it does not “stand to reason” that a sample of ten individuals would tell the same exact story.

    And no, this isn’t about Sharp’s “decency.” What a strange way to put it – it shows that whether or not Sharp has “an agenda,” you definitely do. Which is fine. The numbers don’t care.

    The fact is that every readily-available box score does the same thing that Sharp did: lump all fumbles together. It’s not an “agenda” or a matter of “decency,” it’s an artifact of the way football stats are collected. Also, even though you could figure out how many “touches” the special teams got (that’s available in the kick returns stats), those are in fact meaningless if you want to do this perfectly. What you need to do is remove the fumbles on kick returns, not add in special teams touches. That should be obvious.

    And that corrected analysis is what I’d love to see, because that’s what’s important here. Go ahead, give it a try. Again, Sharp’s methodology was flawed, but not fatally so. It’s actually a pretty clever approach, and as I said, I’m not sure it wouldn’t hold up once his model was corrected.

    Nor are you.

    Again, we’re dealing with less than one-tenth of touches, which means about one fifth of fumbles on average. So even if the Pats are insanely awesome at keeping the ball during kick returns (sneak preview: they’re not) Sharp’s model would be skewed against the Pats by a maximum of 20%.

    20% is a helluva a lot of statistical noise, I admit, but as I said, that’s a theoretical maximum. The Pats do indeed fumble kickoffs. So go ahead, run the numbers for all the teams without those special teams fumbles and whatnot. I suspect the Pats will still have an absurdly low fumble rate on plays from scrimmage.

    Phantom “trend analysis” notwithstanding.

  18. Touches/fumble is a poorly designed test statistic, since is unnecessarily excludes the contribution from players who do not fumble. Also, perform a t-test to see if the resulting test statistics likely come from different populations, and if so, at what level of significance. Anything else is just hand-waving.

  19. Also, isn’t lost fumbles a measure of (a) the ability to hang on to the ball and (b) the ability of your team to recover the ball. That seems like two separate tests, one of which would seem to have little or no bearing on the inflation of the ball. I have no idea what the analysis would show, but wouldn’t fumbles (period – non-special teams) be a more appropriate test? (Putting aside the many other errors in Sharp’s analysis.)

    Second, Tom Brady handles the ball on 99.x% of these plays. When he doesn’t run himself on a QB sneak or scramble, another player also handles the ball (either hand-off or reception). It seems to me the overall analysis is very Brady-centric (i.e., Brady is responsible for approximately 50% of the touches). In this vein, if Brady gets rid of the ball faster, is protected better, has stronger hands, etc. over time, that could significantly account for a difference.

    1. I think Barnwell has done some pretty extensive analysis on this and basically come to the conclusion some teams fumble less than other teams, however, once a fumble hits the ground, it’s basically 50/50 who recovers. A lot of that is just the randomness of how balls are shaped and bounce around.

  20. Luckily, no one really needs to argue about whether a 7% or a 23% or an 88% increase in “touches per fumble” is significant, because this is not a judgement call. There are real, established statistical tools to determine if a difference like this is significant or could have occurred by chance.

    To illustrate this, imagine Bob and Alice each flip 5 coins and are trying to get heads to come up. Bob gets 2 heads. Alice gets 3 heads. So Alice’s success rate at getting heads is 50% higher than Bob’s. So Alice must be cheating, right?

    The relevant question is this: is 19 fumbles in 2526 touches (a 0.75% fumble rate) significantly better in a statistical sense than 13 fumbles in 1403 touches (a 0.93% fumble rate)? The number to look at is the “test statistic”, Z*. The formula is:

    Z* = (p2-p1)/sqrt(pp*qp*(1/n2+1/n1))

    where p1 and p2 are the fumble rates with and without the Patriots (0.0075, 0.0093). pp is the “pooled” fumble rate ((13+19)/(1403+2526) = 0.0081). qp = 1 – pp. And n1 and n2 are the number of touches with and without the Patriots.

    When you run the calculation, Z* = 0.582. From here, you can calculate the “p-value” (I won’t get into the details, but there is a single excel function, NORMSDIST(), which calculates p-value from Z*. In this case, the p-value is 0.28. The standard statistical rule is that the p-value must be less than 0.05 to be confident that there is an actual statistical difference.

    The way to think of p-value is this: Assuming there isn’t actually a real difference in the fundamental fumble rates of the players when they played with Patriots or not, if you went back in time and reran all the plays 1000 times, you would see a difference in the fumble rate that was as high as this or higher 28% of the time just by pure random chance.

    1. OK Mark, so if I follow what you’re saying — even taking at face value this analysis (which leaves out some players and some fumbles) — it’s more likely (72%) than not (28%) that the Patriots were up to some funny business?

      1. Mike: No. This doesn’t mean that there is a 28% chance that the difference was random and a 72% chance that is was cheating. That is a common misunderstanding of the p-value. The p-value of 0.28 simply means that, assuming there was no cheating involved, you would expect to see a difference this big or larger 28% of the time.

        To illustrate this, let’s take the coin flipping example again. Let’s say Bob and Alice each flip a coin 1000 times. Bob gets 498 heads and Alice gets 501 heads. Obviously there is nothing unusual about those results. When you calculate the p-value to see if Alice flipped a statistically significantly higher percentage of heads, the p-value is 0.45. This clearly does not mean that there was a 45% chance that the result occurred randomly and a 55% chance that Alice was cheating. It simply means that every time you run this trial, Alice will get 3 or more heads than Bob 45% of the time. The other 55% of the time, she’ll get 2 or fewer heads more than Bob.

        This also illustrates how taking percentages of percentages can be misleading. At face value, this group of players had a 20% lower RELATIVE fumble rate when they played with the Patriots. But the difference in their absolute fumble rate, as a group, was 0.18%. That translates to a difference of less than one fumble every 500 touches. Some of these players didn’t even have 500 touches with the Patriots.

          1. Gotcha, thanks, Mark. As you said, “assuming there was no cheating involved, you would expect to see a difference this big 28% of the time.” Therefore, 72% of the time, you would not expect to see a difference this big, assuming no cheating was involved.

          2. Right. In exactly the same sense that 72% of the time you flip 100 coins, you expect to get less than 53 heads, assuming there is no cheating involved. But if someone tells you they flipped 100 coins and got 53 heads, you wouldn’t assume they were cheating or lying.

            Putting it yet another way… assuming fumble rates don’t fundamentally change when players change teams, if you performed this same analysis for all 32 teams in the NFL (i.e. comparing fumble rates for players when they were with the team vs on other teams), you would expect 8 other teams to show as large or larger improvement in fumble rate as the Patriots.

          3. Mark,

            I also read somewhere (can’t remember now) that the 28% number you cite would also need to be cut in half (14%) because the 28% figure takes into account the likelihood that such a big difference could occur “either way.” In this case, it would be a 28% chance that EITHER the players would fumble less for the Patriots OR less for the other teams. Since in this case we’re only talking about the likelihood that they fumbled less for the Patriots, the number should therefore be half — 14%. Or did you already factor that in? Thanks.

  21. Yes every coach preaches ball protection but when people say you can’t attribute any difference between the Patriots and other teams in coaching this is missing the point. Preaching and coaching are 2 different things. Bill Polian just mentioned the incredible job Belichick does with special teams – the last 6 games he watched, he said, all of them – there was a key special teams play that turned the tide for the Patriots. So yes, there will be statistical outliers. Watch him break down film sometime. A game winning drive – he points out a receiver that was more open. Even a successful play he will notice how it could be better. And this is just what he is saying publicly. So yes it’s important and everyone preaches ball security, but COACHING is an entirely different matter. Saying all coaches preach how important ball security is so that makes a level playing field is silly. Of course, it’s subjective and not something that you could quantify in statistics, but it wouldn’t be surprising to have one of the greatest coaches ever have a team with better statistics. Not only Patriot fans call Belichick one of the greatest ever – it’s almost every coach, player, former staff member, analyst. Denying a Hall of Fame coach gives you ANY statistical advantage at all is asinine.

  22. Even if you believe the statistics show the Patriots were an outlier in having less fumbles than every other team many make the assumption they know the reason – under inflated footballs. The only science test I’ve seen on whether a football with 2 psi less gives you an advantage concluded the advantage was so slim it was almost negligible. In fact there were disadvantages as well so it could be they cancel each other out. Sports Science ESPN did the study. For example the grip on the ball would be less than one millimeter more for the average person and a 30 yard pass would travel just a bit slower so a defender could get the tip of a fingertip closer. The advantages were negligible – and they cancel each other out. But, if these tiny differences did add up over time they would probably show themselves in other areas. For example, a ball that is less prone to be fumbled could probably be caught easier. Where do the Patriots rank in drops since 2007? On average, 21st, as in near the bottom of the league – or highest percentage of drops per target. If the ball traveling slower made any statistical difference wouldn’t it show up somewhere in incompletion percentage or interceptions? So even if you believe there is a fumble anomaly – another step is needed to show the under inflated ball is the reason. So even if the Patriots have been using under inflated the football for 8 years without detection – making the assumption the under inflated ball gave them an advantage that would lead to a statistical anomaly isn’t backed up the science. And currently this entirely different football the Patriots might have been playing with has yet to reveal any other statistical anomalies that have been mentioned yet, but would logically be expected.

  23. People assume Warren Sharp, who wrote the original fumble anomaly article, has a motivation to bash the Patriots. Warren is actually an analyst. He has a website where you can sign up and pay for his football tips and analysis. His website says he is “a licensed Professional Engineer by trade who applies the same critical thought process and problem solving techniques into his passion, football.” Let’s be clear: He’s not putting information out there to be generous or to protect the integrity of football or to bash the Patriots. And if he was exceptional we certainly wouldn’t know his name because he would be keeping his discoveries quiet and winning a fortune for himself. Just a guy who looked into a trendy topic and searched for anomalies and got his work done to the best of his ability in order to drum up some business. I’m sure it doesn’t hurt the article went viral right before the most heavily bet game of the year.

  24. First and foremost, I didn’t write this post nor choose the charts displayed. My email is attached (they ask for it at least) so if you’re curious then ask me and ill show the data, corrected, that disproves any notion that the patriots benefited from a rule change that ACTUALLY took place in 06, but who cares right?

    I was simply explaining Blount, and the major point was that he’s proven he’s progressed beyond just the patriots by having better fumble rates elsewhere. The fact you don’t understand why Amendola can’t be used is ridiculous when fumbling on kr/pr happens about 4x as often as a wr on offense and his usage rates were much different.

    The regression model uses every team, every play, since 2001. Again it predicted the patriots to decrease their rate of fumbling based on their offensive mix changes, that began specifically in 2007. Weird, isn’t that the time frame he cherry picked? The patriots became a pass heavy team that had it’s lowest kr/pr and sack rate (both heavily influencing fumbles) since 2001! So confidence? Clearly not, very predictable.

    Also, if you remove incompletions, because let’s face it you absolutely cannot fumble an incompletion so they artificially raise fumble rates for heavy passing teams, the patriots aren’t even the best in football during this time frame. If you look at the individual seasons the patriots do have a few seasons that are amazing, but the others are above average at best. Which, if you roll then altogether masks the fact that an unfair advantage may not exist.

    So, if the rule changed in 06 and you looked at another 5 year trend you would see the patriots in the same light, right? Well, they’re good but the Colts are definitively better at not fumbling. Cheaters!!!! Or, coincidentally, it’s another heavy passing offense with a stellar oline and hof qb that has the quickest release in football. Coincidence? Seems to be a lot of them at this point.

    At the end of the day there are 2 major issues with Sharps work. It’s done incorrectly with a lot of statistical errors and when corrected it no longer holds up. The patriots are one of the best teams in the nfl at not fumbling, but its predictable, they’re not THE best, and they fit nicely on the plot. No anomaly.

    For those idiots complaining of bias, I’m a cowboys fan. But I’m also an analyst, and someone falsely (purposely or not) parading poor “analysis” as an insinuation it proves a team cheated is something I can’t ignore. Especially when done by someone with no credentials (other than self proclamation) or review with other statisticians. And when nate silver comes out and puts your flaws on blast, fighting it is just an obvious admittance of ignorance.

    1. Chris,
      My response was to your comments describing the supposedly “legitimate” reasons for ignoring stats.

      Leaving out stats including Blount’s 9 fumbles on rushes and receptions with Tampa in 2010-11 — http://www.nfl.com/player/legarretteblount/497149/profile — is a bizarre decision, any way you slice it. If we’re trying to compare players’ performances with New England and with other teams, we can’t just pretend they never played before coming to the Patriots. Not with multiple examples of players who fumbled at a higher rate after leaving New England — therefore undermining the claim that they had acquired improved ball security techniques.

      As for Amendola, I agree his special teams fumbles should be excluded, but that doesn’t justify dumping his 4 fumbles as a receiver in St. Louis, where he had 196 receptions and 1,766 yards in 4 seasons. http://espn.go.com/nfl/player/stats/_/id/11674/danny-amendola Regardless of his “usage rates,” those are pertinent numbers.

      Sammy Morris also has plenty of relevant stats — http://www.nfl.com/player/sammymorris/2504126/profile — including 4 seasons in New England during the period in question. Once again, mysteriously MIA.

    2. Brady’s sack rate did decline starting in 2007, but for the period of 2007-2014, it was only 14.28% lower than from 2001-2006 — http://www.nfl.com/player/tombrady/2504211/profile However, his fumbling rate on passing plays declined at a much more dramatic rate, 48.18% — http://espn.go.com/nfl/player/stats/_/id/2330/tom-brady
      His fumbling rate on rushes follows a very similar trend.

      I agree that the Colts’ and Falcons’ numbers do warrant closer examination as well. Interesting the Peyton Manning also lobbied for the rule change. Just as it’s highly unlikely that the AFC Championship Game was the first time the Patriots have been using significantly underinflated balls, it’s also unlikely that they were the only team that’s been doing so.

      1. Why are 2001-2006 and 2007-2014 continually cited? 2006 is the first year of the rule change, and he fumbled 12 times, three times as much as he did the previous season. It seems like an obvious case of moving the goalposts to make the decline seem more extreme. If the rule change caused the decline, why on earth aren’t you including 2006 in the post rule change totals?

        1. A fair question, Steve. Statisticians often use a “washout” period when examining the effect of a change, because there typically is an adjustment that must be made to the new status quo. In this case, as teams adjusted to the new system, it’s unlikely they would have tried to “push the envelope” right away. There’s probably going to be a “feeling out” phase first as teams and officials learn the new protocols — and officials probably were more strict initially. To use another example, a shoplifter typically starts off stealing gum or candy, and if he gets away with that for awhile, he gradually shifts to stealing an Xbox or a TV.

          1. It’s equally credible to say this testing the envelope would have taken 2 or 3 games, or 2 or 3 seasons. If they were progressing pushing the envelope a slow consistent increase should have been detected. Steve’s point remains undisputed – the “washout period” of exactly one season is a random start point.

  25. Sharp tried a good analytical approach to support his pre-determined conclusion – Patriots deflated their balls and therefore gained advantage in fumbles. The best example of this bias is seen in excluding dome teams from the damning conclusion. Why exclude indoor teams? They are likely to play some games outside every season. If indoor teams are always going to be better at fumbles, then why are indoor teams spread all over the list in fumble rates? Is it possible that some teams are just better or lucky in fumbles, regardless of whether they play indoors or outdoors?

    Worst part is that he excluded Falcons. That is what made me think that the exclusions were delibrate. He simply did not want another ‘outlier’ to disprove his conclusions. Falcons have a higher percentage lead over Patriots in fumbles rates. They outstrip NFL average fumble rate, outdoor teams fumble average AND indoor teams fumble average by bigger margins than Patriots! They outstrip indoor team fumble average by 50%! And Sharp just excluded them. Unbelievable!

    1. From 2004 to 2010, the Colts had at least the 4th fewest fumbles each year, including several years with the fewest. Shockingly, they were ALSO a dome team

        1. Just to point out, fumbles per reception is a considerably smaller sample size than fumbles per offensive play or fumbles per touch (rushes+receptions+sacks). Smaller sample size means higher variation and less reliability.

          1. Giants also fumbled less per reception over the same time period. Although a smaller sample size than if you add rushes and sacks, 8 years is not a small sample size! We are still operating within the handpicked years of the study, which begins in 2007 when the Patriots went 16-0, which has never been done before or since. The claim is being made that “the results that the Patriots have gotten over this period, (happen) once in 16,233.77 instances”. (The 2 decimals make it seem almost believable.) What are the odds these 2 teams would outperform the rest of the league AND the Patriots, who have an “impossible” fumble rate? It’s simply miraculous (if the Patriot’s are indeed impossible). Miracle on top of miracle the Giants also play outdoors. While the Patriots fumbled the least per 100 rushes, the difference from league average amounts to 1.5-2 fumbles per season. Not nearly an impossible number to wrap your mind around. Also note: Sharp begins his piece by noting (in bold) what he calls a “remarkable” fact: “The 2014 Patriots were just the 3rd team in the last 25 years to never have lost a fumble at home!” Well, that’s awful catchy isn’t it? But we know that recovering a fumble is random so why leave out how many fumbles they had? They had 6 fumbles at home this season. The Vikings had 2 fumbles on the road this season. If that Patriot fact is indeed remarkable, what is the Viking’s road statistic?

  26. One stat that has almost no meaning at all is how Patriots players fumble on other teams, before and after. The more understanding people have of the game of football the less relevant it becomes. Much as people deny or disregard this fact, personnel decisions and coaching are fundamental to a team’s success. Since it’s difficult to quantify the statistical effect it is left out of the equation entirely and you are left with a bunch of nothing. Since Sharp’s study starts with 2007 let’s look at 2 additions the Patriots made in 2007 – Wes Welker and Randy Moss. Combined the Patriots got 4x as many touchdowns from these players as the previous 2 teams in the previous 2 years. (Moss missed 3 games with the Raiders, otherwise both players were healthy for all 4 years.) Four times as many touchdowns! And one of those seasons was with Matt Cassell at QB! Wes Welker wasn’t even a starter with the Dolphins. Randy Moss was thought to be washed up. For 3 years before joining the Pats Randy Moss 3 season avg. 775 yd per game and 8 tds. First 3 with Pats 1,255/15.6TD. What did these players do after leaving the Patriots? Welker’s numbers went down significantly. Moss dropped off the map. The Patriots simply get more out of players than other teams. A reduction in fumbles while on the Patriots should be expected. Maybe an under inflated ball was so much easier to catch? No, the Patriots, on average, rank 21st, as in highest percentage of drops per target, or near the bottom of the league since 2007. (Maybe catching is something you just can’t coach.) The Welker Moss stats are off the charts anyway you look at them – more than the fumbles – but this amazing increased production on the Patriots (before and after other teams) is not even mentioned or even factored in to the data in any way. In addition, other variables would affect the fumble rate on other teams. How the player is utilized makes a difference (coaching). How often he plays makes a difference. Opponent’s defenses make a difference (random). Whether protecting a lead or not makes a big difference. Blount, for example, played for the Bucs for a few years. This is a historically losing franchise. His style was different, he was utilized more as a pass catcher, he was playing from behind, etc. The Bucs tried to get TOO MUCH out of him. The Pats get just the right amount. Blount is a guy with a history of temper problems. A great coach could change his attitude and fundamentals and get better production, as happened with Randy Moss. Every single team in the league passed on Blount when he hit the waivers mid-season! Looking at Blount and saying it’s suspicious he fumbled less with the Patriots is absurd you have to remove all the factors I mentioned which is absurd and it makes ANY statistical analysis of the Patriots (compared to most teams in the league over the past 15 years) VERY limited.

  27. Are you retarded? What you point out makes an even more compelling case AGAINST the Patriots. Correct, kickoffs and punts have much higher rates of fumbles per play than do plays from scrimmage. Correct, kickoffs and punts should not be used in the data to study whether the Pats fumble less because of doctoring the ball. And by eliminating those stats Welker’s fumble rate with the Pats goes from 1 in 60 to 1 per 120. i.e. If the ratio of fumbles during kickoffs/punts vs. from line of scrimmage is statistically greater than other NFL teams (which it sounds like it is), this is only further evidence something strange is going on.

    This reminds me of my favorite joke from Car Talk:
    A young man in Cambridge puts all his items on the conveyor belt at the local grocery. He gets to the cashier. She looks at him and says, “You must go to Harvard or MIT.” He responds, “Why yes, that’s correct. How did you know.” She replies, “Because this is the ten items or fewer line, and you have sixteen items. So either you go to MIT and you can’t read. Or you go to Harvard, and you can’t count.”

    I see the author went to Harvard.

  28. There are errors in the chart that result it in not footing correctly. Laurence Maroney has 18 receptions and 470 carries but is listed with only 425 touches. Stallworth has 47 receptions and 1 rush but is listed with a total of 4 touches. Additionally, the rush and total touches columns don’t foot (the total touches shows 2526, it would show 2482 footed properly, but it should say 2589 if you fixed the mistakes mentioned earlier).

Leave a Reply

Your email address will not be published. Required fields are marked *