The Bad Science of Player Comparisons

[dt_divider style=”thick” /]Football evaluation has reached a new frontier. The long-standing practice of watching film to judge a prospect’s abilities remains the status quo, but new (and perhaps controversial) methods are emerging. Analytics – which has become the foundation of scouting in the post-Moneyball baseball era – has found its way to America’s gladiator sport. Not only are coaches using advanced data to guide situational decision-making, but front offices have begun using analytics, to varying degrees, to supplement player evaluations. There’s even a camp of zealots within the media that champions these metrics as the ultimate form of evaluation. (The most reasonable stance, in my estimation, is that a marriage of the two systems will produce the best results, but that’s a topic for another day).

However, this ongoing war of ideas does remind me of old lessons about rationalism and empiricism. Without going on a tangent about the diametric schools of philosophy, the former espoused the innate human powers of logic and deduction as the source from which all knowledge is derived, whereas the latter claimed sense and experience to be the basis thereof. The parallels are obvious.

Nerds and bean counters believe that information about a player can be disentangled from a mound of data. Many would likely point out the limitations of the human mind; its susceptibility to misperceptions and cognitive bias. Stubborn football purists, stuck in the ways of old, would bemoan a lack of context and limited understanding of the technical nuances of the game.

But, my point is this – if we evaluate a player based off of film, we are relying on our senses and experiences. And this is why I believe player comparisons are bad science.

Okay, so maybe evaluation isn’t an exact science. I’ll concede that. But, any scout worth their salt damn sure has a process they follow. And every scout is different, just as every player is different. We are all the sum of our experiences and we all perceive things differently, which is to say that there are too many working parts for player comparisons to be a substantive component of analysis.

When Scout A – with his or her different set of training, skills, and experiences – watches Player A, what they see will be different than what Scout B sees, to some degree. Scouting reports are like snowflakes in that no two are the same, so this is the first variable. In and of itself, this is not a huge problem, but it is compounded by several other factors. Scout A compares Player A to Player B, whom they presumably saw differently than Scout B. To make matters worse, Scout B was reminded of Player C when watching Player A. And needless to say, Scout A had his or her own interpretation of Player C.

On top of all of this, human beings have imperfect minds and are subject to subconscious biases not only in the evaluation process, but in the manufacturing of comparisons. What if Scout A’s recollection of Player B is hazy? We are incapable of ‘infinite recall,’ so the limitations of our respective memories is yet another factor. Then, our unique biases that we often unknowingly hold play a part, too. I could be reminded of a player because of a pet move. Their backstories or career trajectories could be the same. Perhaps they have similar physical profiles. Or maybe they played in comparable systems and were utilized in similar ways. Sometimes it can as trivial as the fact they went to the same school. There’s certainly a myriad of possibilities that could skew a comparison.

And just as every scout is different, every player is different. Can a comparison, in a vacuum, accurately capture a player completely and holistically? When unpacking comparisons, I’ve often found that they’re used to succinctly capture one (or a few) traits, or to deliver a narrative or projection. And after all, how could they ever capture more than a handful of traits, considering every player is different? But, when only a few aspects of a player’s game go into a comparison, scouts can ultimately wind up arguing apples and oranges. Therefore, there’s more value in discussing how and why one arrived at that comparison, than in the comparison itself.

Eventually, snowflakes become snowballs and get rolling downhill until the difference of opinion is far greater than what it would have been had the two scouts just compared their reports. It’s not that player comparisons are a lazy tool (although not all comparisons are created equally), it’s that there are too many variables for them to function as we would like them to. They fail because they are essentially a game of telephone in a field which requires precise communication. Talking about what we see, with as much detail as we can possibly offer, will always trump leaving things to the imaginations of other. Scouting is already an imperfect process executed by imperfect people.

Follow Bryce on twitter @btrossler. You can check out his other work here, such as his breakdown of New England’s run and shoot concepts on offense.

Want more Inside the Pylon? Subscribe to our podcasts, follow us on Twitter, like us on Facebook or catch us on our YouTube channel.

Editor’s note: The second to last paragraph of the piece did not originally post to the site, it has since been updated. 

Leave a Reply

Your email address will not be published.