There’s been a lot of of sturm und drang in the soccer analytics Twittersphere over the past few months, as fewer analysts publish work and concerns grow over theft of intellectual property. I briefly spoke to this a couple of weeks ago, but I think it’s worth addressing now in full.
If you haven’t seen it yet, Sam Wallace’s feature on Fulham’s use of analytics in the Telegraph is a must-read. It comes on the back of comments from manager Slavisa Jokanovic, who feels that he doesn’t have input or knowledge in the analytics side of the club’s player recruitment process overseen by American analyst Craig Kline.
Though we must always urge caution when commenting on what goes behind the scenes at clubs (there’s always more than meets the eye with these stories), I thought this was particularly interesting:
Kline is the architect at Fulham of what he calls the “Both Boxes Checked” system which is intended to give equal weight to traditional scouting methods and Kline’s own brand of statistical analysis. Without a positive score in both categories, a player will not be signed and that means, at the very least, that he has equal authority on signings with Jokanovic and the club’s chief football officer, Mike Rigg.
The thinking behind the scheme is that there should be consensus between the analytics department and the manager. Neither side should lay claim to the final say but instead, by a process of elimination, they alight upon a player who checks both boxes.
On the surface, this would seem like the ideal system. Players are scouted and vetted with the use of data analytics, while traditional scouts do their own work in tandem. If both parties are happy with a particular recruit (and presumably Jokanovic gives the okay), the signing goes ahead.
However, all parties are clearly not happy with the arrangement, enough that Jokanovic felt the need to make his grievances known to the press and risk recrimination from owners, or worse.
It’s impossible to know whether Jokanovic is overstating his lack of input in the situation, or whether the system Fulham uses is exactly as Wallace describes. For the sake of argument, however, let’s assume this “Both Boxes Checked” system is roughly as Wallace presents it.
On the one hand, The ‘BBC’ method is a simple way to balance so-called “traditional” and “analytical” approaches to player recruitment. Simply let both sides operate in parallel, and if both give the green light, both can take the blame should the signing fail.
Not only does this do away with the need for the analytics department to have to communicate their methods to a potentially skeptical audience, but it ensures that Kline’s particular analytical toolbox—the property of Tony Kahn’s company TruMedia—is not widely shared. As Wallace writes, “The actual detail of the model is the intellectual property of Tony, adjusted by Kline for European football, and it is not divulged to staff.”
On the other, the BBC method makes no attempt to integrate these two traditions. Kline the quant works in one silo, the traditional scouts and Jokanovic in another. There is a definite element of, “Trust us” going on here, and not much in the way of communication to win over any hearts and minds.
Kline himself may be bound by the legitimate need for the company to keep its wares secret, but this gets into a major issue right now in sports analytics: How do analysts balance the need to protect intellectual property with the need to communicate the effectiveness of their methods to clubs and the wider public? How can clubs validate these methods with, say, a non-interested but qualified third party, if the analyst is reluctant to divulge them?
I don’t think there is an easy answer, but I think this warrants a broader discussion about exactly what a sports quant/analyst should be in practice.
Before trying to speak to this, I want to briefly turn to Matthew B Crawford’s deeply compelling book, The World Outside Your Head. Crawford’s subject is the current age of autonomy, in which we see ourselves as individual, atomized selves working alone in a vacuum to achieve professional and personal realisation, an apt description perhaps for the less than aptly named ‘analytics community.’ Crawford attempts to hearken back to an age where knowledge was embedded in a physical world, and learning and understanding was embodied in the object and practice of our work, rather than in some disembodied mental representation in our heads.
Reading it recently, I was struck by his ostensibly idiosyncratic example of baroque pipe organ builders. To join this esteemed group, you must first apprentice with a master, from whom you will learn both the general accepted practices of the industry, in addition to the particular tricks and lessons learned from years of experience, which is itself grounded in centuries of communal tradition.
These individual shops are deeply competitive, yet what distinguishes them isn’t necessarily their tools or methods, which are often an open—if obscure—book. Rather, it’s the confidence to improvise and improve that only comes with years steeped in an experience that is itself connected to an ancient art.
This, I think, strikes at the problem currently facing the analytics movement. It is young, but it is also atomized, split into proprietary lines and black boxes. If ever there was a group of autonomous selves all striving alone and in parallel, it’s sports analysts. While there is at least some communication and development in the public sphere, there is almost no literature on the actual experience of working effectively with a team. While this ensures that analysts can build their own mathematical models and tools, there is no conversation, only fear and competition and ‘signalling’.
This atomization also limits the idea of what a quant actually does. Are analysts just metrics and visualization salespeople, plumbers who make, sell and argue for the use of a limited set of tools?
Or are they responsive, thoughtful people who are capable of creative solutions, reasoned caution, and helpful insight, individuals who change and grow over their careers?
Which is more valuable to a club? If the hope is that analysts will be hired on the merits of their metrics alone, why would a club not hire a third party consultancy instead?
Consider Leicester City, a club that, as others have mentioned, don’t even use particularly advanced methods, but have nonetheless encouraged a working dialogue between PAs, stat analysts and the manager, one that, for one season at least, produced unworldly results. Perhaps, in the end, the living, breathing, and thinking analyst, one who discusses methods and comes up with new ideas, is more valuable than the models upon which they hang their hopes. And maybe the only way an analytics tradition can flourish is if both clubs and quants can put their paranoia aside, break out of their silos, and actually start talking.