How sharp are Hawk-Eyes really? I’m talking about commercial systems, such as Hawk-Eye and GoalControl, that use cameras to record whether tennis, soccer, cricket and other balls fall in or out of bounds/the goal box/whatever.
Novak Djokovic relied on Hawk-Eye extensively this weekend to challenge calls during his Wimbledon match with Andy Murray, which Murray won. Those Hawk-Eye replays they show on TV look pretty authoritative, but actually, not all of the system’s calls are accurate. Even robot eyes can fail sometimes. The Guardian posted an opinion piece today outlining some of Hawk-Eye’s potential weaknesses.
Because of flaws in cameras, Hawk-Eye can’t always be right, but publicly, there’s not a lot of information about how big its errors could be. Acting as many companies would, Hawk-Eye doesn’t publish the math behind its system in detail. Officials at sports federations probably have an inside scoop, but in a 2008 interview, the head of the International Tennis Federation’s technical center, Stuart Miller, didn’t seem inclined to let watchers know more about Hawk-Eye’s errors. “I’m not sure that the role of Hawk-Eye is public education,” he told Nature News.
Some researchers—and mathematically inclined sports fans—argue that there’s a better way. At least one sport does it right, they say. Cricket is transparent about when Hawk-Eye is uncertain and umpires have to use their own judgment. The way that tennis uses Hawk-Eye, on the other hand, hides the technology’s uncertainty.
For a few years now, both researchers and science writers have called for Hawk-Eye replays to show the range of error in the system’s reconstruction of plays. That is, they want to see, in that replay cartoon, how far off a Hawk-Eye call could be.
Why is there error in the first place? As The Guardian explains, cameras can’t catch every moment of a ball’s flight and fast-flying tennis balls could move pretty far in between frames. Systems like Hawk-Eye depend on mathematical formulas to estimate what actually happened in between.
Companies’ reluctance to publish their formulas helps them protect their intellectual property, but makes it difficult to know what to think about public debates about their errors.
In 2008, two social scientists from Cardiff University in the U.K. published a paper in which they guessed at how many millimeters off a Hawk-Eye call could be. At the time, Hawk-Eye’s inventors disputed the Cardiff analysis, saying that it assumed a pattern of error that the actual Hawk-Eye doesn’t follow. The Cardiff folks said (I’m summarizing here; this is not a real quote): “Well, we used a normal distribution and we can’t use anything else unless you tell us more about your math.” Many things in nature have a normal distribution, which describes the likelihood of any particular outcome in a continuum.
Hawk-Eye didn’t release more information about its math. Instead, its inventors just said its error doesn’t fall in a normal distribution. Inventor Paul Hawkins also said that in tests, the system has made the right call every single time.
For now, the statistical sausage is a secret. Whether it stays that way perhaps depends on how many more fans learn about ball-callers’ flaws and say they want to know more.