Technology Means The End Of Reviewers Fudging Numbers

The Times published a scathing review of a test drive in the new Tesla Model S, in which reporter John Broder described the futuristic electric supercar (itself a winner of a Popular Science Best Of What’s New award) repeatedly losing its charge, behaving oddly, and shutting down during a long-distance road trip along the Northeast Corridor.

In response, Elon Musk, founder of Tesla, wrote a furious blog post in which he, in no uncertain terms, accused Broder of lying, of fudging his numbers. More about that here.

One of the most interesting things about this very interesting (and ongoing) story is the news that Tesla has begun implementing detailed tracking systems in their test cars specifically when those cars are lent to journalists. That way they can know exactly what happened during the test, and in the case of a potentially inaccurate review, they have proof of what really happened.

This is totally unprecedented, but I suspect it will become more commonplace very soon. Most consumer products now have the capability to record data–there’s no reason why, say, Apple can’t do this when they send out review units of the new iPhone. That way when a reviewer says the battery life could hardly make it through a day, Apple can check to make sure the phone was fully charged at the beginning of the day, or that the way the phone was used is representative of a normal daily use case.

And this could be done for just about any review. Restaurants could have temperature gauges in plates and utensils. (“According to the smartfork, the veal was 44 degrees C when it touched the reviewer’s tongue. Hardly ‘tepid.'”) Books could communicate whether the reviewer really did finish the book, and how quickly. Clothing could measure whether wear and tear were natural. Travel reviewers could have GPS trackers to make sure they actually visited the places they say they did. It’s creepy, sure, but reviewers have a responsibility to the public to get things right.

The other side of this is that, unlike what Gizmodo blogger Jesus Diaz suggests, of course you can argue against data. Data lies all the time! In the case of Elon Musk and the New York Times, a perfect example is that Broder, the Times writer, describes his Model S running out of charge and having to be lifted onto a flatbed and towed to a charging station. In his response, Musk writes: “As the State of Charge log shows, the Model S battery never ran out of energy at any time, including when Broder called the flatbed truck.” Seems pretty cut and dried, right? Broder must have lied!

But not necessarily! Electronic items, including electric cars but also items like smartphones, will shut down with some charge remaining, perhaps even up to five percent. That’s so the gadget can maintain its memory, and also sometimes because the gadget in question needs more than that very low percent of charge to operate properly. It is totally factually possible that Broder’s Model S never ran out of energy, even while on the flatbed–but that also doesn’t mean the car could actually drive. And in fact, Jalopnik reports that the towing company says the Model S was, in fact, dead when they arrived.

Data can have the potential to keep reviewers precise and honest. And as a reviewer myself, I don’t fault Tesla at all for wanting to track the test drive–in fact, I’m surprised other companies don’t also do this, and I wouldn’t mind if the gadgets I test were also tracked. But as this kind of debate happens more and more, it’s important to remember that raw data isn’t always a smoking gun. It illuminates, but can also obscure.