The Glory of Big Data

Suddenly, we can know the world completely. Next, we reprogram it

By Juan Enriquez Posted 10.31.2011 at 9:00 am 22 Comments

The Glory of Big Data Jesse Lenz

Late in the first day of this year’s TED Conference, its understated curator, Chris Anderson, took the stage and made a pronouncement. “The computing power in some of the things that we’re seeing is really startling,” he said. “It feels to me as if things have suddenly notched up a level in an unexpected way. We’re used to Moore’s Law. We’re used to things getting better and better and better. And then some years, it suddenly just feels as if—kapow!—there’s a step change.”

Back when it was just an exclusive conference for geek geniuses and venture capitalists, TED stood for Technology, Entertainment, Design. Now that more than 1,000 TED Talks are available free online and have been watched 300 million times, those three initials are less meaningful than is the ethos expressed in the conference’s motto, “Ideas Worth Spreading.” Either way, TED has never been about gathering its speakers together to support some predigested “trend.”

In 2011 the volume of available data is predicted to continue along its exponential growth curve to 1.8 zettabytes. (A zettabyte is a trillion gigabytes; that’s a 1 with 21 zeros trailing behind it.)Which I guess is why Chris sounded so surprised and delighted by his own statement. We’d just heard from Mattias Astrom, whose company digitally maps the world’s cities and every building in them in gloriously faithful three-dimensional renderings. After that, the digital artist Aaron Koblin explained how he visualizes massive data sets (for instance, by tracing every flight in North America and then breaking down the data by time, type of plane, altitude and so on, and presenting it all in a sequentially spoolable rendering that is both revelatory and beautiful). Koblin specializes in crowdsourced projects, such as a Johnny Cash music video drawn by thousands of strangers, frame by frame. The level of detail is breathtaking; one fan lavished 31,000 brushstrokes on a single frame. And because this is digital space and each frame’s creation is recorded and cataloged online, you can watch every stroke, just as each artist drew it.

But it wasn’t until the next morning that we truly understood what Chris had been getting at. That’s when Deb Roy, who directs the Cognitive Machines group at the MIT Media Lab, took the stage and introduced us to the ultimate home movie, 240,000 hours of video and audio covering almost every interaction his son had with anyone throughout the Roy household from the moment the newborn arrived home from the hospital. This provides a complete 1:1 scale map of how the boy learned, especially how he learned to talk—to navigate the world of abstraction, language, data. Using a raw data set that exceeds 200 terabytes (more than 20 times the size of the complete printed collection of the Library of Congress in 2000), Roy can trace exactly how his son experienced each of the words he eventually uttered, and he has teased out some fascinating insights about language acquisition.

Roy has also shown that the methods he developed to store and analyze one child’s speech lessons can be applied more broadly, and he has begun doing so. In particular, he has turned his sweeping computational eye to the social-media sphere, to watch, say, a presidential pronouncement and all of its multiplying repercussions, tweets, retweets, abbreviations, distortions and rebuttals in real time, and in doing so drawing a detailed map of large social networks and how they evolve.

The amount of data available to us is increasingly vast. In 2010 we played, swam, wallowed, and drowned in 1.2 zettabytes of the stuff, and in 2011 the volume is predicted to continue along its exponential growth curve to 1.8 zettabytes. (A zettabyte is a trillion gigabytes; that’s a 1 with 21 zeros trailing behind it.) The IDC Digital Universe study from which I’ve plucked these numbers helpfully notes that if you were inclined to store all that data on the hard drives of 32-gigabyte iPads, doing so would require 57.5 billion devices—enough to erect a 61-foot-high wall 4,005 miles long, from Miami all the way to Anchorage.

One tiny part of that vast wall would house Google’s effort to create as complete a census as possible of the published word since 1500. The company has already gathered enough data—some 500 billion words from more than five million books—to plausibly claim the emergence of a new science, culturomics. Eventually the coinage, evolution and decline of every word and phrase could be traced across centuries. Using Google’s handy Ngram Viewer, we can already observe the explosion of the word “sex” after 1960. Or watch Rembrandt citations gradually grow, exceeding those of Cezanne in 1940, only to witness Picasso blow past both of them less than a decade later. These are not scholarly samples and inferences drawn painstakingly from a few great books; this is the exacting examination of how a word or phrase’s spelling and use actually mutated year by year.

Previous Article: Today in Good Reads: The Case of the Cryptic Currency Creator

Next Article: Scottish Doctors Use Ultrasound to Heal Broken Bones

22 Comments

Subscribe Now

22 Comments

Popular Tags

Technology

Photo Galleries

Technology

Innovation Challenges

Popular on Popsci

Most Viewed

Technology

Most Emailed

Technology

Most Commented

Technology

Today on PopSci.com

Footer Menu