In their book, "Big Data: A Revolution That Will Transform How We Live, Work, and Think," Kenneth Cukier and Viktor Mayer-Schönberger mention Wal-Mart's discovery, gleaned by mining sales data, that people preparing for a hurricane bought lots of Pop-Tarts. Now, when a storm is on the way, Wal-Mart puts Pop-Tarts on the shelves next to the flashlights.
But what excites and concerns people about Big Data is more far-reaching than that. One way of seeing the bigger picture: taking a closer look at some of the people in the digital trenches.
* * *
I follow Mandelbaum and Mr. Youkel down a corridor of the Library of Congress, past exhibits redolent of history and what you might expect from what we call "America's library," with its 38 million books on 838 miles of shelving.
They open a door. We pass behind people staring at huge computer screens and enter a room that doesn't look as if it belongs in a library at all. It's the size of a gym, with fluorescent lights overhead and tall metal boxes rising from the floor.
"The tweets come here," Mandelbaum says.
It's been three years since Twitter approached the library with a question. What the online networking service started in 2006 had become a new way of communicating. Would there, Twitter asked, be historical value in archiving tweets?
"We saw the value right away," says Robert Dizard, deputy director of the library. "[Our] mission is, preserve the record of America."
Certainly the record of what millions of Americans say, think, and feel each day would be a treasure-trove for historians. But was the technology feasible, and – important for a federal agency – cost-effective to handle the three V's that form the fingerprint of a Big Data project – volume, velocity, and variety?