Switch to Desktop Site

The new age of algorithms: How it affects the way we live

Next Previous

Page 7 of 15

About these ads

The library said yes. But the task is daunting.

Volume? It will archive 172 billion tweets in 2013 alone, about 300 each from the world's 500 million-plus tweeters.

Velocity? That means absorbing more than 20 million tweets an hour, 24 hours a day, seven days a week, each stored in a way that can last.

Variety? There are tweets from a woman who may run for president in 2016 – and from Lady Gaga. And they're different in other ways.

"Sure, a tweet is 140 characters," says Jim Gallagher, the library's director of strategic initiatives. "But there are 50 fields. We need to record who wrote it. Where. When."

Because many tweets seem banal, the project has inspired ridicule. When the library posted its announcement of the project, one reader wrote in the comments box: "I'm guessing a good chunk ... came from the Kardashians."

But isn't banality the point? Historians want to know not just what happened in the past but how people lived. It is why they rejoice in finding a semiliterate diary kept by a Confederate soldier, or pottery fragments in a colonial town.

It's as if a historian today writing about Lincoln could listen in on what millions of Americans were saying on the day he was shot.

Youkel and Mandelbaum might seem like an odd couple to carry out a Big Data project: One is a career Library of Congress researcher with an undergraduate degree in history, the other a geologist who worked for years with oil companies. But they demonstrate something Babson's Mr. Davenport has written about the emerging field of analytics: "hybrid specialization."

For organizations to use the new technology well, traditional skills, like computer science, aren't enough. Davenport points out that just as Big Data combines many innovations, finding meaning in the world's welter of statistics means combining many different disciplines.

Next Previous

Page 7 of 15