Share this story
Close X
Switch to Desktop Site

The new age of algorithms: How it affects the way we live

Next Previous

Page 6 of 14

About these ads

They open a door. We pass behind people staring at huge computer screens and enter a room that doesn't look as if it belongs in a library at all. It's the size of a gym, with fluorescent lights overhead and tall metal boxes rising from the floor.

"The tweets come here," Mandelbaum says.

It's been three years since Twitter approached the library with a question. What the online networking service started in 2006 had become a new way of communicating. Would there, Twitter asked, be historical value in archiving tweets?

"We saw the value right away," says Robert Dizard, deputy director of the library. "[Our] mission is, preserve the record of America."

Certainly the record of what millions of Americans say, think, and feel each day would be a treasure-trove for historians. But was the technology feasible, and – important for a federal agency – cost-effective to handle the three V's that form the fingerprint of a Big Data project – volume, velocity, and variety?

The library said yes. But the task is daunting.

Volume? It will archive 172 billion tweets in 2013 alone, about 300 each from the world's 500 million-plus tweeters.

Velocity? That means absorbing more than 20 million tweets an hour, 24 hours a day, seven days a week, each stored in a way that can last.

Variety? There are tweets from a woman who may run for president in 2016 – and from Lady Gaga. And they're different in other ways.

"Sure, a tweet is 140 characters," says Jim Gallagher, the library's director of strategic initiatives. "But there are 50 fields. We need to record who wrote it. Where. When."

Because many tweets seem banal, the project has inspired ridicule. When the library posted its announcement of the project, one reader wrote in the comments box: "I'm guessing a good chunk ... came from the Kardashians."

Next Previous

Page 6 of 14

Follow Stories Like This
Get the Monitor stories you care about delivered to your inbox.