Blast From the Past: The Internet Archive

What's in the Internet Archive?

Of course, you won’t find “all knowledge” in the Internet Archive. Most digital artifacts are someone’s intellectual property, and they aren’t given away freely. But the Archive contains over 3 petabytes of public domain material and items contributed by supporters under the Creative Commons License. (Just for reference, you’d need three thousand 1-terabyte hard drives to hold all that data.) Most of the material is old, its copyrights having expired. Most of it is of interest to rather limited audiences. But as they say, one man’s junk is another one’s treasure.

You can find novels dating back to the 15th Century; old-time radio programs; cult films such as “Nosferatu” and “Attack of the Giant Leeches.” You’ll also find obscure podcasts that someone didn’t want to pay to host, and really bad writing that a deluded author thought should be preserved forever. Garage bands upload their amateur music, but you’ll also find old Muddy Waters albums and even a collection of Grateful Dead performances. In fact, the Archive has an entire section of fan recordings of live concerts.

The Internet Archive is sort of like a Goodwill Industries store. People drop off donations of all kinds; the donations get cleaned up a little bit, loosely categorized, and that’s about it. You’re welcome to come in and browse, but don’t expect to find everything. What do you want for free?

The Internet Archive got started in 1996, when founder Brewster Kahle decided to preserve a copy of every Web page ever published. Kahle and his partner wrote a Web crawler that searched for publicly available pages and copied them to the Archive’s tape drives. There they sat for over five years, occasionally visited by researchers, until a Web-based interface called The Wayback Machine allowed the public to surf back through time.

The Wayback Machine - Digital Time Travel

The Wayback Machine (http://archive.org/web/web.php) is simple enough. Just enter a URL in its address box and click the “take me back” button. Up pops a set of calendar pages. Dates highlighted in blue are dates when one or more snapshots of the URL’s page were taken. Click a date to see what that page looked like on the date in question. In addition to the general index, there are some interesting historical collections, such as Web Pioneers, World Trade Center Attack - September 11, 2001 and Hurricane Katrina.

The Wayback Machine has not captured every Web page that has existed since 1996, of course. It didn’t scan every day; often weeks or years passed between scans of the same site. But there are lots and lots of snapshots in the Wayback Machine archive. According to the Archive.org website, there are over 150 billion web pages archived, dating from 1996 to “a few months ago.” But I can’t find any updates since July 2011. The site’s FAQ says to expect at least 6 (and up to 24) months before new web pages appear in the Wayback Machine.

If you're interested in the history of the Internet, and how the World Wide Web came to be, check out my related articles History of the Internet and
The First Internet Celebrity.

Brewster Kahle is honored by many as an Internet librarian and preservationist. Others think he has a hoarding problem. Kahle’s latest project is to preserve one copy of every paper book ever printed. Personally, I think the Archive and the Wayback Machine are pretty cool, and appreciate the ability to see how lame my websites looked in the past. :-)

Your thoughts on this topic are welcome. Post your comment or question below...