Total Recall: How the E-Memory Revolution Will Change Everything (5 page)

By January 2001 my sixteen-gigabyte e-memory contained more than five thousand photographs and about one hundred thousand pages of paper: letters, memos, bills, receipts, financial statements, legal documents, ticket stubs, business cards, greeting cards, brochures, meeting agendas, symposium programs, diplomas, warrantees, manuals, purchase orders, circuit diagrams, employee evaluations, annual reports, newspaper clippings, article printouts, stock certificates, report cards, childhood drawings, birth certificates.

I’d hung on to those hundreds of pounds of yellowing paper not because I wanted to help found a thriving community of sil verfish in my home, but because I knew that someday, for some reason, I would certainly need to refind at least one old item. The vast majority of them would never see the light of day again, but I had no way to predict which one I’d need back. I couldn’t possibly know in advance which check I might need to end a payment dispute or whether I would face a tax audit for a certain year. So I’d felt trapped into keeping all of them. It took me more than a decade to throw away circuit-theory class notes from MIT, even though it was clear I wouldn’t be designing any of those kinds of circuits.

Scanning and digitizing that much paper turned out to be a very big job, so in April 1999, I hired Vicki Rozycki as my personal assistant. Over the next two years we would scan, a handful at a time, what we then thought was everything. Then, for several years after that, we continued unearthing more to scan. It took up a large amount of her time. The hard part was finding stuff and getting it ready for the scanner. (Nowadays there are commercial services that will do this sort of thing for you much faster and cheaper, using automated bulk scanners.)

I never knew quite how much I’d resented the need to stockpile so much paper until I saw it dwindle away like dirty old winter snow in the spring thaw. Folder by folder, box by box, week after week, it disappeared. The clutter and hassle of keeping paper files had been like the half-noticed droning of an electric motor that suddenly goes silent, leaving me in a startled state of peace. Not to make light of tragedy, but this passage from a recent novel,
Extremely Loud and Incredibly Close
by Jonathan Safran Foer, struck a chord with me:

[It] was the paper that kept the [World Trade Center] towers burning. All of those notepads, and Xeroxes, and printed e-mails, and photographs of kids, and books, and dollar bills in wallets, and documents in files . . . all of them were fuel. Maybe if we lived in a paperless society, which lots of scientists say we’ll probably live in one day soon, Dad would still be alive.

I also made digital records of all my physical memorabilia. The scanner worked fine for smallish, flattish items such as medals, coins, and pins, but for larger and more fully three-dimensional objects I had to use a digital camera. I took down all my paintings and made high-quality photographs of them. One of my favorites, titled
Meeting on Gauguin’s Beach,
began as a sketch rendered in 1988 by a computer program called Aaron. Then the artist who developed Aaron, Harold Cohen, hand-painted a version for me in vivid oil paints on a five-by-seven-foot canvas. Now I completed the cycle and sent it back to cyberspace.

I took pictures of all my eagles. I love eagles and have amassed a collection of eagle sculptures, picture books, postcards, knick knacks, and hand puppets. I took pictures of mugs. I have a collection of special coffee mugs. A few of them have eagles, but all of them have some connection to my past. I call some of them my “one-hundred-thousand-dollar mugs” because they are all I have to show for a one-hundred-thousand-dollar investment in a start-up company. I photographed all of those along with their companion T-shirts. The most rewarding part was putting them all in a box and delivering the whole collection to the Computer History Museum—they were someone else’s clutter now!

If any of these treasures are ever lost in an earthquake or a fire, I’ll have nice remembrances of them. And if my heirs don’t want to hang on to the cluttered remnants of my life after I’m gone, they’ll always have these images and my notes about them if they want to see what was important to me.

Of course, my collection of two hundred CDs also needed to be ripped onto my computer. I also had several drawers- and shelves ful of home movies, videotaped lectures, and voice recordings on audiocassette that were collecting dust and needed to be digitized. A service converted the 8mm movies to VHS tapes, which were later converted to CDs.

The second prong of getting my life digitized was to start scanning and recording everything I did from that point forward. In 2002 I decided to go paperless and to scan and not store or file any paper documents. I had already resolved in 1995 not to take any more paper newspapers. (Later, I was at a dinner with
New York Times
publisher Arthur Sulzberger Jr., and his description of their investments in new printing plants convinced me I was right in believing that paper newspapers were on the wane.)

For new paper it was easy. When billing statements or important notices came in, I scanned them in less time than it used to take me to physically file them away. And thankfully, the amount of paper I’ve needed to scan has shrunk each year since I began. If you piled up all the paper mail I used to receive annually, it would measure about thirteen feet high. Twelve of those feet were utter junk that could be safely thrown away—generic credit card offers, random bulk catalogs, “you may already be a winner” mailings, and the like. (Note: My goal is to record everything I actually read, not what others send me. It’s my choice, not theirs, that counts.) Only the remaining foot of paper was worth scanning, and even that has grown less as I’ve switched to digital bills and statements.

Nowadays it’s much easier to avoid paper. All the technical magazines and news sources I read are “born digital,” as are nearly all books. For legal reasons I keep a few items in paper, such as stock certificates, but not many. At that time, I started signing everything I could digitally, avoiding the creation, transmission, and especially storage of any paper.

As scanned documents and pictures piled up into my surrogate memory, I faced the challenge of figuring out how best to organize it. I began with what I had to work with: the folder hierarchy that every computer user is familiar with (every folder contains a list of files and subfolders, and every subfolder contains its own list of files and sub-subfolders, and so on). I filed my documents in folders according to a set of categories that made sense to me. From the design point of view this was not perfect, but I had to get started somehow.

In this earliest stab at organizing my scanned data, I split my e-memory into two top-level divisions: items related to current events in my life, and an archive of older, inactive information. Under those two main folders, I had dozens of subfolders for categories including books, medical records, the Computer History Museum (which I’d helped start), trips, underwater photos, food, and so on. Under “Animals” you could find a picture of alligators, various images of San Francisco’s wild parrots, and an astonishing set of images showing a snake swallowing a kangaroo whole. I had my “Eagles” folder, of all things eagle-related, and a “Fun” folder, which included a picture of the adult me swinging from a rope.

To help find things again easily, I gave each item a long, detailed file name. For example, the file name of a technical article would include the title, where it was published, the date, keywords, and other pertinent details.

But even with all my documents and pictures stashed away in a well-thought-out classification hierarchy of file folders, it was hard to find particular items quickly, if at all, because it required remembering where it was put. It was just like a library organized by subject without a card catalog. Poring through multiple folders for the right name or thumbnail icon took too much time. Without better labels, even my photos were not much use. When I looked at some, I couldn’t recall what they were about. It was painfully clear that the problem would get far, far worse once I started adding hundreds of daily pictures and hours of daily audio to the jumble.

My friend and boss, Jim Gray, teased me about it. When you burn data onto most compact discs, the operation is permanent, and this is known as “Write Once, Read Many” or WORM. Jim mocked me as the inventor of WORN: “Write Once, Read Never.”

“It’s all just a bunch of bits unless it’s annotated,” I grumbled.

I began to realize the magnitude of what was lacking. This was not a project to store my life bits; it was about how to get them back!

Scanned documents are image files, not text files, and as such, they’re invisible to keyword searches. But with thousands upon thousands of documents in my e-memory, keyword searching would be the only way to re-locate an old file that I could only recollect one or two fragments of, such as a name, a dollar amount, or a dateline. So I ran all the scanned documents through optical character recognition (OCR) software, which is able to recognize written letters and numbers in an image and reconstruct them in a text file. What I ended up with were thousands upon thousands of text files that were neatly interleaved among the scanned files.

Now I just needed desktop search software, that is, software that would allow me to search through my thousands of files for some desired text, just like you search for Web pages now using Yahoo or Google. But at this time operating systems were still several years away from offering desktop search. Desktop search was in its infancy, and every such product I tried was pretty “bleeding edge.”

I tried to get Microsoft to take the lead in desktop search, starting with the acquisition of a leading start-up, but was unable to convince the right people. I would have to wait for others to revolutionize search technology. In the meantime, if I wanted to continue my little lifelogging experiment, I would have to cobble together my own solution. In October 2001, Jim Gemmell and Roger Lueder, who had been working with me on other projects at Microsoft, decided that this would make a great research project for them to get involved in. We started out like we do with any new research project, by combing through the published literature to see what others had learned.

I dug up an old paper that I recalled as being relevant, and was surprised at just how relevant it was. In fact, it specified a system almost made to order for us. That’s pretty amazing, when you consider that it had been written more than fifty years earlier.

MEMEX

In 1945, when electronic computers were actually multistory buildings, the director of the federal Office of Scientific Research and Development, Dr. Vannevar Bush, published an essay in the
Atlantic Monthly
titled “As We May Think,” which outlined a radical new vision of how people might one day keep their own libraries of personal media. He proposed the memex:

A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.

It consists of a desk, and while it can presumably be operated from a distance, it is primarily the piece of furniture at which he works. On the top are slanting translucent screens, on which material can be projected for convenient reading. There is a keyboard, and sets of buttons and levers. Otherwise it looks like an ordinary desk.

Most of the memex contents are purchased on microfilm ready for insertion. Books of all sorts, pictures, current periodicals, newspapers, are thus obtained and dropped into place. Business correspondence takes the same path. And there is provision for direct entry. On the top of the memex is a transparent platen. On this are placed longhand notes, photographs, memoranda, all sorts of things. When one is in place, the depression of a lever causes it to be photographed. . . .

. . . As he ponders over his notes in the evening, he again talks his comments into the record. . . . He can add marginal notes and comments . . . and it could even be arranged so that he can do this by a stylus scheme. . . .

Another way to get material into the memex was with a wearable camera:

The camera hound of the future wears on his forehead a lump a little larger than a walnut. It takes pictures. . . . The lens is of universal focus. . . . There is a built-in photocell on the walnut . . . which automatically adjusts exposure for a wide range of illumination. . . . It produces its result in full color. It may well be stereoscopic, and record with two spaced glass eyes. . . .

The cord which trips its shutter may reach down a man’s sleeve within easy reach of his fingers. A quick squeeze, and the picture is taken. On a pair of ordinary glasses is a square of fine lines near the top of one lens, where it is out of the way of ordinary vision. When an object appears in that square, it is lined up for its picture. As the scientist of the future moves about the laboratory or the field, every time he looks at something worthy of the record, he trips the shutter and in it goes, without even an audible click. . . .

I love Bush’s description of the memex. The image he conjures is like something straight out of a Jules Verne novel. I envision a luxurious mahogany desk festooned with brass push-buttons, levers, and translucent screens. I can just hear the muffled clickity clicking of mechanical registers crunching numbers deep inside the casing. But even though most of Bush’s hardware suggestions are now obsolete, the antiquated trappings belie the sheer brilliance of his prescience. Bush’s desk with storage, screens, keyboard, stylus, and platen is the equivalent of today’s desktop PC with a microphone, multiple monitors, and a scanner. Add in a tablet PC and you gain pen-based input. And sub-walnut-size cameras are now affordable and plentiful. Just about all new cell phones and laptops come with one built in, and they can also be bought and worn on their own.