Systems to Capture Everything:

Beyond cameras and desktopswww.MyLifeBits.com

Gordon Bell, Jim Gemmell, Roger LuederGordon Bell, Jim Gemmell, Roger Lueder

Outline

MyLifeBits aka MemexMyLifeBits aka MemexHow has the project evolved? How has the project evolved? How do we use MyLifeBits? How do we use MyLifeBits? How is it built?How is it built?Shape of the database? Shape of the database?

CARPE- Continuous archiving and CARPE- Continuous archiving and recording of personal experiencerecording of personal experienceWhat is the vision?What is the vision?Relevance for devices and software?Relevance for devices and software?

I am data

History: Telepresence

Tele-presentations

Tele-meetings

Ambience and Presence:Being there while being here

Dining at home on the “Orient Express”

History: The remote worker re-discovers the PERSONAL

computer

Oct 1998

Can we scan your books and put them online?

Raj Reddy

Sure! Don’t worry about

copyright stuff. Microsoft has lots of lawyers

1999 – Scanning starts in earnest“we” start to scan,

put content intofolders & files

My docs and archive

Self....Biographical

X- Employer

EmployerEmployer

X-EmployerProjectProject

ProjectProject

Employer

Library/file cab

Library/file cab

Library/file cab

Library/file cab

Library/file cab

Library/file cab

Active Employer

Library/file cab

Library/file cabLibrary/file cab

<1980s Library/file cabLibrary/file cab

Project

BusinessInvests, family $s, & Legal

Personal, including Medical

Library/file cab

Now that it’s in CyberspaceHow do you remember the 20,000+ file names?

Or in which of 1500 folders they live?

What’s about a tool for finding stuff?

Jan 2001 CACM “A Personal Digital Store”

16 GB; +2/yr16 GB; +2/yr A good place to A good place to

stopstop Began search for Began search for

search engines, search engines, especially for email.especially for email.

Jim suggests that Jim suggests that we build a system we build a system that would be that would be easier to use and easier to use and have many more have many more capabilities.capabilities.

2001 Capture goes beyond paper

Jim, I don’t need no

stinkin’ database!

Gordon, You should be using a database.

Re-discovery of MemexAs We May Think, Vannevar Bush, 1945

““A memex is a device in which an individual stores all A memex is a device in which an individual stores all his books, records, and communications, and which his books, records, and communications, and which is mechanized so that it may be consulted with is mechanized so that it may be consulted with exceeding speed and flexibility”exceeding speed and flexibility”

Full-text search, text & audio annotations, and Full-text search, text & audio annotations, and hyperlinkshyperlinks

Even more capture

Telephone calls, more video, all web pages Telephone calls, more video, all web pages visited, keyboard and mouse usage logging, visited, keyboard and mouse usage logging, radio, TV…radio, TV…

2003 - SenseCam

Feb 2005

Epiphany!Epiphany!

Memex is a database Memex is a database & &

personal TP systempersonal TP system

Demo Clips & Screens

747 Screen…

Vue de jour

Timeline

Pivoting: contact> call> t> web page

GPS Photo location

Reports

The Stew family tree

Copyright Mark Stewart, 2004

Vibe report

Quindi Meeting Capture

SenseCam

SenseCam around Cambridge

MyLifeBits Software

Everything goes in a database

MyLIfeBits need all the features of a databaseMyLIfeBits need all the features of a database(Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, (Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, Replication)Replication)

If we didn’t use one, we’ll eventually create one!If we didn’t use one, we’ll eventually create one! Files as blobs; sync with file system for legacy appsFiles as blobs; sync with file system for legacy apps We are part of Jim Gray’s Bay Area Research LabWe are part of Jim Gray’s Bay Area Research Lab

SQLSQL

MyLifeBits Software

MyLifeBits store

database

Voice Voice annotation annotation tooltool

Telephone Telephone capture toolcapture tool

TV capture TV capture tooltool

TV EPG TV EPG download tooldownload tool

Radio Radio capture capture & EPG& EPG

PocketPC PocketPC transfer transfer tooltool

PocketRadio PocketRadio playerplayer

Import filesImport files

MyLifeBits MyLifeBits ShellShell

Browser Browser tooltool

InternetInternet

IM captureIM capture

GPS import & GPS import & Map displayMap display

SenseCamSenseCam

Screen saverScreen saver

Text Text annotation annotation tooltool

MAPI MAPI interfaceinterface

Legacy Legacy email clientemail client

OutlookOutlookinterfaceinterface

files

Legacy Legacy applicationsapplications

VIBEVIBElogginglogging

RoomRoomCaptureCapture

IMG_Images2

PK,FK1,I1,I2,I3 item_id

Width HeightI1 Date Taken Camera Make Camera ModelI2 LatitudeI3 Longitude Elevation

Outlook_CalendarItems2

PK,FK1 item_id

Subject Start End Description Location Creation Time Modified

Common ground with WinFS:Items, Links & Meta-data

AnnotatesAnnotates

Caller in Phone CallCaller in Phone Call

Photo of EventPhoto of Event TAPI_PhoneCalls2

PK,FK1,I1 item_id

Phone Call Type CID CID Name CID #I1 Begin End Seconds Connected Ended Roaming Trimmed Recorded Transcript

PhotoFinder - Shneiderman and Kang

The Shape & Size of Gordon’s LifeBits

MyLifeBits10/31/2005

242K items 110 GB

by number of Items.

MyLifeBits10/31/05

110 GB242 K items

By Size (GB)

Bell Growth: 1GB/month

=1.1 TB/lifetime

Size (MB) by TypeSize (MB) by Type

1

10

100

1000

10000

1895 1905 1915 1925 1935 1945 1955 1965 1975 1985 1995 2005

Year Mpix Manufacturer1997 .25 Ricoh1999 1 Kodak2001 2 Canon2002 3 Sony2003 4 Sony2005 5 Panasonic

Year Mpix Manufacturer1997 .25 Ricoh1999 1 Kodak2001 2 Canon2002 3 Sony2003 4 Sony2005 5 Panasonic

15,000 photos

Monthly & Lifetime Storage Use

Item Daily number

Total* MB|GB

Month|Life

1 MB Books|reports 0.1 3

5KB Emails 100 13

100 KB Image scans 5 13

0.4 MB Photos 10 100

75 KB Web pages|docs 100 188

100 MB Music 0.1 250

1 KB/s Listened audio, speech 40,000 1,000

50 KB Daily photos 1,000 1,250

2 GB/hr TV 4 200,000

Observations about use(rs)

1.1. Cell phone sized device (CPSD) will be the platform!Cell phone sized device (CPSD) will be the platform!2.2. On Applications… think about CPSD as the platform and contextOn Applications… think about CPSD as the platform and context

Search is the “killer app” pretty much as Bush described. Search is the “killer app” pretty much as Bush described. Screen savers “memory refreshers” also provide ambienceScreen savers “memory refreshers” also provide ambience Where did my day to?Where did my day to?

3.3. Users are unwilling to spend time managing their computers or data. Users are unwilling to spend time managing their computers or data. Meta-data, classification, etc. must be automaticMeta-data, classification, etc. must be automatic User-input meta-data e.g. Dublin Core – naïve’ Librarian’s dream.User-input meta-data e.g. Dublin Core – naïve’ Librarian’s dream. We have nice scheme for classification using facets. It requires work.We have nice scheme for classification using facets. It requires work.

4.4. Time is the most important meta-data. Photos: place (GPS), subject.Time is the most important meta-data. Photos: place (GPS), subject.5.5. Folders are a good and bad idea. Folders are a good and bad idea.

Most users don’t know what they are or how they workMost users don’t know what they are or how they work If used, over time, they become useless: too many, miss-file, etc.If used, over time, they become useless: too many, miss-file, etc.

6.6. User User shouldshould put “every” information fragment into the system. e.g., to put “every” information fragment into the system. e.g., to dos, call backs, business cards numbers, attention events. It pays. dos, call backs, business cards numbers, attention events. It pays.

7.7. Same information in multiple places always becomes obsolete.Same information in multiple places always becomes obsolete.

Capturing Everything:

Phone calls in context of cell phone as a Phone calls in context of cell phone as a platform for communication and captureplatform for communication and capture

Formal Meetings Formal Meetings RoomsRoomsEverything in daily lifeEverything in daily lifePersonal health and medical monitoringPersonal health and medical monitoringMemex for scientists and engineersMemex for scientists and engineers

BodyMedia Output

Real time health monitoringPolysomnogram for sleep apnea.

Microsoft Research SensCam II

Sensors:Sensors:VGA camera w/ wide-angle lensVGA camera w/ wide-angle lens light level in R,G,B and whitelight level in R,G,B and whiteambient temperatureambient temperaturepassive infrared for person detectionpassive infrared for person detectionaccelerometersaccelerometers three, programmable buttons, LEDs, sounderthree, programmable buttons, LEDs, sounderaudio level & audio recordingaudio level & audio recordingUSB 2 and SD memory. 1-2 K photos/dayUSB 2 and SD memory. 1-2 K photos/dayNot GPSNot GPS

SenseCam University Grant ProgramMSFT supplies money, software, SenseCams

Memex vision: Notebook for engineers & scientistsMemex vision: Notebook for engineers & scientists Medical & health: observations & memory recall, Medical & health: observations & memory recall,

including diet and exerciseincluding diet and exercise Education: How do people learn? Education: How do people learn?

Help me learn/remember!Help me learn/remember! Tourist e.g. museum experienceTourist e.g. museum experience PlumbingPlumbing

SecuritySecurity Filtering many images, voice & location annotationFiltering many images, voice & location annotation

More real time experience capture

Real time medical & health monitoringReal time medical & health monitoringMIT. Deb Roy home capture to understan MIT. Deb Roy home capture to understan

how his children learnhow his children learnU. of Tokyo. Ubiquitous homeU. of Tokyo. Ubiquitous homeColumbia U. Voice & sound record & profileColumbia U. Voice & sound record & profileMIT. iDat. Electronic lab that records MIT. iDat. Electronic lab that records

everything into your notebookeverything into your notebook

Experience Retrieval in a Ubiquitous Home(chamds, byon, yamasaki, aizawa)@hal.k.u-tokyo.ac.jp

Experience Retrieval in a Ubiquitious Home

MIT iDAT Project aka notebook

Samsung challenge Going beyond plain old photography and videographyGoing beyond plain old photography and videography

Print, view, and file in scrapbook or shoeboxPrint, view, and file in scrapbook or shoebox Digitized bits offers worldwide sharing and easy sharingDigitized bits offers worldwide sharing and easy sharing Screensaver is useful, but is it a killer app?Screensaver is useful, but is it a killer app?

The cell phone sized device (CPSD)… one deviceThe cell phone sized device (CPSD)… one device Next generation platformNext generation platform Phones and messaging e.g. sms, mail, web, iM, bloggingPhones and messaging e.g. sms, mail, web, iM, blogging Audio, photo, video record and viewing (incl. broadcast)Audio, photo, video record and viewing (incl. broadcast) Within 5 years and with supplemental devices, will take on Within 5 years and with supplemental devices, will take on

the PCthe PC

Capture, storage, retrieval, and displayChallenge putting them together

Capture ….Capture …. Cell phone sized devices (CPSD). The “killer app”!!Cell phone sized devices (CPSD). The “killer app”!! Consumer… photo, video, audio… experienceConsumer… photo, video, audio… experience ProfessionalProfessional

StorageStorage CaptureCapture ArchivalArchival

Retrieval = f(use). Archive… ambienceRetrieval = f(use). Archive… ambience DisplayDisplay

Personal: Cell phonePersonal: Cell phone PCPC WallWall

www.MyLifeBits.com

BONUS SLIDES

Challenges Data-typesData-types

Quantity expanding i.e. info explosionQuantity expanding i.e. info explosion New capabilities e.g. real time create new data-typesNew capabilities e.g. real time create new data-types Meta-data to increase value & provide pivots Meta-data to increase value & provide pivots

Going beyond a PC to a distributed environmentGoing beyond a PC to a distributed environment

Network environment, including media centerNetwork environment, including media center Into the cloud. Especially important for social aspectsInto the cloud. Especially important for social aspects Periphery… smart buildings, objects, Periphery… smart buildings, objects, Backup, migration, and caching for beyond a TerabyteBackup, migration, and caching for beyond a Terabyte Expanding network: PC > LANs > web > p2p(eer)Expanding network: PC > LANs > web > p2p(eer)

Schema sharing among disparate systemsSchema sharing among disparate systems CARPE (real time data capture)CARPE (real time data capture)

Rooms, phone calls, SenseCam, Health transducers, etc.Rooms, phone calls, SenseCam, Health transducers, etc. Security, privacy, forgetfulness, deniability, etc.Security, privacy, forgetfulness, deniability, etc.

More challenges

Dear Appy: Monitoring and automatic migration of files that Dear Appy: Monitoring and automatic migration of files that are unlikely to be understood on future platforms as well as are unlikely to be understood on future platforms as well as platform migration.platform migration.

Get What I Need: GWIN…Endless, but evolutionary Get What I Need: GWIN…Endless, but evolutionary improvements in search: misspellings, stemming synonyms improvements in search: misspellings, stemming synonyms

Endless frontier of schema and extensions to them for new Endless frontier of schema and extensions to them for new applications e.g. making org charts, family relationships.applications e.g. making org charts, family relationships.

CARPE… a whole new game!CARPE… a whole new game! Versioning is essentialVersioning is essential Scaling.. We don’t know what happens at a TerabyteScaling.. We don’t know what happens at a Terabyte What can, should be, or will be in the cloud? Books… videosWhat can, should be, or will be in the cloud? Books… videos Will we be allowed to use such systems? Copyright laws vary: Will we be allowed to use such systems? Copyright laws vary:

E.g. ripping CDs, copy of anything, photos, conversationsE.g. ripping CDs, copy of anything, photos, conversations

The “dear appy” problemDear Appy, Dear Appy,

How committed are you?How committed are you?Please come back to me.Please come back to me.Forever yours truly, Forever yours truly, Lost and forgotten dataLost and forgotten data

Who’s responsible?Who’s responsible?Media or 8 track cassette, 8” floppyMedia or 8 track cassette, 8” floppyEvolving platform, file, and databaseEvolving platform, file, and databaseEvolving, incompatible standards & formats for Evolving, incompatible standards & formats for legacy data that disregard ancestorslegacy data that disregard ancestorsEvolving and/or disappearing appsEvolving and/or disappearing apps   

Is Cyberspace a safe store?

Don’t your physical records e.g. paper last forever?

What about information on your CDs, tapes, hard drives, solid state devices?

Automatic classification problem XML on bills and imported content… transactionsXML on bills and imported content… transactions We need to download classifications rather than We need to download classifications rather than

build thembuild them Definitions & synonyms should help find what I wantDefinitions & synonyms should help find what I want

Today it is too expensive to manually classify Today it is too expensive to manually classify scanned paper. E.g. “right time” meta-data is critical!scanned paper. E.g. “right time” meta-data is critical!

We hope “the system” can classify papers and other We hope “the system” can classify papers and other documents e.g. bills. Ideally, build Dublin Coredocuments e.g. bills. Ideally, build Dublin Core

In 10 years we need all documents to appear In 10 years we need all documents to appear electronically & classified electronically & classified with a little help from mewith a little help from me