LIFELOGGING A NEW ERA OF PERSONAL DATA

Dr. Cathal Gurrin (@cathal) lifelogger - researcher - educator

Dublin City University & Insight Centre for Data AnalyticsBiohackers Summit 2015

24th September 2015

From Cave Paintings…

… to Diaries

Technology allows us to record our lives in previously unimaginable detail…

… but why ?

Using mobile/wearable devices and information loggers to automatically record everything you see, hear, learn and experience. Creates a complete and accurate record of an individual - a Lifelog.

Challenge is to extract value from this new data.

Lifelogging

In the era of lifelogging, you will be able to summon up any memory or life experience…

It will change the way we work and learn, improve our health, change relationships…

It will change what it means to be human, and it is happening now. In fact, it is inevitable…

MyLifeBits & Sensecam

Gordon Bell (Microsoft)

2004

Sense and analyse factors of interest through numbers to gain

knowledge

Using knowledge for self-improvement

through experimentation Digitise as much as you can of

life experience… for many reasons, mostly unknown…

LifeloggingQuantifiedSelf

Biohacking

Positioning my Research

Lifelogging cangenerate thousands of

images per day, hours of audio/video

and tens of thousandsof sensor readings,

biometrics,EEG, communications,

interactions…

The challenge is to automatically analyse this data and make it useful for the individual.

Quantified Self

Enhanced Knowledge

Power to Change

Performance Enhancement

Data for Empowerment

New Insights

Population-wide studies

Healthcare Enhancement

Enhancing Human Memory

Upgraded Recall

Assistive Technologies

Enhanced Memory

New Interactions

Rich Sharing

Data Partners/Carers

Social Enhancements

Why? To provide knowledge to empower…

Quantified-Self Memory

Quantified Self Analytics

BASIS Tracker Watch

Memory Enhancement

RECALL/RETRIEVAL

REFLECTION REMINISCENCE

Quantified-Self Analytics with Limitless

REFLECTION

A Search Engine for Life Experience. Never Forget.

RECALL/RETRIEVAL

Reliving Past Memories for Personal Uses or Sharing.

REMINISCENCE

The aim is to develop prototype memory upgrading software. An assistive technology that experiences what you experience and is always on and does

not need any user input, except queries.

Sensory(Memory(

Short-term(Memory(

Long-term(Memory(

Musical?( Explicit((conscious)(

Declara<ve((events(and(facts)(

Episodic((events(and(experiences)(

Seman<c((concepts(&(facts)(

Implicit((unconscious)(

Procedural(That means looking at human memory, and how it works…

Episodic Memory, Query Mechanisms, etc…

And develop targeted applications… such as LoggerMan.org to build various memory models

!

!

Or large-scale tools to understand individuals and their activities… even on a societal level

Automatically annotate, enrich, link and store for future search, retrieval and

access.

IndexPervasive access to

support Reminiscing,

Reflection and Retrieval of Experiences

InteractAutomatically sense using a small set of

wearable and informational

sensors.

SenseAutomatically

generate meaningful units of retrieval by modeling human

memory.

Segment

Four Core Components are Required to build a Lifelogging Platform

There are a lot of research challenges here, at every step.

But they are all needed to develop a lifelogging platform technology.

SENSE(HARDWARE AND SOFTWARE)

There is no sensor that can record everything experience, in multiple dimensions

Autographer Panasonic 4K Google Glass

Moves App Google Fit BASIS Watch Strava

RescueTime LoggerMan Camlapse

MyTracks OpenPaths

CameraPhoneInstagramMedia Lifelogging

Activity

Information Access

SMS Backup

CallRecorderPro

VoiceRecorder

WebServices

SwarmLocation

23&Me EEG DietOthers

Health

Last.fm

NarrativeClip

Secure Personal Lifelog

The big challenge is Visual Lifelogging

It is not conventional photos, just data, 2,000 - 5,000 per day! Too many for an individual to analyse

My Lifelog in Numbers

70+ Papers and 12 first generation prototypes

10 Years of location log, with millions of

GPS points

80 Million: heartbeats,

with GSR and activity

1 Year of computer interactions

(mouse, keyboard)

9 Years

of lifelog, since 2006

16.5 Million wearable camera images

About 1TB per year

SEGMENT, INDEX AND INTERACT

Segmentation of raw data into units such as events or moments.

These can be enriched automatically with metadata,

increasing their value.

Events are analogous to our episodic

memory

Event Detection

Sharing

Search

API coming in the next few months

Narrative Clip

Event Segmentation - Concept Filtering

EyeAware Platform

Automatic Event Detection with Linkage & Browsing

Like all multimedia data, we began by browsing, but there is too much data,

much repetition. We need search (Googlisation).

2.5 year study into locating important items: Increase from 25% to 75% success in 1/10th the time when

searching not browsing.

Searching is based on data analytics and machine/deep learning to ‘understand’ the sensor data.Segmentation

Find the unit of retrieval for many use-cases… there is no one correct unit

EnrichmentAutomatically turn raw

sensor data into meaningful information

Search EngineTo index the data

InterfacesSupporting Applications

Aiden Doherty, DCU, office setting, conversation, indoor,

discussing CHI paper.

“On Sept 23rd, I was in DCU discussing the CHI paper with Aiden at his desk”

The challenge is to automatically extract knowledgefrom the lifelog data to supportrecall/retrieval, reminiscence

and reflection.

Raw$Sensors$

What$doing$

What$Environment$

Movement$• Ac8vity$• Energy$

Where$Who$is$there$

When$$

Why$

“Shopping for a coat lastTuesday in Helsinki”

Enrich Semantics by Applying Data Analytics

Integrating AI - Deep Learning To understand what the user is seeing and doing

Kahneman et al. A survey method for characterizing daily life experience: The day reconstruction method. Science, 306(5702):1776–1780, 2004.

1

2

3

4

Intimate Relations

5

6

Socialising

Relaxing

Pray/Worship/Meditate

Eating

Exercising

7 Watching TV

8 Shopping

9

10

11

12

Preparing Food

13

14

On the Phone

Napping

Taking Care of Children

Computer/Internet

Housework

15 Working

16 Commuting

Recognising Life Activities

Summarisation REFLECTION

Search EngineRECALL/RETRIEVAL

BrowsingREMINISCENCE

Considering the Use-Cases and

Developing Applications

Amended from: Abigail J. Sellen and Steve Whittaker. 2010. Beyond total capture: a constructive critique of lifelogging. Commun. ACM 53, 5 (May 2010)

Recall / Retrieval - Prototype Search Engines

Reflecting on Life at a Glance - Colour of Life

Reflection for Enhancing Self Awareness

Reminiscence Supporting Digital Memory

How does this relate to BioHacking?There is a lot of R&D still to do… no consideration of UI

No adequate lifelogging device or software yet

Privacy (lifelogging looks out, QS looks in)Trust for sharing and storage

Get data now, you can not get data retrospectivelyNew tools and software will extract value later

Some Final Thoughts

Privacy Awareness - Automated Negative Face Blurringwith real-time Policy-driven Access Restrictions

Privacy

THANK YOUCathal Gurrin

@cathal

Any Questions? Interested in working with us, let me know…

“ L i f e l o g g i n g - Personal Big Data” from the Foundation a n d T r e n d s i n Information Retrieval s e r i e s . F r e e d o w n l o a d , a s k Google.