Smart Photo Selection: Interpret Gaze as Personal Interest

1

Smart Photo Selection: Interpret Gaze as Personal Interest

Tina Walber1, Ansgar Scherp2,3, Steffen Staab1

1 Institute WeST, University of Koblenz, Germany2 Kiel University, Germany3 Leibniz Information Center for Economics, Kiel, Germany

Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 2

Managment of Digital Photos● Its a mess!● We take a lot of photos● Manually selecting photos is cumbersome● Like to have photo selections for

– Sharing photos online

– Creating photo products like photo books

– Creating presentations


State of the Art: Automatic Creation of Photo Selections

● Content-based approaches– Analysis of low-level features

● Context-based approaches– Analysis of context information

● What about individual aestetics, personal preferences, user interests, ….?


Interpret Gaze as Personal Interest

● Gaze delivers information on user's interest● Useful for creating individual photo selections?

● Principal approach – Merely observe what users are doing anyway

– Do not ask to perfom additional tasks


● Starting from $99


Research Questions

1. Is there a need for individual photo selections?

2. Does a gaze-based selection outperform selections based on content and context analysis when comparing to those created manually?

3. Does the personal interest in a viewed photo set have an impact on the obtained selection results?


Experiment Setup

Photo Viewing

Task: „get an overview“

Step 1

Recording of the eye tracking data


Photo Viewing

32 pages with 9 photos each


Experiment Setup

Photo Viewing

Task: „get an overview“

Step 1

Photo Selection

Task: „select photos for your private photo

collection“

Step 2

Recording of the eye tracking data

Creation of Ground Truth

Sm


Manual Selection Creator:LibreOffice 3.5 LanguageLevel:2


Collection CA Collection CB 162 photos 126 photos

Experiment Data Set C = CA CB

∩

Two Data Sets and Two User Groups

Institute A Institute B

Home collectionHome collectionForeign collection

● Taken during social events of the research institutes


Participants

● 33 participants (12 of them female)● 21 associated to Institute A, 12 to Institute B● Aged between 25 and 62 (Ø 33.5 ± 9.57)● 20 graduate students, 4 postdocs,

9 other professions


Overview Analysis and Evaluation

Se

Collection C

GazeBased

Selection

Calculation of Precision

P

ManualSelection

Content andContext Based

Selection

Sb+e

Sb

Ground Truth

Sm


Baseline Measures

# Name Description

1 concentrationTime

Photo was taken with other photos in a short period of time

2 sharpness Sharpness score from related work

3 numberOfFaces Number of faces

4 faceGaussian Size and position of faces

5 personsPopularity

Popularity of the depicted persons

6 faceArea Areas in pixels covered by faces

Selection of photos based on:

Calculated for each photo


Eye Tracking Data

● Fixations and saccades● Analysed gaze data with eye tracking measures

Creator:LibreOffice 3.5 LanguageLevel:2

● Viewing duration / page: M = 12.6 s

● Number of fixations / photo: M = 3.25


Eye Tracking Measures# Name Description

7 fixated Was the photo was fixated?





8 fixationCount Counts the number of fixations






9 fixationDuration Sum of duration of all fixations Creator:LibreOffice 3.5 LanguageLevel:2





9 fixationDuration Sum of duration of all fixations

10 firstFixationDuration Duration of the first fixation

11 lastFixationDuration Duration of the last fixation

12 avgFixationDuration Average fixation duration

13 maxVisitDuration Maximum visit length

14 meanVisitDuration Average visit length

15 visitCount Number of visits

16 saccLength Mean length of the saccades

17 pupilMax Maximum pupil diameter

18 pupilMaxChange Maximum pupil diameter change

19 pupilAvg Average pupil diameter


Combination of Measures

● Using a model learned from logistic regression● Assigns each image a probability of being

selected● 30 random splits for training and test data


1. Is there a need for individual photo selections?

1 21 41 61 81 1011211411611812012212412612810

5

10

15

20

25

30Photo with the highest number of selections

Photos with no selections

Photos in data set C

10

40

70

100

130

160

190

220

250

280

Sel

ectio

n F

requ

ency

● Manually created photo selections are diverse


2. Evaluation of the Photo Selections

Pre

cisi

on P

Sb Sb+e Se

**

Random Selection

P = 0.428P = 0.365 P = 0.426

● Improvement of 17% over baseline


3. Impact of personal interest?

Pre

cisi

on P

Results for Sb+e

Foreign Collection Home Collection

P = 0.446P = 0.404

*


Conclusion● Photo selection behavior is individual● Gaze helps capture personal preferences● Results are better for photos with personal interest

● Might work even better for real personal photos● Potential application in photo book authoring

Thank you for your attention!

Science

Smart Photo Selection: Interpret Gaze as Personal Interest