Jan Hendrik Hammer, Fraunhofer, KIT, Eyetracking and Gaze Analysis

RESEARCH PUI@IOSB

Eye Tracking, Gaze Analysis and Gaze-Based InteractionJan Hendrik HammerKarlsruhe Institute of Technology, Germany

Evaluating Use and ImpactWorkshop of theScottish Network on Digital Cultural Resources Evaluation,Glasgow, United Kingdom,March 31, 2016

Nr.

Fraunhofer IOSB

OutlineApplications for eye trackingMobile and stationary eye trackingFrom eye tracking to gaze analysisWhat can we get from eye tracking? An example.Further options for eye tracking

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Applications for Eye TrackingAssistive technology as interaction modality for people with disabilitiesUser experience and interaction (interface design)Marketing and consumer researchInteraction with productsPerception of advertisementCompare viewing behaviour of experts and novicesDuring sportsDuring the perception of artworks...

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.

Mobile and Stationary Devices

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Stationary DevicesSmall headboxOnly for gaze on displays (mostly only one)Prices:Device: 80 .. 40,000+ Software:SDKs for free programmingAnalysis software: 800 .. 8,000+ Tobii Pro TX300 (www.tobiipro.com)SMI RED250 (www.smivision.com)Smart eye pro (http://smarteye.se)

SMI RED250Mobile (www.smivision.com)Tobii Pro X3-120 (www.tobiipro.com)Aurora (http://smarteye.se)Gazepoint GP3 (www.gazept.com)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Eyetrackers.net/en/buy

SMI:iView X READ 18900 ETG 2 9.900 RED m 14.900 Experiment Suit 360: 4.900

Gazepoint (http://www.gazept.com/shop/):GP3 Eye Tracker: $ 495Gazepoint Analysis Professional Edition Software $995.00

Tobii:Tobii Pro Analystics SDK: free and for all their screen based eye trackersTobii Pro Studio: Basic: 4700 $, Profession: 8900$, Enterprise: 12900$Tobii X60 29,900 $Tobii Pro X3-120: ca. 100

Smart eye pro:Smart eye Aurora:

Mobile DevicesBinocular eye trackingBelow 100 gHardware and live viewing:1,800-10,000 Gaze analysisSDKs for free or software(~8,000 )Gaze registration further costsSemi-automatic: gaze annotationsFully-automatic: pose estimation needed

Dikablis Professional(www.ergoneers.com)Tobii Pro Glasses 2(www.tobiipro.com)SMI ETG 2 Professional(www.eyetracking-glasses.com)Pupil labs eye tracker (https://pupil-labs.com)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Pubil: 2300

Tobii pro glasses 2 live vieweing: 15000 $Tobii pro glasses 2 incl. Analysis software 25000 $

ETG Analysis Pro (60 Hz Glasses, Data Recorder, Analysis Software): ~ 30,000 $

Estimation of 3D Eye Ball PositionPose estimation of scene cameraInside-out trackingScene cameraMarker trackingMarkers distributed over the sceneDistraction of visual attentionOutside-in trackingExternal cameras + infrared light sourcesInfrared marker attached to deviceHigh accuracyCosts: 4,000 .. 150,000+

Eye tracker e.g. from Pupil Labs (https://pupil-labs.com)Marker attached in sceneInfrared markers attached to e.g. SMI ETG 2(http://www.eyetracking-glasses.com)

Camera tracking system e.g. from ART (http://www.ar-tracking.com)

Dikablis Professional(www.ergoneers.com)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Outside tracking: Costs: 5,000 .. 200,000 $+

7

From Eye Tracking to Gaze Analysis

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Gaze Analysis Overview

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Eye tracking solves the question of where your eyes are directed to in the given scene. This is NOT what you are looking at because looking involves some kind processing in our brains. Eye tracking is the processing of the raw data from the used device to reconstruct the line of sight for gaze point computation.What you are looking at is more part of the gaze movement computation. Our gaze path consists of different gaze movements. The two most important ones, I want you to remember are fixations and saccades. Fixations are periods during which our gaze remains still at some point and we process what we see. Fixations range between 60 ms and somewhere below 2 seconds. Between fixations, the points we look at, our gaze jumps and these ballistic movements are called saccades.Using these gaze path we can then analyse the gaze using different gaze metrics and AOIs, so called areas of interest, to e.g. infer the visually most relevant objects of a scene.

Gaze Analysis Overview


Nodal point of eye relative to scene cameraViewing direction

view cone

Eye Tracking Data for Line of Sight ReconstructionGuestrin, E. D. D. , Eizenman, M. (2006) General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE transactions on bio-medical engineering, 53, 1124-1133

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Geometrical model for computation of the viewing direction

Estimation of 3D Eye Ball PositionPose estimation of scene cameraInside-out trackingScene cameraMarker trackingMarkers distributed over the sceneDistraction of visual attentionOutside-in trackingExternal cameras + infrared light sourcesInfrared marker attached to deviceHigh accuracyCosts: 4,000 .. 150,000+

Eye tracker e.g. from Pupil Labs (https://pupil-labs.com)Marker attached in sceneInfrared markers attached to e.g. SMI ETG 2(http://www.eyetracking-glasses.com)

Camera tracking system e.g. from ART (http://www.ar-tracking.com)

Dikablis Professional(www.ergoneers.com)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Outside tracking: Costs: 5,000 .. 200,000 +

12

3D Modell of EnvironmentManual creationVery time consumingOnly static scenes

Valencian Kitchen, National Museum of Decorative Arts (Madrid, Ministry of Culture, Spain)The Laboratory of Lavoisier, Muse des arts et mtiers, (Paris, France)

SmartControlRoom, Fraunhofer IOSB (Karlsruhe, Germany)


13

Line-of-Sight Reconstruction3D gaze point computation = intersection line-of-sight with 3D world

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Gaze analysis overview


3D Scanpath VisualizationGazepoints (violet):7221 gaze points, 4 min 48 s8820 gaze points, 5 min 52 s


3D Scanpath Visualization7221 gaze points, 4 min 48 s8820 gaze points, 5 min 52 s


3D Scanpath VisualizationFixations and saccades:


Gaze analysis overview


Areas of Interest (AOIs)

Definition of AOIsVisualization of a hit AOI

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Areas of Interest (AOIs)

Definition of AOIsVisualization of a hit AOI

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Definition of AOIs in Valencian KitchenHit AOIs are highlighted in realtime

Entry time + 1st fixation numberDuration of 1st fixationNumber of fixations on AOICumulative fixation timeDwell timeCumulative fixation time + cumulative duration of saccades on AOIRevisitsNumber of saccades entering the AOI minus 1AOI-based Metrics

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Metrics for attractiveness of an AOI

What can we get from eye tracking? An example.

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Two types of experimentsT1: Freely viewing artworksHow do subjects look at the scene?What are the most attractive areas?T2: Freely viewing artworks while listening to audio guideDo people follow the told stories?Experiments in the EU-Project ARtSENSE


Different Scanning BehaviorNormalized heat map (green to red)

Freely viewing (5 minutes)Listening to audio guide (6.5 minutes)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Different Scanning Behavior

Food and kitchen tools

Important persons and animals (part of audio guide stories)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Cumulative Fixation Time (CFT)

Food and kitchen tools

Important persons and animals (part of audio guide stories)

CFT = 45 seconds (41 %)CFT = 64 seconds (59 %)CFT = 30 seconds (14%)CFT = 181 seconds (86 %)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Scanpath ComparisonScanpath 1: Freely viewing

Scanpath 2: Listening to audio guide


3D Scanpath VisualizationScanpath2: Listening to audio guide


Live Computation of Visually Most Relevant Objectsin Small Time Windows

Time window [t - 4 s, t]

Time window [t - 3 s, t + 1 s]

Basis for recommendations in AR Glasses

tablet1tablet2tablet3servant1servant2

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Gaze-Based Interaction

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Stationary Gaze Key Press Video stream selection using gaze key press

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Mobile Gaze Key Press First tests

Selection by pointing

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Mobile Gaze Key Press First tests

Pointing (26.5 sec)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Further Interesting StuffStationary eye tracking via webcam (www.eyezag.de)No eye tracker neededSubjects can participate from all over the world

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Duke of Wellington

Further Interesting StuffEye tracking in virtual realityYou can walk around and the immersion is great!Cheap pose estimation for 3D eye trackingModeling of 3D world neededEye tracking hardware and analysis software not available for all VR devices but in probably in the future

HTC Vive(www.htcvive.com)

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.SummaryWhat do you want the people to look at?Images on display stationary eye trackingLarge volumes mobile eye trackingWorkflow: Eye tracking gaze movement computation gaze analysisGaze-based interaction (gaze key press)Alternatives to usual eye tracking and eye tracking in VR

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Thank you for your attention!

Questions?

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Contribution1st fully automated real-time capable 3D gaze analysis for mobile applicationsPublication: Jan Hendrik Hammer, Michael Maurus, Jrgen Beyerer. Real-time 3D gaze analysis in mobile applications. In: Proceedings of the 2013 Conference on Eye Tracking South Africa, ETSA '13, S. 75-78, ACM, August 2013.

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Realistic Heatmap Generation in 3D EnvironmentsVisual acuity decreases with deviation from visual axis Projected Gaussian

Projected GaussianOcclusion test

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Contribution1st method for realistic heatmap visualization of gaze data in 3D environmentsPublication: Michael Maurus, Jan Hendrik Hammer, Jrgen Beyerer. Realistic Heatmap Visualization for Interactive Analysis of 3D Gaze Data. In: Proceedings of the Symposium on Eye Tracking Research and Applications, S. 295--298, ACM, Mrz 2014.




Real-Time 3D Gaze Analysis inMobile ApplicationsNr.Motivation: Adaptive AR Museum Guide

1. Freely viewing artwork3. Recommendation4. Gesture interaction

2. Detection of visually most relevant objects

Real-Time 3D Gaze Analysis inMobile ApplicationsNr.audio guides tablets -> further informationHMD -> figure out where people lookScanpath -> tartletsSystem determines tartlets as visually most relevant objects -> recommendation systemQuestion -> Explicit interaction by hand gesturesToday - implicit gaze interaction

Pupil and glint detection using eye trackers with infrared illuminationSimple example method for pupil detection1:

Modern mobile devicesuse several IR-LEDs faster calibration

Eye Trackingglintpupil

1. Input image3. Noise removal2. Thresholding5. Compactness comparison4. Size comparison1: Goni, S., Echeto, J., Villanueva, A., Cabeza, R. (2004) Robust algorithm for pupil-glint vector detection in a video-oculography eyetracking system. Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 4 - Volume 04, IEEE Computer Society, 941-944


Scanpath: sequence of gaze points or fixations (and saccades)Scanpath metrics:Statistics about saccade duration/amplitude fatigue or stressFixation/saccade ratio degree of observing detailsScanpath length extent of spatial inspectionScanpath duration extent of processing complexityScanpath analysis

[GOLDBERG, J. , KOTVAL, X. P. (1999) Computer interfaceevaluation using eye movements: methods and constructs. In International Journal of Industrial Ergonomics , 24, 631645]


Metrics using areas of interest (AOIs)Time to 1st fixation of an AOI attractivenessPost-target fixations: Number of fixations on AOI after 1st fixation attractivenessFixation duration on AOIOn-AOI-fixations/all-AOI-fixations ratio Degree of processing complexity of AOISpatial density directness of inspection

[GOLDBERG, J. , KOTVAL, X. P. (1999) Computer interface evaluation using eye movements: methods and constructs. In International Journal of Industrial Ergonomics, 24, 631645]Grid of AOIs


Other scan path metricsArea of convex hull extent of inspectionAOIs covered by convex hullCumulative relative angle Directness of gaze[GOLDBERG, J. , KOTVAL, X. P. (1999) Computer interfaceevaluation using eye movements: methods and constructs. In International Journal of Industrial Ergonomics , 24, 631645]


Transition matricesContain number of transitions from AOI i to AOI jMetrics:Transition density inspection behaviourNumber of transitionsinto an AOI amount of information content of AOI

[GOLDBERG, J. , KOTVAL, X. P. (1999) Computer interfaceevaluation using eye movements: methods and constructs. In International Journal of Industrial Ergonomics , 24, 631645]
