CREATE DAV talk at York U

The cognitive science of visual analytics

"The science of analytical reasoning facilitated by interactive visual interfaces"

Brian Fisher Simon Fraser University

School of Interactive Arts & Technology

UBC Media & Graphics Interdisciplinary Centre

Engineer or psych scientist?

• Ph.D in Experimental Psych

• Postdoc w Cognitive

Science society founder &

president

• Psychonomics Fellow

• VIS-related symposia at

Cogsci, & APS, papers on

cogsci of interaction

• Fuzzy-logic/Bayes models

• Postdoc funded by Inst. for

Robotics and AI

• VAST SC, VEC, VACCINE

• Led “Interaction w Information for

Visual Reasoning” at Dagstuhl

Cogsci-based papers at VIS,

CHI, BELIV.

Scientific Programmer at Varian in Palo Alto

Applications• Emergency Management (NSERC, DHS)

• Mobile analytics / sensor analytics • “Virtual EOC” visual analytic environment

• Aircraft Safety, Reliability (Boeing/MITACS,HRL) • “Pair analytics” of complex quant and text data • NextGen Air traffic control

• Gov, econ, & finance (MITACS, NSF, World Bank) • Behavioural economics (portfolios) • Twitter analysis of political sentiment in Brazil

• Healthcare Monitoring & Management (DHS) • Complex data in health w BC Child & Family Research U • Public health w BC Injury Research and Prevention Unit • Personalized Health (LSI, PMI, MYco, NSERC, MITACS)

Theory & method papersEbert, D., Fisher, B. & Isenberg, P. (2014) Interaction with Information for Visual Reasoning. Report from Dagstuhl Seminar 13352. Dagstuhl Reports Schloss Dagstuhl - Leibniz-Zentrum fur Informatik, Dagstuhl Publishing, GermanyGreen, T.M. , Arias-Hernández, R. , Fisher, B. (2014) Individual differences and translational science in the design of human centered visualizations” in Human Centric Visualization Theories, Methodologies and Case Studies. Springer-Verlag, Berlin, Heidelberg, New York. Green, T.M. & Fisher, B. (2011) The Personal Equation of Complex Individual Cognition during Visual Interface Interaction. Human Aspects of Visualization: Revised selected papers, Second IFIP WG 13. 7 Workshop on Human-Computer Interaction and Visualization, HCIV (INTERACT) 2009, Uppsala, Sweden, August 24, 2009, In Lecture Notes in Computer Science 6431, pp 38-57 Springer-Verlag.Meyer J., Thomas, J., Diehl, S., Fisher, B., Keim, D., Laidlaw, D. Miksch S., Mueller, K. Ribarsky, W., Preim, B., & Ynnerman, A. (2010) From Visualization to Visually Enabled Reasoning. In “Scientific Visualization: Advanced Concepts”. vol. 1 pp. 227-245. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany. I978-3-939897-19-4Kurzhals, K., Fisher, B., Burch, M., and Weiskopf, D. (2015) Eye Tracking Evaluation of Visual Analytics . Information Visualization, pp 1-19.Arias-Hernández, R. , Green, T.M., Fisher,B. (2012) From Cognitive Amplifiers to Cognitive Prostheses: Understandings of the Material Basis of Cognition in Visual Analytics. "Computational Picturing," a special issue for Interdisciplinary Science Reviews 37(1).Green, T.M. & Fisher,B. (2012) Impact of personality factors on interface interaction and development of user profiles: Next steps in the Personal Equation of Interaction". Information Visualization. 11(2). 1 -17. Fisher,B., Green, T.M., Arias-Hernández, R. (2011) "Visual Analytics as a Translational Cognitive Science," Topics in Cognitive Science 3,3 609–625.Green, T.M., Ribarsky, W. & Fisher, B. (2009). “Building and applying a human cognition model for visual analytics,” Information Visualization 8(1) 254-262.Franconeri, S. L., Lin, J. Y., Pylyshyn, Z. W., Fisher, B., Enns, J.T. (2008) Evidence against a speed limit in multiple-object tracking. Psychonomic Bulletin & Review 15 (4), 802-808Liu, G. Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005) Multiple Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of Experimental Psychology: Human Perception and Performance. 31(2), Apr 2005, 235-247.Po, B. Fisher, B. Booth, K. (2005) A Two Visual Systems Approach to Understanding Voice and Gestural Interaction. Virtual Reality (Special Issue on Language, Speech, and Gesture) 8, pg. 231-241. Ribarsky, W. & Fisher, B. (2016) The Human-Computer System: Towards an Operational Model for Problem Solving. Proceedings of the 49th Annual Hawaii International Conference on System Sciences. IEEE Digital LibraryKaastra, L.T., Arias-Hernandez, R., & Fisher, B. (2014) Tracking Joint Activities in Visual Analyses . Psychonomics Society Meeting. Long Beach CA.Kaastra, L.T., & Fisher, B. (2014) Field Experiment Methodology for Pair Analytics. BELIV 2014: Beyond Time and Errors - Novel Evaluation Methods for Visualization. Visweek 2014, Paris France.Kurzhals, K., Fisher, B. Burch, M., & Weiskopf, D. (2014) Evaluating Visual Analytics with Eye Tracking. BELIV 2014: Beyond Time and Errors - Novel Evaluation Methods for Visualization. Visweek 2014, Paris France. Kaastra, L.T., Arias-Hernandez, R., & Fisher, B. (2012) Evaluating Analytic Performance. BELIV 2012: Beyond Time and Errors - Novel Evaluation Methods for Visualization. Visweek 2012, Seattle WA

School of Interactive Arts & Technology

My SIAT Design course• How does knowledge inform design?

• Simon’s Sciences of the Artificial

• How can I use knowledge in my design practice? • Schön’s Reflective Practice

• How is technology design special? • Arthur’s Nature of Technology: What it is and how it

evolves.

• How do we design interaction of technology with people? • Norman’s Cognitive Engineering

• How do we build the knowledge we need to do all this?

• Bloom’s Taxonomy of Educational Objectives • Perry’s Scheme of Intellectual and Ethical Development

Science & design of technologies

Simon: “science of the artificial”

Craft

Schön:Reflective practice

add new knowledge

use design science

Arthur: “design language” of

phenomena and how they are controlled

skill in “language”

new vocabulary from sciences

Norman: Cognitive Engineering

for HCI

Arthur on technology design 2009

• Technologies harness phenomena to achieve an objective • In normal engineering, tech improves incrementally • Control of new phenomena leads to dramatic

improvement • Examples are hardware (e.g. aircraft) tech

• Materials — from aluminum to composites • New engine designs based on tech advances

• Arthur says his concepts apply much more broadly to technology design including HCI

Is visual analytics about reasoning (i.e. Cognitive Science) or interactive

interfaces (i.e. Engineering) ?

Visualization origins

• NSF meeting: “Visualization in Scientific Computing” Nov. 1987 Computer Graphics

• First IEEE Visualization conference in 1990

“The purpose of [scientific] computing is insight, not numbers.” Richard Hamming

“Visualization is a method of computing.” Authors of report

Information visualization • 1990 Conference

on diagrammatic reasoning

• 1995 InfoVis Conference • “Information

Visualization: Wings for the Mind” Keynote by Stuart Card

Stuart Card’s 1995 view• Increase the memory & processing

resources available to users • Reduce the search for information by using

visual representations to enhance the detection of patterns

• Engage perceptual inference operations • Use perceptual attention mechanisms for

monitoring • Support manipulation of information

http://www.visual-literacy.org/periodic_table/periodic_table.html

On the Death of Visualization (Lorensen 2004)

Can It Survive Without Customers?

• Visualization, alone, is not a solution.

• Visualization is a critical part of many applications.

• Visualization, the Community, lacks application domain knowledge.

• Visualization has become a commodity.

• Visualization is not having an impact in applications.

Visual analytics

“This science must be built on integrated perceptual and cognitive theories that embrace the dynamic interaction between cognition, perception, and action. It must provide insight on fundamental cognitive concepts such as attention and memory. It must build basic knowledge about the psychological foundations of concepts such as ‘meaning,’ ‘flow,’ ‘confidence,’ and ‘abstraction.’ “ “Illuminating the Path” (IEEE Press)

“The science of analytical reasoning facilitated by interactive visual interfaces”

How are VA systems different?

• Development based on understanding of expert cognition in situ • Informed by current cognitive & social science • Engagement with community of experts

• Interface support for analytical processes-- reasoning, collaboration & interaction • Graphical analog of analytic processes • Emergent cognitive science of visually-enabled

reasoning • Collaborative analysis and coordination of action

in the organization or community

Bridging science & engineering

Controller/display systems in air traffic control

• “Close reading” NextGen ATC fishtank mockup

• Change camera position for better view

• How will global motion affect tracking?

Liu, G. Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005) Multiple Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of Experimental Psychology: Human Perception and Performance. 31(2), Apr 2005, 235-247.

http://www.youtube.com/watch?v=tKJVB4id_TY

FINST theory of spatial

Multiple object tracking

Fit human tracking function

... Then add display motion

Tracking vs object speed

Tracking in warped space

Tracking in warped space

We track in display space

• Retinal speed of targets does not determine performance

• Motion of targets relative to each other does • But only if motion preserves good metric

characteristics of space • Explanation is at the level of a human -

display cognitive system

Boeing Support for Visual Analytics in Canada

• $1.3M (of $5M) from The Boeing Company for VA work, matched by MITACS

• Need to support analytic processes across the company, some defined by “big data” others by complexity of task and integration in the organization (e.g. safety analysis)

• Goal is better analysis • Technology is necessary but not sufficient- need

training, customization, integration with organization

Another approach to building a bridge

Boeing project calls for • Integrative Science

• Study analytic cognition “in the wild” & lab

• Build theory that is predictive & broad in scope

• Cognitive Engineering of applications

• Evidence-based design and evaluation

• Demonstrate value in applications

• Generate new research questions for Cogsci

Social D-Cog in safety analysis• “Pair analytics” sessions

• Student visual analyst & trained domain expert collaborate on analytic task

• Student “drives”, expert “navigates”

• Video session & capture screen

Joint Activity Theory (Clark) • Communication is a Joint Activity, like

playing duet or paddling canoe • Clark’s theory focuses on language:

• Defines kinds of common ground • Formalizes the notion of activity as a “joint

action” • Describes the processes by which common

ground is developed through joint action

• We extend Clark to focus on joint analysis activities that take place in interactive visualization environments

Qualitative video analysis w JAT

• Observe joint attention management by project markers • Multimodal markers — mouse gesture & verbal (content

& prosody) • Vertical and horizontal markers

• We define event structure for analysis based on markers & actions

• Social constraints seen in self-talk incidents, provides effective verbal protocols for protocol analysis methods

Wade Internship•Video recorded and screen captured over 10

Paired Analysis sessions using both Tableau and IN-SPIRE

• Influenced design decisions on: • 777 •P8-A • 787 • 747-8

•Changes to pilot training manual

Wade Internship• Presented work to:

• 787 Engineers • Aviation Safety Community of Practice • Aerodynamics, Performance, Stability and Control

flight data recorder analysis group • Advanced Analytics group • UW Aeronautics and Astronautics students • Boeing Educational Network webcast (400+)

• 500+ people exposed to Visual Analytics, Paired Analysis for Aviation Safety

Visualization work • Design and evaluation of applications using

cognitive science theory & methods

Goal is to use existing theory and build new HCM powerful enough to guide design, evaluation and customization of visualization environments

Implementation examples• CZTalk for Collaborative Knowledge Work • Adaptive Windows Layout Manager • Financial Portfolio Selection with Visual Analytics • Augmenting visual representation of affectively

charged information using sound graphs • Mixed-Initiative Social Media Analytics at the

World Bank: Observations of Citizen Sentiment in Twitter Data to Explore Trust of Political Actors and State Institutions and its Relationship to Social Protest

Child Injury PreventionSamar Al-Hajj

SIAT Ph.D. student

Dr. Samar Al-Hajj Dr. Ian Pike

BC Child Injury Project

• Goal is to develop and evaluate a socio-technical system for making group decisions that are informed by data

• Build & evaluate collaborative VA to facilitate insight generation, knowledge construction and decision-making.

• Don’t forget the benefits of risky play— work with Mariana Brussoni “Risk Reframing”

BCIRPU Injury Indicators• Modified Delphi Method - Indicators to monitor

and evaluate health & well being of infants, children & youth • 34 useful and actionable injury indicators. • Indicator characteristics: (Defined (based on

group definition), Valid/Reliable/Repeatable, Consistent, Sensitive, and Feasible).

• Specify the way indicators are measured and calculated.

Pike, I., Piedt,S., Warda, L., Yanchar, N., Macarthur, C., Babul, S., MacphersonS.K., (2010). Developing injury indicators for Canadian children and youth: a modified-

Delphi approach. Inj Prev 2010;16:154-160

Injury Indicators for Children & Youth

• A suite of injury indicators were developed: Policy Indicators, Risk Factor Indicators, & Outcome indicators

• Iterative consultation and workshops to obtain input on the look and feel of a visual data display dashboard

• Proof of concept completed utilizing BC CHIRPP data (BC Children’s Hospital ER injury data)

• Samar Al-Hajj PhD project to ensure dashboard is based in research evidence

Interactive Visual Analytics Dashboard System

Al-Hajj, S., Pike, I., Riecke, B., & Fisher, B. (2013) Visual Analytics for Public Health: Supporting Knowledge Construction and Decision-Making (full paper). Proceedings of the 46th Annual Hawaii International Conference on System Sciences. IEEE Digital Library

Collaborative Visual Analytics for Public Health: Facilitating Problem Solving and Supporting Decision-Making (2014 SFU Doctoral dissertation)

Analysis of sessions

24

Personalized HealthSamar Al-Hajj

SIAT Ph.D. student

Nadya CalderónPeter Cullis Rob Fraser Solveig Johannssen

45

Personalized Health

Manage your health journey �

Health Journal

Health DataYour personalized reports �

Connect Learn Health JournalBOOK YOUR NEXT TEST THE SCIENCE OF YOU MANAGE YOUR HEALTH JOURNEY

Health DataYOUR PERSONALIZED REPORTS

� Health Data � Health Journal � Connect � Learn

Summary ReportsReport Overview Profile Details Data Maps

��� Health Journal � Connect � Learn � Health Data

z z z

Health Categories ; Disease Associations ; Molecular Measures clear filter; � Search

Body MapData Map

OUTLYING MEASURES

Category: Mineral

Type of Measure: Clinical

Current Value: 0.17

Value Within Range:

Range Over Time:

ASSOCIATIONS

Health Categories: 6

Conditions: 6

SUMMARY

Calcium is the most common metal in many animals. Physiologically, it exists as an ion in the body, although it often combines with phosphorus to form calcium phosphate in the bones and teeth. As a result, calcium is essential for the normal growth and maintenance of bones and teeth. Calcium requirements are greatest during periods of growth,

&+)2(6

Alzheimer’s

Meningitis

Depression

Dizziness

Brain Hemmorrhage

Headaches

900 2,000

2183 ug/ml

CalciumV

Summary ReportsReport Overview Profile Details Data Maps

��� Health Journal � Connect � Learn � Health Data

'

)

*5

7

& (

+

/

12:

9

3

6

4

%

-Calcium

� SearchOptimal MeasuresView: Outlying Measures

OUTLYING MEASURES

Category: Mineral

Type of Measure: Clinical

Current Value: 0.17

Value Within Range:

Range Over Time:

ASSOCIATIONS

Health Categories: 6

Conditions: 6

SUMMARY

Calcium is the most common metal in many animals. Physiologically, it exists as an ion in the body, although it often combines with phosphorus to form calcium phosphate in the bones and teeth. As a result, calcium is essential for the normal growth and maintenance of bones and teeth. Calcium requirements are greatest during periods of growth,

Calcium

&+)2(6

Alzheimer’s

Meningitis

Depression

Dizziness

Brain Hemmorrhage

Headaches

900 2,000

2183 ug/ml

Body MapData Map

V

DOWNLOAD REPORT# SHARE REPORT'

Ovaries are the primary reproductive organs of a female, and like the male testes, ovaries serve a dual purpose: they produce the female gametes (ova or egg) and sex hormones, estrogen and progesterone. Estrogen includes estradiol, estrone, and estriol, but estradiol is the most abundant and is most responsible for estrogenic effects in humans. The ovaries and duct system of the female reproductive system are located within the pelvic cavity. The female’s accessory ducts, from the vicinity of the ovary to the body exterior, are the uterine tubes,

��� Health Journal � Connect � Learn � Health Data

M E AS U R E N A M E C U R R E N T VA L U E VA L U E W I T H I N R A N G E VA L U E OV E R T I M E S TAT U S

Thyroid Stimulating Hormone (TSH)

Triglycerides

Leptin

Serotransferrin

S

U

S

V

R

R

REPORT 06 — JUNE 15, 2016

� SearchShowing 22 Most Critical out of 358 testedAllView: Most Critical

2183 ug/ml900 2,000

900 2,000

900 2,000

900 2,000

B Disease Indicators for 3 conditions

B Disease Indicators for 3 conditions

B Disease Indicators for 3 conditions

B Disease Indicators for 3 conditions

2183 ug/ml

2183 ug/ml

2183 ug/ml

Metabolomic

Microbiomic

Genomic

Clinical

ProteomicSUTRS

PROFILE M O L E C U L A R VA L U E S

REPORT 01January 22, 2015

REPORT 02April 18, 2015

REPORT 03July 19, 2015

REPORT 04December 21, 2015

REPORT 05March 11, 2016

REPORT 06June 15, 2016

Disease Score Over TimeMolecular Profile Overview

62

63

6

6

8

12

4

28

4

4

critical values normal and good values 0-10 10 20-20-30-40-50-60-70

1

10

time

1

10

time

1

10

time

1

10

time

Fixed value over time:4 out of 10

S C O R E OV E R T I M E

62

EndometriosisLast tested: June 15, 2016

Current Score:

RE T U RN TO H E A LTH DATA

Cognition Perceptual

Science Methods

Social Science Methods

Computation and Visualization

Methods

Graphic & Interaction

Design Methods

Analysis “in the wild”

What the field needs

Shneiderman's latest

• Ben Shneiderman (HCIL lab) "New ABCs of Research"

• Call for interdisciplinary tech design: "Applied & Basic Combined"

• "Science, Engineering, & Design" • Simon, Schoen, Arthur... http://

www.cs.umd.edu/hcil/newabcs/

Cogsci society

• Visual analytics ->

Students & PDAs• Dr. Richard Arias-Hernández • Dr. Linda Kaastra • Dr. Samar Al-Hajj • Dr. Tera Marie Green • Nadya Calderón • Andrew Wade

• Applied Visual Analyst (approved!!!) • Perceptually rich human-information discourse • Tech-mediated social cognition & collaboration • Learning model, personal equation

• Analytic Designer • Develop VA technologies • Customize display & interaction for user & task • Consult on organizational roles, communication

• Analytic Researcher • Advance the field

Educational Programs

Applied Visual Analyst

• Graduate Certificate accompanies *any* Masters degree at SFU •MSc computer science -> Build VA systems •MSc forestry -> Use visualization in planning •MSc public health -> Support public health

decisions •MA in history -> Visualize geotemporal

change in ancient Rome

Certificate structure

•Students complete five courses, •IAT 856 Visual Analytics Graduate Seminar •And four additional courses from at least two different Departments : •Technology Courses •Social Systems Courses •Cognitive Processes Courses

Visual Analytics Technology• Courses examine the creation, selection, and

customization of information systems in the student's discipline or area of interest. This includes data processing and modeling as well as interactive visualization. •BUS 709-3 - Managing Information •CMPT 721-3 - Knowledge Representation &

Reasoning •CMPT 767-3 - Visualization • IAT 813 - Artificial Intelligence • IAT 814 - Knowledge Visualization & Communication

Cogs/IAT 885 Visually Enabled Reasoning

• Cognitive theory • Psychology of human reasoning • Modelling causality • Alternative (modal/hybrid) logics

• Analytic practice • Problems and datasets from SEMVAST, Boeing • Teams w paper, IN-SPIRE, Tableau Jigsaw

• Outcomes • Learn reflective analytic practice • Prepare for internship as analyst

885 Textbooks

• How We Reason. Philip Johnson-Laird, Oxford Press

• Causal Models: How People Think about the World and Its Alternatives. S. Sloman, Oxford Press R

• Human Reasoning and Cognitive Science. Keith Stenning & Michiel van Lambalgen, MIT Press