Ensuring Human Control & Responsibility While Increasing...

Ensuring Human Control & Responsibility

While Increasing Automation: Design Principles for Supporting Insight & Reducing Errors

Ben Shneiderman ben@cs.umd.edu @benbendc

Founding Director (1983-2000), Human-Computer Interaction Lab

Professor, Department of Computer Science

Member, Institute for Advanced Computer Studies

University of Maryland

College Park, MD 20742

Enabling Teams of Humans

to Harness Networks of Machines

Ben Shneiderman ben@cs.umd.edu @benbendc

Founding Director (1983-2000), Human-Computer Interaction Lab

Professor, Department of Computer Science

Member, Institute for Advanced Computer Studies

University of Maryland

College Park, MD 20742

Interdisciplinary research community

- Computer Science & Info Studies

- Psych, Socio, Poli Sci & MITH

(www.cs.umd.edu/hcil)

Design Issues

• Input devices & strategies

• Keyboards, pointing devices, voice

• Direct manipulation

• Menus, forms, commands

• Output devices & formats

• Screens, windows, color, sound

• Text, tables, graphics

• Instructions, messages, help

• Collaboration & Social Media

• Help, tutorials, training

• Search

www.awl.com/DTUI

Fifth Edition: 2010

• Visualization

HCI Pride: Serving 5B Users

Mobile, desktop, web, cloud

Diverse users: novice/expert, young/old, literate/illiterate,

abled/disabled, cultural, ethnic & linguistic diversity, gender,

personality, skills, motivation, ...

Diverse applications: E-commerce, law, health/wellness,

education, creative arts, community relationships, politics,

IT4ID, policy negotiation, mediation, peace studies, ...

Diverse interfaces: Ubiquitous, pervasive, embedded, tangible,

invisible, multimodal, immersive/augmented/virtual, ambient,

social, affective, empathic, persuasive, ...

Integrating Humans, Machines & Networks

1) key research goals as they relate to the abilities of humans,

machines and networks to share the cognitive load to make

decisions

2) relevant milestones that have been reached in subfields

3) relevant impediments to achieving technological breakthroughs

4) systems-integration challenges to improving data-to-decision

capabilities

5) the scope & character of international approaches

6) policy implications of international research for the U.S.

Enabling Teams of Humans

to Harness Networks of Machines

My position:

Teams & Humans have initiatives, goals, and responsibility

Machines & Networks are remarkable tools,

operated, maintained & designed by teams & humans

Enabling Teams of Humans

to Harness Networks of Machines

My position:

Teams & Humans have initiatives, goals, and responsibility

Machines & Networks are remarkable tools,

operated, maintained & designed by teams & humans

More effective systems recognize

the differences between humans & machines

Clarifying responsibility supports continuous improvement

Visual presentations provide high bandwidth,

while simple clicks/gestures enable rapid operation

Enabling Teams of Humans

to Harness Networks of Machines

Research agenda:

- better design for individuals

- easier collaboration for groups

- effective social mechanisms for

teams/organizations/communities

Visual design promotes:

- sense-making

- situation awareness

- accurate decision-making in stressful distracting situations

Metrics expand:

- peta-flops, tera-bytes & giga-hertz

- peta-contribs, tera-thank-yous & giga-collabs

Information Visualization

• Visual bandwidth is enormous

• Human perceptual skills are remarkable

• Trend, cluster, gap, outlier...

• Color, size, shape, proximity...

• Goals

• Detect errors & understand source data

• Develop & test hypotheses

• Make insights & support decisions

• Collaborate & persuade

• Challenges

• Meaningful visual displays of massive data

• Interaction: widgets & window coordination

• Process models for discovery

Information Visualization & Visual Analytics

• Visual bands

• Human percle

• Trend, clus..

• Color, size,..

• Three challe

• Meaningful vi

• Interaction: w

• Process mo

1999

Information Visualization & Visual Analytics

• Visual bandwidth is enormous

• Human perceptual skills are remarkable

• Trend, cluster, gap, outlier...

• Color, size, shape, proximity...

• Three challenges

• Meaningful visual displays of massive da

• Interaction: widgets & window coordinati

• Process models for discovery

1999 2004

Information Visualization & Visual Analytics

• Visual bandwidth is enormous

• Human perceptual skills are remarkable

• Trend, cluster, gap, outlier...

• Color, size, shape, proximity...

• Three challenges

• Meaningful visual displays of massive data

• Interaction: widgets & window coordination

• Process models for discovery

1999 2004 2010

Business takes action

• General Dynamics buys MayaViz

• Agilent buys GeneSpring

• Google buys Gapminder

• Oracle buys Hyperion

• Microsoft buys Proclarity

• InfoBuilders buys Advizor Solutions

• SAP buys (Business Objects buys

Xcelsius & Inxight & Crystal Reports )

• IBM buys (Cognos buys Celequest) & ILOG

• TIBCO buys Spotfire

Spotfire: Retinol’s role in embryos & vision

Spotfire: DC natality data

http://registration.spotfire.com/eval/default_edu.asp

10M - 100M pixels: Large displays

100M-pixels & more

1M-pixels & less Small mobile devices

Information Visualization: Mantra

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

• Overview, zoom & filter, details-on-demand

Information Visualization: Data Types

• 1-D Linear Document Lens, SeeSoft, Info Mural

• 2-D Map GIS, ArcView, PageMaker, Medical imagery

• 3-D World CAD, Medical, Molecules, Architecture

• Multi-Var Spotfire, Tableau, Qliktech, Visual Insight

• Temporal LifeLines, TimeSearcher, Palantir, DataMontage

• Tree Cone/Cam/Hyperbolic, SpaceTree, Treemap

• Network Pajek, UCINet, NodeXL, Gephi, Tom Sawyer

I

nfo

Viz

S

ciV

iz .

infosthetics.com visualcomplexity.com eagereyes.org

flowingdata.com perceptualedge.com datakind.org

visual.ly Visualizing.org infovis.org

Obama Unveils “Big Data” Initiative (3/2012)

Big Data challenges:

• Developing scalable algorithms

for processing imperfect data in

distributed data stores

• Creating effective human-

computer interaction tools for

facilitating rapidly customizable

visual reasoning for diverse

missions.

http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_press_release_final_2.pdf ̀

Temporal Data: TimeSearcher 1.3

• Time series

• Stocks

• Weather

• Genes

• User-specified

patterns

• Rapid search

Temporal Data: TimeSearcher 2.0

• Long Time series (>10,000 time points)

• Multiple variables

• Controlled precision in match

(Linear, offset, noise, amplitude)

LifeLines: Patient Histories

www.cs.umd.edu/hcil/lifelines

LifeLines2: Contrast+Creatine

LifeLines2: Align-Rank-Filter & Summarize

LifeFlow: Aggregation Strategy

Temporal

Categorical Data

(4 records)

LifeLines2 format

Tree of Event

Sequences

LifeFlow Aggregation

www.cs.umd.edu/hcil/lifeflow

LifeFlow: Interface with User Controls

Treemap: Gene Ontology

www.cs.umd.edu/hcil/treemap/

+ Space filling

+ Space limited

+ Color coding

+ Size coding

- Requires learning

(Shneiderman, ACM Trans. on Graphics, 1992 & 2003)

www.smartmoney.com/marketmap

Treemap: Smartmoney MarketMap

Market falls steeply Feb 27, 2007, with one exception

Market falls steeply Sept 22, 2011, some exceptions

Market mixed, February 8, 2008

Energy & Technology up, Financial & Health Care down

Market rises, September 1, 2010, Gold contrarians

Market rises, March 21, 2011, Sprint declines

newsmap.jp

Treemap: Newsmap (Marcos Weskamp)

Treemap: WHC Emergency Room (6304 patients in Jan2006)

Group by Admissions/MF, size by service time, color by age

Treemap: WHC Emergency Room (6304 patients in Jan2006) (only those service time >12 hours)

Group by Admissions/MF, size by service time, color by age

www.hivegroup.com

Treemap: Supply Chain

www.hivegroup.com

Treemap: Nutritional Analysis

www.spotfire.com

Treemap: Spotfire Bond Portfolio Analysis

Treemap: NY Times – Car&Truck Sales

www.cs.umd.edu/hcil/treemap/

Treemap (Voronoi): NY Times - Inflation

www.nytimes.com/interactive/2008/05/03/business/20080403_SPENDING_GRAPHIC.html

VisualComplexity.com : Manuel Lima

Discovery Process: Systematic Yet Flexible

Preparation

• Own the problem & define the schedule

• Data cleaning & conditioning

• Handle missing & uncertain data

• Extract subsets & link to related information

SocialAction

• Integrates statistics & visualization

• 4 case studies, 4-8 weeks (journalist, bibliometrician, terrorist analyst, organizational analyst)

• Identified desired features, gave strong positive feedback about benefits of integration

Perer & Shneiderman, CHI2008, IEEE CG&A 2009 www.cs.umd.edu/hcil/socialaction

www.centrifugesystems.com

Network from Database Tables

NodeXL:

Network Overview for Discovery & Exploration in Excel

www.codeplex.com/nodexl

NodeXL:

Network Overview for Discovery & Exploration in Excel

www.codeplex.com/nodexl

NodeXL: Import Dialogs

www.codeplex.com/nodexl

Tweets at #WIN09 Conference: 2 groups

‘GOP’ tweets, clustered (red-Republicans)

Twitter networks: #SOTU

WWW2010 Twitter Community

Twitter Network for “TTW”

Twitter Network for #CI2012

Pittsburgh Metro

Westinghouse Electric

Pharmaceutical/Medical

No Location Philadelphia

Innovation Clusters: People, Locations, Companies

11,000 nodes

26,000 links

Pittsburgh Metro

Westinghouse Electric

Pharmaceutical/Medical

No Location Philadelphia

Innovation Clusters: People, Locations, Companies

Patent

Tech

SBIR (federal)

PA DCED (state)

Related patent

2: Federal agency

3: Enterprise

5: Inventors

9: Universities

10: PA DCED

11/12: Phil/Pitt metro cnty

13-15: Semi-rural/rural cnty

17: Foreign countries

19: Other states

Pittsburgh Metro

Westinghouse Electric

Pharmaceutical/Medical

No Location Philadelphia

Navy

Innovation Clusters: People, Locations, Companies

CHI2010 Twitter Community

www.codeplex.com/nodexl/

Flickr networks

Analyzing Social Media Networks with NodeXL

I. Getting Started with Analyzing Social Media Networks

1. Introduction to Social Media and Social Networks

2. Social media: New Technologies of Collaboration

3. Social Network Analysis

II. NodeXL Tutorial: Learning by Doing

4. Layout, Visual Design & Labeling

5. Calculating & Visualizing Network Metrics

6. Preparing Data & Filtering

7. Clustering &Grouping

III Social Media Network Analysis Case Studies

8. Email

9. Threaded Networks

10. Twitter

11. Facebook

12. WWW

13. Flickr

14. YouTube

15. Wiki Networks

www.elsevier.com/wps/find/bookdescription.cws_home/723354/description

Social Media Research Foundation

Researchers who want to

- create open tools

- generate & host open data

- support open scholarship

Map, measure & understand

social media

Support tool projects to

collection, analyze & visualize

social media data.

smrfoundation.org

Sense-Making Loop

Thomas & Cook: Illuminating the Path (2004)

Sense-Making Loop: Expanded

Thomas & Cook: Illuminating the Path (2004)

Discovery Process: Systematic Yet Flexible

Preparation

• Own the problem & define the schedule

• Data cleaning & conditioning

• Handle missing & uncertain data

• Extract subsets & link to related information

Preparation

• Own the problem & define the schedule

• Data cleaning & conditioning

• Handle missing & uncertain data

• Extract subsets & link to related information

Purposeful exploration – Hypothesis testing

• Range & distribution

• Relationships & correlations

• Clusters & gaps

• Outliers & anomalies

• Aggregation & summary

• Split & trellis

• Temporal comparisons & multiple views

• Statistics & forecasts

Discovery Process: Systematic Yet Flexible

Preparation

• Own the problem & define the schedule

• Data cleaning & conditioning

• Handle missing & uncertain data

• Extract subsets & link to related information

Purposeful exploration – Hypothesis testing

• Range & distribution

• Relationships & correlations

• Clusters & gaps

• Outliers & anomalies

• Aggregation & summary

• Split & trellis

• Temporal comparisons & multiple views

• Statistics & forecasts

Situated decision making - Social context

• Annotation & marking

• Collaboration & coordination

• Decisions & presentations

Discovery Process: Systematic Yet Flexible

CHMNI Agenda

• Brain-computer interface

• Machine learning

• Natural language dialogue

• Sensing & perception

• Software agents

• Cognitive & social science

CHMNI Agenda: Extended

• Brain-computer interface

• Machine learning

• Natural language dialogue

• Sensing & perception

• Software agents

• Cognitive & social science

• Visualization

• Collaboration

• Social media networks

• Persuasion & Motivation

• Trust & Responsibility

• History-keeping & Logging

• Continuous improvement

Research Agenda in Visualization

• Presenting complex information to diverse users on small mobile devices

• Offering busy users timely information in the right format to support rapid and accurate decision

making

• Providing decision-makers powerful temporal and geo-spatial tools to detect patterns or subtle

changes over years.

• Thorough data cleaning to cope with missing values, duplicated records, incorrect data entry,

patient name entity resolution, and proper date-time stamps.

• Efficient and effective anonymization and de-identification algorithms so data sets can be made

more widely available, while protecting patient privacy.

• Scaling visualization techniques to billions of records by filtering and dynamically forming

aggregated values. This supports retrospective analyses by researchers who may seek to

compare across patients, physicians, hospitals, time periods, and geographic regions.

• Systematic yet flexible visual analytics processes that promote complete coverage of data and

analyses questions, while preserving the option of exploration in depth when novel insights are

found.

• Logging of complex sequences of visual analytics operations so users know what was done to

produce results, can save these workflows to reapply to new datasets, and can share the

workflows with colleagues.

• Collaborative insight gathering methods so that multiple analysts can work independently and

then integrate their findings. Team-oriented medical decision-making is especially challenging

since legal liability is involved so all team members must signal their concurrence.

• Better guidelines for presenting insights to diverse audiences. Interactive visualization results,

even color-coded sortable tables, when well-designed can be enormously helpful to many users.

UN Millennium Development Goals

• Eradicate extreme poverty and hunger

• Achieve universal primary education

• Promote gender equality and empower women

• Reduce child mortality

• Improve maternal health

• Combat HIV/AIDS, malaria and other diseases

• Ensure environmental sustainability

• Develop a global partnership for development

To be achieved by 2015

30th Anniversary Symposium

May 22-23, 2013

www.cs.umd.edu/hcil

For More Information

• Visit the HCIL website for 650 papers & info on videos

www.cs.umd.edu/hcil

• Conferences & resources: www.infovis.org

• See Chapter 14 on Info Visualization

Shneiderman, B. and Plaisant, C., Designing the User Interface:

Strategies for Effective Human-Computer Interaction:

Fifth Edition (2010) www.awl.com/DTUI

• Edited Collections:

Card, S., Mackinlay, J., and Shneiderman, B. (1999)

Readings in Information Visualization: Using Vision to Think

Bederson, B. and Shneiderman, B. (2003)

The Craft of Information Visualization: Readings and Reflections

For More Information

• Treemaps • HiveGroup: www.hivegroup.com

• Smartmoney: www.smartmoney.com/marketmap

• HCIL Treemap 4.0: www.cs.umd.edu/hcil/treemap

• Spotfire: www.spotfire.com

• TimeSearcher: www.cs.umd.edu/hcil/timesearcher

• NodeXL: nodexl.codeplex.com

• Hierarchical Clustering Explorer: www.cs.umd.edu/hcil/hce

• LifeLines2: www.cs.umd.edu/hcil/lifelines2

• Similan: www.cs.umd.edu/hcil/similan