On Regulatory and Organizational Constraints in ... · On Regulatory and Organizational Constraints...

On Regulatory and Organizational Constraints in Visualization Design and Evaluation

Anamaria Crisan1,2,3, Jennifer L. Gardy1,2, Tamara Munzner3

1British Columbia Centre for Disease Control2University of British Columbia School of Population and Public Health3University of British Columbia Department of Computer Science

BELIV @ IEEE VIS 2016 1

PARABLE OF THE ANT ON THE BEACH

Simon. The Sciences of the Artificial, 1981

BELIV @ IEEE VIS 2016 2

AgentPath of agent is complex, challenging to simulate

PARABLE OF THE ANT ON THE BEACH

BELIV @ IEEE VIS 2016 3

PARABLE OF THE ANT ON THE BEACH

Agent

Environmental / Contextual Factors

Action of agent is simple, environment drives perceived complexity

BELIV @ IEEE VIS 2016 4

VIS DESIGN & EVALUATION MISSING CONTEXT

§ Contribution: Explicit guidance to incorporate regulatory and organizational contextual constraints in Vis design and evaluation

BELIV @ IEEE VIS 2016 5

VIS DESIGN & EVALUATION MISSING CONTEXT

§ Contribution: Explicit guidance to incorporate regulatory and organizational contextual constraints in Vis design and evaluation

§ [How] Modifications to a Design Study Methodology§ [How] Demonstration with case study in healthcare

Please refer to paper for Case Study

BELIV @ IEEE VIS 2016 6

VIS DESIGN & EVALUATION MISSING CONTEXT

§ Contribution: Explicit guidance to incorporate regulatory and organizational contextual constraints in Vis design and evaluation

§ [How] Modifications to a Design Study Methodology§ [How] Demonstration with case study in healthcare§ Value: Prepare Vis research for highly regulated environments

Please refer to paper for Case Study

BELIV @ IEEE VIS 2016 7

TYPES & IMPACTS OF EXTERNAL CONSTRAINTSRegulatory : Data use and access as encoded in the Law

§ Very important when working with individual-level person data

Organizational: Policies and practices enforced by an organization§ Includes enforcement of regulatory constraints§ Varies across organizations (some stricter, others not)

BELIV @ IEEE VIS 2016 8

TYPES & IMPACTS OF EXTERNAL CONSTRAINTSRegulatory : Data use and access as encoded in the Law

§ Very important when working with individual-level person data

Organizational: Policies and practices enforced by an organization§ Includes enforcement of regulatory constraints§ Varies across organizations (some stricter, others not)

Example : Hypothesis Generation Considered Harmful§ Uncontrolled hypothesis generation can be illegal

§ Can analyze data for one purpose only at a time§ Why: individual bears the burden of accidental disclosure

§ Counter to Vis insight generation principle§ Vis methods must adapt to this constraints

BELIV @ IEEE VIS 2016 9

MODIFYING A DSM FOR HIGHLY REGULATED CONTEXT

writereflectdeployimplementdesigndiscover

Pre-condition Core Analysis

proposecastwinnowlearn

Artifacts • Project proposal*• Stakeholder Power-Interest grid*• List of external constraints*• Required paperwork*

• Task abstraction• Data abstraction• Various project prototypes• Informal stakeholder feedback

End of project summary document*

Staged Design

1 2

Modify DSM by Sedlmair et. al. (2012) to include:§ Creation of additional artefacts§ Modified + added steps§ Mandatory (as opposed to suggested) check backs§ Planning the staging of Vis deliverables§ Using synthetic data

BELIV @ IEEE VIS 2016 10

R1: CLASSIFY USERS BY POWER AND INTEREST

writereflectdeployimplementdesigndiscover

Pre-condition Core Analysis

proposecastwinnowlearn

Artifacts • Project proposal*• Stakeholder Power-Interest grid*• List of external constraints*• Required paperwork*

• Task abstraction• Data abstraction• Various project prototypes• Informal stakeholder feedback

End of project summary document*

Staged Design

1 2

Recommendation 1Classify stakeholders according to power over and interest in project outcomes

BELIV @ IEEE VIS 2016 11

R1 EXAMPLE FROM HEALTHCARE CASE STUDY

§ Core stakeholders may not be aware of all Gatekeepers

§ Use grid to find “unknown” Gatekeepers

Recommendation 1Classify stakeholders according to power over and interest in project outcomes

BELIV @ IEEE VIS 2016 12

R2: ACTIVELY MANAGE COMMUNICATION

writereflectdeployimplementdesigndiscover

Pre-condition Core Analysis

proposecastwinnowlearn

Artifacts • Project proposal*• Stakeholder Power-Interest grid*• List of external constraints*• Required paperwork*

• Task abstraction• Data abstraction• Various project prototypes• Informal stakeholder feedback

End of project summary document*

Staged Design

2

Recommendation 2Actively manage communication with stakeholders using the power-interest grid

1

BELIV @ IEEE VIS 2016 13

R3: PROPOSAL DOCUMENT

writereflectdeployimplementdesigndiscover

Pre-condition Core Analysis

proposecastwinnowlearn

Artifacts • Project proposal*• Stakeholder Power-Interest grid*• List of external constraints*• Required paperwork*

• Task abstraction• Data abstraction• Various project prototypes• Informal stakeholder feedback

End of project summary document*

Staged Design

2

Recommendation 3Create a formal proposal documentLays out stakeholder & researcher expectations

1

BELIV @ IEEE VIS 2016 14

R4: PROJECT SUMMARY DOCUMENTS

writereflectdeployimplementdesigndiscover

Pre-condition Core Analysis

proposecastwinnowlearn

Artifacts • Project proposal*• Stakeholder Power-Interest grid*• List of external constraints*• Required paperwork*

• Task abstraction• Data abstraction• Various project prototypes• Informal stakeholder feedback

End of project summary document*

Staged Design

2

Recommendation 4Create a summary document at the end of a project Demonstrates value of VIS research

1

BELIV @ IEEE VIS 2016 15

R5: STAGED DESIGN PROCESS

writereflectdeployimplementdesigndiscover

Pre-condition Core Analysis

proposecastwinnowlearn

Artifacts • Project proposal*• Stakeholder Power-Interest grid*• List of external constraints*• Required paperwork*

• Task abstraction• Data abstraction• Various project prototypes• Informal stakeholder feedback

End of project summary document*

Staged Design

2

Recommendation 5Plan staged design deliverables around emerging constraintsRe-evaluate project viability at end of each stage

1

BELIV @ IEEE VIS 2016 16

R6: SYNTHETIC DATA

writereflectdeployimplementdesigndiscover

Pre-condition Core Analysis

proposecastwinnowlearn

Artifacts • Project proposal*• Stakeholder Power-Interest grid*• List of external constraints*• Required paperwork*

• Task abstraction• Data abstraction• Various project prototypes• Informal stakeholder feedback

End of project summary document*

Staged Design

2

Recommendation 6Use carefully constructed synthetic dataGet buy-in to use real data later

1

BELIV @ IEEE VIS 2016 17

THEMES OF OUR RECOMMENDATIONSIdentify, classify, mitigate, and evaluate external constraints

BELIV @ IEEE VIS 2016 18

THEMES OF OUR RECOMMENDATIONSIdentify, classify, mitigate, and evaluate external constraints

Identify agents & contextual factors

R1:Classify stakeholders according to power over and interest in project outcomes

IDENTIFY & CLASSIFY

BELIV @ IEEE VIS 2016 19

THEMES OF OUR RECOMMENDATIONSIdentify, classify, mitigate, and evaluate external constraints

Identify agents & contextual factors

R1:Classify stakeholders according to power over and interest in project outcomes

R2: Activity manage communicate with stakeholders

R3: Create a formal proposal document

R4: Create a summary document at the end of a project

Engender trust of Vis methodologies

IDENTIFY & CLASSIFY

MITIGATE

BELIV @ IEEE VIS 2016 20

THEMES OF OUR RECOMMENDATIONS

Identify agents & contextual factors

Model constraints in design and evaluation

R1:Classify stakeholders according to power over and interest in project outcomes

R2: Activity manage communicate with stakeholders

R3: Create a formal proposal document

R4: Create a summary document at the end of a project

R5: Use a staged development process

R6: Use synthetic data early on if real data is not immediately available

Engender trust of Vis methodologies

Identify, classify, mitigate, and evaluate external constraints

IDENTIFY & CLASSIFY

MITIGATE

EVALUATE

BELIV @ IEEE VIS 2016 21

CONCLUSIONS§ Regulatory & Organizational constraints are not insurmountable§ Concrete recommendations as first steps§ Vis research in regulated contexts is rewarding & valuable – go

forth and visualize, but be prepared!

Paper & Contact Info

http://cs.ubc.ca/~acrisan

acrisan@cs.ubc.ca

@amcrisan

ThanksDr. James Johnston, Dr. Maureen Mayhew, Dr. Victoria Cook, Nash Dahlla, Dr. Jason Wong, Dr. James Brooks, Johnathan Spence, Laura MacDougall, Michael Coss, Ciaran Aiken, and David Roth, Matthew Brehmer, Madison Elliott, Zipeng Liu, Dylan Dong, and Kimberly Dextras-Romagnino

http://bit.ly/amcrisan_beliv16

Additional Slides

BELIV @ IEEE VIS 2016 22

BELIV @ IEEE VIS 2016 23

TYPES & IMPACTS OF EXTERNAL CONSTRAINTS

Agile Development Considered Harmful§ Design Study Methodologies are agile development methods§ Benefits of Agile methods overlooked in regulated contexts§ Why: Policy dictates having all requirements and deliverables

before development work begins

Example – Restrictions due to Organization constraints