Visualisationfor Evaluation

Visualisation for Evaluation: Context and causation

diagrams

Introduction by Richard Lansdowne

The observation by Philip K Dick that “reality is what is there, even if you don’t believe in it” is relevant to today’s topic in two ways. Firstly, reality decides the evolutionary battle. We are all descended from an unbroken line of 100 million years of ancestors whose powerful visual system helped them eat – and avoid being eaten.

Realityis that which,

when you stop believing in it, doesn't go away.

Philip K. Dick, in "How To Build A Universe That Doesn't Fall Apart Two Days Later" (1978)

1

1,000

Visual evolution

Language evolution

Our visual brain has evolved to deal with reality 1,000 times longer than our verbal skills. If I say 1,000 x you have a verbal idea of the scale, but if I show 1,000 x you have a more visceral understanding. We can use that huge visual processing power to help our intellectual thinking.

Reality

BeliefsEvaluation tests whether

the intervention is in reality or not ?

Interventions are based on beliefs.

The second way Dick’s observation is relevant to ’evaluation and visualisation’ is that policy interventions are based on beliefs. Our task as evaluators is to check how well those beliefs pass the reality test. Visualisations can help at the planning stage, to understand reality; and data visualisations can help at all stages.

Current reality

Intervention

Plan

Implement

Results

Understand

Explain (motivate action)

Assess (changed results, disentangle causation and other factors)

Roles for visualisation

Future reality

Beliefs

See http://www.datavizcatalogue.com/index.html . Also many resources at BetterEvaluation.org e.g.: Evergreen, S. (2014). Visualise data. Better Evaluation. http://betterevaluation.org/plan/describe/visualise_data

Types, uses and tools for visualisation

The Data Visualisation Catalogue

What is shown How shownWho / what PortraitHow much ChartWhere MapWhen TimelineHow FlowchartWhy Multivariable chart

http://www.danroam.com/the-back-of-the-napkin/ Solving problems and selling ideas with pictures (c) Dan Roam 2008

Dan Roam’s Visual Codex – dimension 1

http://www.danroam.com/the-back-of-the-napkin/ Solving problems and selling ideas with pictures (c) Dan Roam 2008

Dan Roam’s Visual Codex – dimension 2

Simple or Elaborate

Quality or Quantity

Vision or Execution

Individual attributes

or Comparison

Delta (change) or Status Quo

Graphic Display Choices: SQVID

http://www.danroam.com/the-back-of-the-napkin/ Solving problems and selling ideas with pictures (c) Dan Roam 2008

Dan Roam’s Visual Codex – matrix

http://www.danroam.com/the-back-of-the-napkin/ Solving problems and selling ideas with pictures (c) Dan Roam 2008

Dan Roam’s Visual Codex – matrix

Explanation: AnimationsAnimations can help explain complex system.

Explanation: AnimationsFor some longer examples (3 minutes), see animated explanations of e.g. insurance at https://www.commoncraft.com/videolist Also see The Art of Explanation, Lee Lefever., 2103, John Wiley and sons

Program (intervention) logic

Vaccin-ation

Lessdisease

Lowercosts

Visualising ‘how things work’ is relevant to evaluators getting and sharing a good understanding of the subject. For example: what are the points of leverage, and where best to measure progress.Let us follow evaluator Pat visualising ‘how things work’ as part of a policy development team she is working with. Pat’s Story: Pat has been asked to develop a deeper understanding of a health initiative with an initial program logic of using vaccination to reduce health costs. The program logic links actions to broader goals.Pat brainstorms what influences health with a group of colleagues. Using a visual approach engages a different part of their brain, and makes it easy for the group to contribute. They come up with an influence diagram showing a range of factors that affect the rate of illness.

Show factors that contribute or lead to an effect.

Influence diagram

Illness

Germs

Exercise

Nutrition

Poverty

Pat’s Story: Using a visual representation helps Patand the team share and improve their

understanding. For example, they quickly notice that poverty does not directly cause illness, but

does so indirectly by decreasing nutrition. The team rearranges and label the elements to

create a more structured causal diagram.

Show factors that contribute or lead to an effect. Can increase accuracy with annotations, such as indications of conjunction or alternatives, and necessary and sufficient factors

Causal Chain – sequenced and annotated

Illness

Germs

Exercise

Nutrition

Poverty

Necessary

Decreases

Decreases

Pat’s Story: Pat’s boss says “That’s a great linear view, but are there any feedback loops?” The team make a causal loop diagram, showing that vaccination rates can easily get into a boom and bust cycle. Vaccination rates go up after an epidemic and then drop down.

Show factors that contribute or lead to an effect. Can increase accuracy with annotations, and indications of conjunction or alternatives.

Causal Chain – extra example

Fire

Ignition

Oxygen

Fuel

Wind

Increases

Leaf litter

Diesel spill

All

>1

Other

Sufficient

Much more elaboration is possible. For example, the links in the diagrams could show probabilistic contributions. E.g. see dynamic Bayesian networks in Why: A guide to finding and using causes, Samantha Kleinberg, O’Reilly Media, 2016.

Necessary

Necessary

Necessary

less

Show causal loops over time. For example, outbreaks of measles will increase the rate of measles vaccination, which increases resistance and reduces measles outbreaks. The absence of measles outbreaks over time reduces the vaccination rate, and the loop repeats.

Causal Loops: feedback

IllnessGerms Resistance

Vaccination rate

more more

Systems thinking diagrams elaborate from a linear view of causation. Identifies reinforcing and balancing feed back loops. Can allow modelling; can help avoid perverse effects of interventions. Search for ‘causal loop diagram’ and ‘system thinking’.

Causes: Sufficient-component cause diagram

Effect

Sufficient causes?

That, is what are the minimum components of a sufficient cause for a an effect. If these are present, the effect will occur. Visually, the initial blank space invites group discussion; the clear boundary focuses discussion on whether a component is in or out.

Pat’s Story: Pat’s boss says “Now, we understand the necessary conditions of illness and feedback. But do we understand what will be a sufficient intervention for good vaccination rates? For example, public awareness is necessary, but is it sufficient?”

Sufficient-component cause diagram

Low cost

Good vaccin-

ation rates

Sufficient causes!

“Last resort” sanctions

Awareness of

importance

Lower fear of side effects

Shows the components of a sufficient cause for a an effect. There may be more than one sufficient cause (with different components). Useful where there are complex and related sets of causes and mapping detailed causal links is impractical, or unnecessary for effective action. Differs from an influence diagram due to evidence-based approach to ‘sufficiency’.

Ease & Low cost

The overall diagram shows something that is sufficient to cause the effect and then shows components of that sufficient cause.

Use where a range of causal factors are involved, with complex causal pathways. Can indicate relative importance of components. Search for “Sufficient-component cause model”.

Diagram is Excel 2016 Tree Map; also pie chart often used.

Grouping the sufficient causes together in the diagram helps make it visually clear they should be treated as a group.

Causes: Sufficient-component cause diagram

Pat’s Story: Pat’s boss says “Those diagrams have convinced the Minister – but Finance want to know how important this is overall.”Pat prepares a “Sankey diagram” to show the relative importance of the causal factors.

Note causes of chronic respiratory disease vary widely across the planet. The data shown is synthetic to illustrate the Sankey chart format and is not for a specific country

Causes: Causal Chain - quantitativeShow factors in causal chain in a quantitative or relative sense. For example, using a Sankey Chart.

Pat’s Story:

Ta Da! Initiative Approved!

Using a visual approach has helped Pat and the team clarify and communicate their understanding of the system they were wanting to change. Using the right diagrams and conventions has helped clarify:• What affects what, and how much• Factors that are necessary for the desired change• Factors that are sufficient for the desired change• Important feed back loops.This information helps develop the policy intervention and identify measurement points for outcomes and key leverage and progress points.

In particular doing this when planning an intervention can help avoid the problem of not collecting data that is ultimately important to proper evaluation, and in progressively testing of theories of causation and intermediate progress.

The data collected in these measurement points can then be analysed and presented clearly with data visualisation approaches as described in the following section.