Postgraduate Course

One or two things about Evidence-Based Management

Pitstop Evidence-Based HR, VOV lerend netwerk,

Gent, 27 september 2012

Postgraduate Course

Evidence-Based Management?

VOV leden, n= 86

Postgraduate Course

Evidence-Based Practice

1991Medicine

1998Education

1999Social care, public policy

2000Nursing

2000Criminal justice

????Management?

Postgraduate Course

Definition

Evidence-based management means making decisions

about the management of employees, teams or

organizations through the conscientious, explicit and

judicious use of four sources of information:

1. The best available scientific evidence

2. Organizational facts, metrics and characteristics

3. Stakeholders’ values and concerns

4. Practitioner expertise and judgment

Postgraduate Course

Four sources

Postgraduate Course

Four sources

Postgraduate Course

Trust me: 20 years of management experience!

Postgraduate Course

Bounded rationality

Postgraduate Course

Bounded rationality

Postgraduate Course

Seeing order in randomness Mental corner cutting Misinterpretation of incomplete data Halo effect False consensus effect Group think Self serving bias Sunk cost fallacy Cognitive dissonance reduction

Confirmation bias Authority bias Small numbers fallacy In-group bias Recall bias Anchoring bias Inaccurate covariation detection Distortions due to plausibility

Het feilbare brein

Postgraduate Course

if you’re hyperventilating

breathe into a bag

Postgraduate Course

elderly people who have an irregular heartbeat are much more likely to die of

coronary disease

give them a drug that reduces the number of

irregular beats

Postgraduate Course

How 40,000 cardiologists can be wrong

In the early 1980s newly introduced anti-arrhythmic drugs were found to be highly successful at suppressing arrhythmias.

Not until a RCT was performed was it realized that, although these drugs suppressed arrhythmias, they actually increased mortality.

The CAST trial revealed Excess mortality of 56/1000.

By the time the results of this trial were published, at least 100,000 such patients had been taking these drugs.

Postgraduate Course

Errors and Biases of Human Judgment

Doctors and managers hold many erroneous beliefs,

not because they are ignorant or stupid, but because

they seem to be the most sensible conclusion

consistent with their own professional experience!

available evidence.

Postgraduate Course

Half of what you learn will be shown to be either dead

wrong or out-of-date within 7 years of your graduation;

the trouble is that nobody can tell you which half

Sackett: remember that your teachers are full of crap,

just like your parents.

Problem II: false information

Postgraduate Course

1. Incompetent people benefit more from feedback than

highly competent people.

2. Task conflict improves work group performance while

relational conflict harms it.

3. Being intelligent is a disadvantage for performing low

skilled jobs.

Evidence-based?

Postgraduate Course

Evidence-based?

Competentie management

Excellente zorg

Kwaliteits management

Het nieuwe werken

Kennis management

Magnet, Investors in People

Plain Tree, Healing Environment

Balanced Score Card / INK

Lean / Six Sigma / TOC

Postgraduate Course

Think critically about experience,

question your assumptions, and

challenge what you think you know.

Don’t be a parrot!

(Show me the evidence!)

Postgraduate Course

Postgraduate Course

1. Formulate a focused question

2. Searching for the best available evidence

3. Critical appraisal

4. Turning evidence into practice

5. Monitor the outcome

5-step approach

Postgraduate Course

Step 3: Critical appraisal of studies

Postgraduate Course

Best available evidence?

Postgraduate Course

Research designs

What is the BEST car?

Postgraduate Course

Which design for which question?

Research designs

Postgraduate Course

Twee type vragen

Effect vs Non-effect

Postgraduate Course

Type vraag: effect

Werkt het?

Werkt het beter dan ....?

Heeft het een effect op ....?

Wat zijn de succesfactoren voor ....?

Wat is nodig om het te laten werken?

Effect

Postgraduate Course

Type: non-effect

Needs: Wat hebben mensen nodig, waar hebben ze

behoefte aan?

Attitude: Wat denken of vinden mensen van ...?

Experience: Wat zijn de ervaringen van mensen met ...?

Prevalence: Hoeveel mensen / organisaties ....?

Procedure: Hoe kunnen we .... implementeren?

Explanation: Waarom werkt het?

Economics: Hoeveel kost het (tijd en geld)?

Postgraduate Course

Explanation

Which design for which question?

Postgraduate Course

Best research design?

Postgraduate Course

Best available?

Postgraduate Course

Evidence is not the same as ‘proof’ or ‘hard facts’

Evidence can be

- so strong that no one doubts its correctness, or

- so weak that it is hardly convincing at all

What is evidence?

Postgraduate Course

The best available evidence =

Studies with the highest internal validity

Studies with the highest external validity

Postgraduate Course

internal validity = indicates to what extent the

results of the research may be biased and is

thus a comment on the degree to which

alternative explanations for the outcome found

are possible (confounding).

Internal validity

Postgraduate Course

Confounding is the idea that a 3rd variable can distort or

confuse (or confound..) a relationship between two other

variables. For instance, when factor X causes disease Y,

that relationship could be confounded by factor C that is

associated with both factor X and disease Y. C would be

an alternative explanation for the relationship observed

between X and Y.

Confounding

Postgraduate Course

What are the confounders?

1. Shoe size & quality of handwriting

2. Body length & body weight

3. Number of storks & birth rate

4. Smoking youngsters & better lung function

Postgraduate Course

?

Successful companies

Charismatic leaders

Reverse causation

Postgraduate Course

Levels of internal validity

1. Were there enough subjects in the study?

2. Was a control group used?

3. Were the subjects randomly assigned?

4. Was a pretest used?

5. Was the study started prior to the intervention or event?

6. Was the outcome measured in an objective and reliable way?

6x yes = very high (A)

5x yes = high (A)

4-3x yes = limited (B)

2x yes = low (C)

1-0x yes = very low (D)

Postgraduate Course

Levels of internal validity

Postgraduate Course

Levels of internal validity

It is shown that …

It is likely that …

Experts are of the

opinion that …

There are signs

that …

Postgraduate Course

Best available evidence:

external validity

Postgraduate Course

Ecological validity: Is your organization so different from those in the study that its results may be difficult to apply?

Population validity: Is your population so different from those in the study that its results may be difficult to apply?

External validity: generalizability

Always ask yourself to what extent the evidence is generalizable to your situation:

Postgraduate Course

Generalizability

Keep in mind:

What works in one narrowly defined setting

might not work in another,

but some psychological principles

are generalizable to all human beings.

Postgraduate Course

Step 4:

Turning evidence into practice

Postgraduate Course

Postgraduate Course

Feasible?

organizational facts and characteristics

cultural aspects

stakeholders’ values and concerns

political aspects

financial aspects /cost-effectiveness / ROI

priorities

change readiness / resistance to change

implementation capacity

timing

Postgraduate Course

Tot slot:

Focus on the decision making process! (not the outcome)