The Quality of Medical Advice: Vignettes (and more)

Jishnu DasWorld Bank

Saving Meena: Story of a death foretold

From the local to the global

Poor Health

Life Expectancies (Ethiopia 53; Kenya 54; Zambia 42 in 2007)

U5 Mortality (per 1000 born): Ethiopia 127 ; Kenya 114; Zambia 174)

Many countriesFigure 2: Infant Mortality Rates in Selected Countries

050

100

Infa

nt M

orta

lity

Rat

e pe

r 100

0 bi

rths

Mol

dova

Rep

ublic

of

2005

Vie

tnam

2002

Jord

an20

02A

rmen

ia20

05P

hilip

pine

s20

03D

omin

ican

Rep

ublic

2002

Nic

arag

ua20

01E

gypt

2005

Par

agua

y 19

90In

done

sia

2002

/200

3M

oroc

co20

03-2

004

Erit

rea

2002

Bol

ivia

2003

Indi

a 20

05M

adag

asca

r20

03/2

004

Sen

egal

2005

Gha

na20

03N

epal

2001

Ban

glad

esh

2004

Tanz

ania

2004

Cam

eroo

n20

04

Con

go (B

razz

avill

e)20

05M

alaw

i20

04E

thio

pia

2005

Ken

ya20

03B

urki

na F

aso

2003

Rw

anda

2005

Ben

in20

01Le

soth

o20

04G

uine

a20

05Za

mbi

a20

01/0

2N

iger

ia20

03M

ozam

biqu

e20

03C

had

2004

Mal

i200

1

Data from DHS Surveys

IMR: Selected Countries

US

200

0 (A

ll)

US

200

0 (A

frica

n-

Am

eric

ans)

050

100

Infa

nt M

orta

lity

Rat

e pe

r 100

0 bi

rths

Mol

dova

Rep

ublic

of

2005

Vie

tnam

2002

Jord

an20

02A

rmen

ia20

05P

hilip

pine

s20

03D

omin

ican

Rep

ublic

2002

Nic

arag

ua20

01E

gypt

2005

Par

agua

y 19

90In

done

sia

2002

/200

3M

oroc

co20

03-2

004

Erit

rea

2002

Bol

ivia

2003

Indi

a 20

05M

adag

asca

r20

03/2

004

Sen

egal

2005

Gha

na20

03N

epal

2001

Ban

glad

esh

2004

Tanz

ania

2004

Cam

eroo

n20

04

Con

go (B

razz

avill

e)20

05M

alaw

i20

04E

thio

pia

2005

Ken

ya20

03B

urki

na F

aso

2003

Rw

anda

2005

Ben

in20

01Le

soth

o20

04G

uine

a20

05Za

mbi

a20

01/0

2N

iger

ia20

03M

ozam

biqu

e20

03C

had

2004

Mal

i200

1

Data from DHS Surveys

IMR: Selected Countries

US

200

0 (A

ll)

US

200

0 (A

frica

n-

Am

eric

ans)

Note: Data are based on the Demographic and Health Surveys (ORC Macro, 2007. Measure DHS STATcompiler) for all countries except India, which is based on the National Family Health Survey (2005). Countries that are cross-shaded form the basis of this paper. No data are available for Paraguay after 1990.

Why?

Infectious diseases: HIV/AIDS, Malaria, Diarrhea, TB

Kenya: TB incidence rate is 62/10,000 (one of the highest in the world)

Partly to do with low incomes But see Riley on countries that improved

health outcomes at low-income levels Partly to do with access to care

Particularly a problem in rural areas

Some quotes There is an obvious difference between rural and urban

postings. Working in rural areas involves helping the poor… in urban areas one can learn, have more income, have a good school for one’s children.

Doctor in Ethiopia

It is Siberia!. Doctor in Addis

Ababa There is no plan of development for a doctor in the rural

areas; it is as if you are lost. The lack of career development means that it is as if you are punished.

Doctor in Rwanda Promotion is as important as remuneration because you

cannot stay in the same place forever. Doctor in Ghana

Sourced from Serneels and others

And yet… Figure 1: The Use of Health Facilities for ARI

020

4060

80P

ropo

rtion

Usi

ng a

Hea

lth F

acili

ty

Cha

d20

04E

thio

pia

2005

Ban

glad

esh

2004

Nep

al20

01R

wan

da20

05N

iger

ia20

03A

rmen

ia20

05M

alaw

i200

4M

ali2

001

Mor

occo

2003

-200

4B

urki

na F

aso

2003

Cam

eroo

n20

04B

enin

2001

Gui

nea

2005

Erit

rea

2002

Gha

na20

03S

eneg

al20

05C

ongo

(Bra

zzav

ille)

2005

Ken

ya20

03P

arag

uay

1990

Bol

ivia

2003

Mad

agas

car2

003/

2004

Phi

lippi

nes

2003

Moz

ambi

que

2003

Tanz

ania

2004

Leso

tho

2004

Nic

arag

ua20

01In

done

sia

2002

/200

3E

gypt

2005

Indi

a 20

05M

oldo

va R

epub

lic o

f200

5D

omin

ican

Rep

ublic

2002

Vie

tnam

2002

Zam

bia

2001

/02

Jord

an20

02

Data from DHS Surveys

Use of Health Facilities for ARI

020

4060

80P

ropo

rtion

Usi

ng a

Hea

lth F

acili

ty

Cha

d20

04E

thio

pia

2005

Ban

glad

esh

2004

Nep

al20

01R

wan

da20

05N

iger

ia20

03A

rmen

ia20

05M

alaw

i200

4M

ali2

001

Mor

occo

2003

-200

4B

urki

na F

aso

2003

Cam

eroo

n20

04B

enin

2001

Gui

nea

2005

Erit

rea

2002

Gha

na20

03S

eneg

al20

05C

ongo

(Bra

zzav

ille)

2005

Ken

ya20

03P

arag

uay

1990

Bol

ivia

2003

Mad

agas

car2

003/

2004

Phi

lippi

nes

2003

Moz

ambi

que

2003

Tanz

ania

2004

Leso

tho

2004

Nic

arag

ua20

01In

done

sia

2002

/200

3E

gypt

2005

Indi

a 20

05M

oldo

va R

epub

lic o

f200

5D

omin

ican

Rep

ublic

2002

Vie

tnam

2002

Zam

bia

2001

/02

Jord

an20

02

Data from DHS Surveys

Use of Health Facilities for ARI

U.S

. (19

88)

020

4060

80P

ropo

rtion

Usi

ng a

Hea

lth F

acili

ty

Cha

d20

04E

thio

pia

2005

Ban

glad

esh

2004

Nep

al20

01R

wan

da20

05N

iger

ia20

03A

rmen

ia20

05M

alaw

i200

4M

ali2

001

Mor

occo

2003

-200

4B

urki

na F

aso

2003

Cam

eroo

n20

04B

enin

2001

Gui

nea

2005

Erit

rea

2002

Gha

na20

03S

eneg

al20

05C

ongo

(Bra

zzav

ille)

2005

Ken

ya20

03P

arag

uay

1990

Bol

ivia

2003

Mad

agas

car2

003/

2004

Phi

lippi

nes

2003

Moz

ambi

que

2003

Tanz

ania

2004

Leso

tho

2004

Nic

arag

ua20

01In

done

sia

2002

/200

3E

gypt

2005

Indi

a 20

05M

oldo

va R

epub

lic o

f200

5D

omin

ican

Rep

ublic

2002

Vie

tnam

2002

Zam

bia

2001

/02

Jord

an20

02

Data from DHS Surveys

Use of Health Facilities for ARI

020

4060

80P

ropo

rtion

Usi

ng a

Hea

lth F

acili

ty

Cha

d20

04E

thio

pia

2005

Ban

glad

esh

2004

Nep

al20

01R

wan

da20

05N

iger

ia20

03A

rmen

ia20

05M

alaw

i200

4M

ali2

001

Mor

occo

2003

-200

4B

urki

na F

aso

2003

Cam

eroo

n20

04B

enin

2001

Gui

nea

2005

Erit

rea

2002

Gha

na20

03S

eneg

al20

05C

ongo

(Bra

zzav

ille)

2005

Ken

ya20

03P

arag

uay

1990

Bol

ivia

2003

Mad

agas

car2

003/

2004

Phi

lippi

nes

2003

Moz

ambi

que

2003

Tanz

ania

2004

Leso

tho

2004

Nic

arag

ua20

01In

done

sia

2002

/200

3E

gypt

2005

Indi

a 20

05M

oldo

va R

epub

lic o

f200

5D

omin

ican

Rep

ublic

2002

Vie

tnam

2002

Zam

bia

2001

/02

Jord

an20

02

Data from DHS Surveys

Use of Health Facilities for ARI

U.S

. (19

88)

Note: The graph shows the % of respondents who said that they had taken their child to a health facility for the treatment of an Acute Respiratory Infection. Data are based on the Demographic and Health Surveys (ORC Macro, 2007. Measure DHS STATcompiler) for all countries except India, which is based on the National Family Health Survey (2005). Countries that are cross-shaded form the basis of this paper. No data are available for Paraguay after 1990. The equivalent number for the US is not easily available. We use instead health seeking behavior among children for pharyngitis as reported in Stoddard, Jeffrey J., Robert F. St. Peter, and Paul W. Newacheck. 1994.

In many countries around the world, people use facilities a lot more than they do in the US

And yet…0

1020

30To

tal N

umbe

r of P

rovi

ders

19 24 21 36 6 14 26 29 1 12 25 28 30 4 17 32 34 11 15 7 10 13 27 20 23 31 33 38 8 18This is ongoing work

Deep Rural Madhya PradeshHow Many Medical Care Providers

In deep rural Madhya Pradesh, one of the states in India with the worst HD outcomes, there are 11.3 medical care providers accessible for representative rural households who do not live close to national highways or close to urban centers

And yet…

020

4060

8010

0H

ouse

hold

s pe

r Pro

vide

r

28 33 4 38 13 8 18 26 31 6 27 15 11 10 23 17 20 36 29 1 25 12 32 7 30 34 19 21 24 14This is ongoing work

Deep Rural Madhya PradeshHow Many Medical Care Providers

This is not because these villages are huge: for these villages, there are around 20 households per provider

New Studies on Doctor Visits

In 5 countries studied, the poor visit doctors almost as much as the rich Kenya: 60% of poorest quintile seek care

when sick relative to 78% of richest quintile In urban India, new survey methods show

that the poor visit doctors more than the rich The difference arises because of shorter recall

periods In rural India, households visit doctors twice

as much as in the United States

Do the poor really go to doctors less than the rich?

Experimental data from Delhi show that when households are asked on the basis of monthly recall, self-reported doctor visits fall by 65 percent for the poor compared to weekly recall. In fact, the poor go to doctors more than the rich. We don’t know what happens in other countries, because weekly recall questionnaires are very rare!

One alternative Health outcomes may be related to the quality of

health care that people receive

Structural Quality Quality traditionally defined as structural quality

—state of infrastructure, availability of medicines. This is clearly informative BUT Its not correct when demand is a factor (medicines) The quality of medical advice may be equally (or

even more) important The relationship between quality of advice and

structural quality is weak (India, Indonesia, Tanzania)

The Quality of Medical Advice: New evidence Since 2002, team working on the quality of

medical advice at The World Bank and the University of Maryland

Basic Idea What can be measured, and how? How do these measurements help us

understand the quality of medical advice? How can the quality of medical advice be

improved?

Where we are

Quality of medical advice can be decomposed into Competence: What does a doctor or medical care

provider know about how to treat an illness Practice: What does a doctor or medical care provider

do when faced with an illness We systematically find that the two are very

different. This has important policy implications Improving competence is about training Improving practice (given a competence) is about

getting doctors to exert greater effort Countries: Tanzania, Urban India, Indonesia,

Paraguay Ongoing: Rwanda, Rural India, Argentina

Remainder of Presentation

Basic facts about competence Basic facts about practice More interesting facts Lessons learnt (thus far!)

What are Vignettes Standardized mix of (in our case) 5 cases One interviewer is `patient’; other is `recorder and observer’ Child with Diarrhea

“My child has been suffering from diarrhea for the last two days, and I do not know what to do”

History Doc gets 1 point if he asks about last urination (for

instance) Examination

Doc gets 1 point if he asks about temperature (IF asked, recorded responds `98.8 F’)

Treatment Doc prescribes treatment: treatment is graded by

independent raters in South Asia and the US

Scoring Vignettes The questions that doctors ask are compared to

a checklist of essential procedures An aggregate “index” is compiled that accounts

for the different “difficulties” of different checklist items

We call this index “competence” We normalize mean = 0 and standard-

deviation=1 Moving from 0 to 2 moves you from 50th percentile to

95th percentile

Vignettes: Advantages Advantages

Standardized case-load (same 5 cases to all docs) Standardized patient (specify that patient will

`comply fully with all medications and tests’

What they know0

2040

6080

100

Pro

babi

lity

of N

on-H

arm

ful T

reat

men

t

Diarrhea PreEclampsia TB ViralPharyngitis

Leas

t Com

peten

t

2nd Q

uintile

Averag

e Com

peten

ce

4th Q

uintile

Most C

ompe

tent

Leas

t Com

peten

t

2nd Q

uintile

Averag

e Com

peten

ce

4th Q

uintile

Most C

ompe

tent

Leas

t Com

peten

t

2nd Q

uintile

Averag

e Com

peten

ce

4th Q

uintile

Most C

ompe

tent

Leas

t Com

peten

t

2nd Q

uintile

Averag

e Com

peten

ce

4th Q

uintile

Most C

ompe

tent

Delhi DoctorsWhat doctors know

Public or Private?0

.1.2

.3.4

.5D

ens

ity

-2 -1 0 1 2C om petence

His togram K ernel Dens i ty

Pu b l i c--A l l M BB S

0.1

.2.3

.4.5

De

nsity

-2 -1 0 1 2C om petence

His togram K ernel Dens i ty

Priva te --M BB S

0.1

.2.3

.4.5

De

nsity

-2 -1 0 1 2C om petence

His togram K ernel Dens i ty

Priva te --Non -M BBS

0.1

.2.3

.4.5

De

nsity

/Pe

rcen

t

-2 -1 0 1 2C om petence

P ubl ic P roviders P rivate--MB B S

P rivate--Non-MB B S

A l l P ro vide rs

Distribution of Competence by Qua lification

An MBBS is the formal Indian medical degree—roughly the equivalent of an MD in the US

Distribution of Competence

-.5

0

.5

Low Middle High

Competence Across Neighborhood Incomes

Overall Competence Public PHCs

Regression ModelsTable 5: What Explains Quality? (1) (2) (3) (4) Income and

Institution Income, Institution and Qualification

All variables excluding price

All variables including price

% of poor households in community

-0.013 (0.002)***

-0.007 (0.002)***

-0.009 (0.002)***

-0.009 (0.002)***

% middle income households in community

-0.011 (0.003)***

-0.006 (0.003)*

-0.004 (0.003)

-0.003 (0.003)

Public Doctor 0.451 (0.172)***

-0.185 (0.175)

-0.174 (0.171)

0.365 (0.348)

MBBS Degree Holder 1.132 (0.153)***

0.957 (0.156)***

0.848 (0.171)***

Tenure in Locality -0.020 (0.007)***

-0.019 (0.007)***

Usual Price Charged (from census)

0.144 (0.080)*

Controls for Origin of Provider?

NO NO YES YES

Constant 0.616 (0.154)***

-0.194 (0.175)

-0.399 (0.255)

-0.804 (0.347)**

Observations 204

204 194 187

R-squared 0.19

0.36 0.44 0.45

R-squared corrected for measurement error

0.28 0.53 0.64 0.65

Results very similar in other countries• More competent doctors in

urban areas• More competent doctors in

richer areas (within urban and within rural areas)

• More training increases competence

Competence across countries

Measuring practice Sit in the doctors

office Record details

about every interaction between the doctor and patient Time History taking Physical exams Drugs prescribed

Photo Credit: Ken Leonard

Practice: Some numbersTable 3: International Comparisons of Effort

Sample Effort Categories or Country Time Spent Questions asked of Patient

% Who do Physical Exams

Poly-pharmacy (Total number of medicines given)

Delhi

Doctors who exert low effort 1.9 1.36 14 2.13

Doctors who exert medium effort 3.36 2.94 78 2.72

Doctors who exert high effort 6.15 5.32 98 3.05

All Doctors 3.80 3.20 63 2.63

Paraguay

Doctors who exert low effort 5.79 5.33 1.38 1.36

Doctors who exert medium effort

7.90 7.50 2.93 1.55

Doctors who exert high effort 11.34 11.91 3.64 1.65

All Doctors 8.33 8.23 2.65 1.52

Tanzania Doctors who exert low effort

(25th Percentile) 3 2 0 N/ A

All Doctors 6.32 3.96 1.51 N/ A

International Comparisons

Tanzaniac (1991) 3.0 N/ A N/ A 2.2

Nigeriac 6.3 N/ A N/ A 2.8

Malawic 2.3 N/ A N/ A 1.8

UKd 9.4 N/ A N/ A N/ A Notes: We divide doctors by terciles of effort in India and Paraguay, and the 25th percentile versus all doctors for Tanzania. The data are based on the following sources India: Das and Hammer (2007); Paraguay: Das and Sohnesen (2007); Tanzania: Leonard (Mimeo); International Comparisons: Hogelzeir and others (1993) and Deveugele and others (2003).

Another look at practice

0

1

2

3

4

5

6

7

time questions exams

low effortmediumhigh

Less than 2 minutes Just one question

Almost none!

Public-Private Again

4.22

1.56

4.76

2.28

4.06

2.19

5.39

3.1

5.89

3.26

0

2

4

6

Tim

e in

Min

utes

Lowest Second Median Fourth Top

Private

Public

Private

Public

Private

Public

Private

Public

Private

Public

Got Time?

Public-Private AgainTable 4: Practice and Competence across Sectors

Quintiles of ability Type of provider Total history questionsProbability of. examination

(percent) Time spent (minutes) Fees charged (Rs.)Lowest Private 2.93 70 4.22 21.5

Public 1.72 28 1.56 0

2nd Group Private 3.37 71 4.76 34.6

Public 1.88 41 2.28 0

Median Private 3.55 75 4.06 32.8

Public 2.17 42 2.19 0

4th Group Private 3.67 81 5.39 44.0

Public 3.55 41 3.10 0

Highest Private 4.71 81 5.89 57.3

Public 4.04 72 3.26 .01

Note: This table disaggregates provider practice (observed in the clinic) by competence measured in the vignettes and sector (public/private). History questions refer to the number of questions regarding the illness that the provider asked the patient. An examination consists of any physical contact between the provider and the patient or the use of measuring instruments, such as a thermometer, sphygmomanometer, or stethoscope. Note that an examination only implies that the device was used, not that is was used correctly. Fees charged refers to the total payment at the end of the interaction. Finally, public providers are those who were observed in their public practice and need not be providers who work only in the public sector.

Private MBBS Public MBBS Private, No MBBS

What they know, what they do40% of essential questions asked

Perc

enta

ge o

f Ess

entia

l Tas

ks C

ompl

eted

Lost Training: Tanzania

010

2030

4050

6070

Perf

orm

ance

(% o

f req

uire

d ite

ms)

0 10 20 30 40 50 60 70 80 90Competence (% of required items)

Individual clinician's competence and performancePredicted quadratic relationship of competence to performance (Public)Predicted quadratic relationship of competence to performance (Non-Public)Performance = Competence

Sourced from Ken Leonard

Lost Training: India0

.2.4

.6.8

1W

hat t

hey

Do

0 .2 .4 .6 .8 1What they said they would do

What they know W hat they Do: PrivateWhat they do: Public

Rotating The Curve

Lost Training: Private

Additional Lost Training: Public

These doctors are operating at the frontier of their knowledge

0.2

.4.6

.81

Wha

t the

y D

o

0 .2 .4 .6 .8 1What they said they would do

What they know W hat they Do: PrivateWhat they do: Public

Rotating The Curve

Lost Training: Private

Additional Lost Training: Public

0.2

.4.6

.81

Wha

t the

y D

o

0 .2 .4 .6 .8 1What they said they would do

What they know W hat they Do: PrivateWhat they do: Public

Rotating The Curve

Lost Training: Private

Additional Lost Training: Public

These doctors are operating at the frontier of their knowledge

Training or Effort?

Because of lost training We simulate that the impact of training is very

small relative to improvements in effort

The big caveat? Is it that public doctors put in a lot less effort

because they see many more patients? We find more effort in India in hospitals, which

typically see many more patients Two new and incredibly sad pieces of

research

Case loads and doctor shortage Maestad and others (2009)

went and sat in many doctors clinics in rural Tanzania

“The average doctor sees 18.5 patients per day and total time use is 5.7 mins per patient. The doctor completes 22% of essential tasks”

That’s less than 2 hours a day in an 8 hour day

There is no relationship between caseload and doctor effort!

The Hawthorne Effect35

4045

5055

6065

7075

Perc

enta

ge o

f Ite

ms C

orre

ct

-10 -5 0 5 10 15Number of Previous Consultations Under Observation

doctor observed from t = 1 doctor never observed

Effort jumps when doctors are observed

Doctors significantly increase their effort in Tanzania when they know they are being observed with no detrimental effect on patient (Leonard, 2007) Based on Ken Leonard (2007) in Journal of Health Economics

Improving health outcomes

New research suggests that improving effort could also improve outcomes Bjorkman and Svensson, 2008: Uganda Rwanda Pay-for-Performance experiment:

Ongoing, Rwanda

Summary The quality of medical advice is key and

understanding the levels and correlates of quality is an urgent priority

Measuring either competence or practice is a good start But measuring both vastly increases our

understanding of what is going on and where the policy levers may be

Resources exist to help do this A new website ready soon with all studies and

resources in one site Support from the Chief Economists office and DEC

Lessons Learnt: What worked (1)

The overall methodology is sound and important Sound: Correlates with various characteristics

as predicted by common sense Important: Highlights potential and limitations

of different policy measures The distribution of quality across public/private or

rich/poor The distribution of effort The know-do gap

Lessons Learnt: What worked (2) Initial worry that the

variation in doctors is too large in India (Allopathic, Ayurvedas, Unani, Homeopaths) for a single instrument to capture quality Turned out to not be a

bit worry because doctors were all treating patients using the same (allopathic) medicines

Therefore, they could be graded on the same scale

0.0

0.5

2.6

0.5

0.5

2.5

0.1

0.7

2.7

0 1 2 3

MBBSBIMS/B

AMS/BUMS/B

HMS

RMP/No Trai

ning

Practitioner Qualifications and Drug Use

Medicines per patient Antibiotics per patientAlternative Medicines per patient

Regardless of the style of the provider’s training, the type of medications dispensed were very similar

Lessons Learnt: What did not workVignettes

We chose the simplest cases possible with no complications

Even then, our vignettes are not good at distinguishing bad from very bad doctors (high standard-errors for less than average providers)

Perhaps adding in a simple set of written questions would help

For instance: “If a child is suffering from diarrhea, what should you give the child?”

-6-4

-20

295

% C

onfid

ence

Inte

rval

s

05

1015

Per

cent

-2 -1 0 1 2Com petence

Sam ple Density Upper Confidence IntervalCom petence Lower Confidence Interval

Classification Errors and Sample Density

Source: Author’s calculations based on World Bank-ISERDD (2003). The horizontal axis in the graph is competence, the left vertical axis is the density (in percentages) for the histogram of competence and the right vertical axis shows confidence intervals of competence. The solid line is estimated competence, which is plotted against itself (this would be the 45o line if the scales were the same). The two dashed curves represent the upper and lower confidence intervals at the 95% level of confidence. The histogram underlying the confidence interval curves shows how competence is distributed at values of the index with large and small standard errors.

Lessons Learnt: What did not workVignettes

All the 5 cases we chose did not require any treatment at the primary level. Therefore, the “mistakes” we usually pick up are

errors of commission—doctors doing things that they should not have

Definitely consider including cases that require treatment at the primary level (pneumonia?) This allows us to pick up errors of omission

Lessons Learnt: What did not workPractice

The vignettes standardize the case-mix and the patient-mix Observing real patients does not. This leads to problems

Unobserved patient characteristics could affect inference (for instance sorting)

Use an exit-survey to pick some of this up, if possible

The cases that overlap with the vignettes are limited (rare cases almost never seen)

And may not be perfect overlaps Possibility of using “simulated standardized patients”

Pilot currently underway: if this works out, it vastly improves our diagnostic abilities

Lessons Learnt: What did not workOverall

The studies thus far are mostly “boutique” studies We are working on how to mainstream them We are not sure what the cost of deviation from the “boutique”

approach would be

What I would do (again) Make sure that you keep lots of time for case

development Pilot the vignettes until all (>95%) questions that

providers ask have a predefined answer Train enumerators until they have memorized the

entire vignettes module NEW: use video-recordings of doctor-patient

interactions as training material for direct observation (these are being developed)

Things to do differently Post-code treatments (the ultimate nightmare) Make some changes to the direct observation form

Patient order Interviewer assessments of patients (to be piloted) Number of questions that patients ask

Have a clear idea of the timeline and the work program (but that’s for any of this work!)

Papers on which this presentation is based