PrECIsPrECIs(Pragmatic-Explanatory (Pragmatic-Explanatory Continuum Indicators)Continuum Indicators)

“Spokes”“Spokes”

Dave Sackett, on behalf of at Dave Sackett, on behalf of at least 23 collaboratorsleast 23 collaborators

Why the smartass title?Why the smartass title?

1.1. We have to have an acronym to be in the We have to have an acronym to be in the same league as the cardiologists and same league as the cardiologists and many other trialistsmany other trialists

2.2. PrECIs = precis PrECIs = precis (in both Canadian languages)(in both Canadian languages) = a = a summarysummary or abstract of a longer text or abstract of a longer text or speech.or speech.

The traditional distinction - 1The traditional distinction - 1

• Some trials ask whether an Some trials ask whether an intervention intervention cancan work, under work, under tightly-tightly-controlled, controlled, idealideal conditions conditions. .

• We call theseWe call these “Explanatory” “Explanatory” or or “Efficacy”“Efficacy” trials. trials.

Example of an Example of an ExplanatoryExplanatory Trial Trial

• ““Among patients with Among patients with angiographically- angiographically- confirmedconfirmed, symptomatic 70-99% stenosis of a , symptomatic 70-99% stenosis of a carotid artery, can the addition of carotid carotid artery, can the addition of carotid endarterectomy (endarterectomy (performed by an expert performed by an expert vascular or neurosurgeon with an excellent track vascular or neurosurgeon with an excellent track recordrecord) to ) to best medical therapybest medical therapy, vs. best medical , vs. best medical therapy alone, reduce the risk of therapy alone, reduce the risk of major or fatal major or fatal strokestroke over the next two years of over the next two years of rigorous follow-rigorous follow-upup?” ?” (NASCET: NEJM 1991;325:445-53)(NASCET: NEJM 1991;325:445-53)

Advantage of an Advantage of an explanatoryexplanatory trial trial::

If negative, you can abandon the If negative, you can abandon the treatment (it won’t work anywhere)treatment (it won’t work anywhere)

Disadvantage of an Disadvantage of an explanatoryexplanatory trial trial::

If positive, you still don’t know whether it If positive, you still don’t know whether it will work in will work in usualusual health care conditions health care conditions

The traditional distinction - 2The traditional distinction - 2

• Other trials ask whether an intervention Other trials ask whether an intervention doesdoes work under the work under the usualusual conditions that conditions that apply where it would be used.apply where it would be used.

• We call theseWe call these “Pragmatic”“Pragmatic” or or “Effectiveness”“Effectiveness” trials. trials.

• They are the primary focus of PraCTiHC They are the primary focus of PraCTiHC and SUPPORTand SUPPORT

Example of a Example of a PragmaticPragmatic Trial Trial

Among Among women at 12-32 weeks gestationwomen at 12-32 weeks gestation whose clinicians thought they were at whose clinicians thought they were at sufficient risk for pre-eclampsia or IUGR to sufficient risk for pre-eclampsia or IUGR to be uncertain whether they should be be uncertain whether they should be prescribed ASAprescribed ASA, does , does simplysimply prescribing prescribing ASAASA (compared with placebo), and with (compared with placebo), and with no no study follow-up visitsstudy follow-up visits, reduce the risk of a , reduce the risk of a composite of bad outcomes for her or her composite of bad outcomes for her or her babybaby? ? (CLASP: Lancet 1994;343:619-29)(CLASP: Lancet 1994;343:619-29)

Advantage of a Advantage of a pragmaticpragmatic trial trial::

If positive, it really works and you can implement If positive, it really works and you can implement the treatment just about the treatment just about everywhereeverywhere

Disadvantage of a Disadvantage of a pragmaticpragmatic trial trial::

If negative, you can’t distinguish a worthless If negative, you can’t distinguish a worthless treatment from an efficacious treatment that treatment from an efficacious treatment that isn’t applied/accepted widely enough.isn’t applied/accepted widely enough.

Because of these differences in Because of these differences in interpretation and application . . .interpretation and application . . .

It is important to be able to distinguish It is important to be able to distinguish PragmaticPragmatic from from ExplanatoryExplanatory trials trials

A UNC group developed a A UNC group developed a diagnostic test to distinguish themdiagnostic test to distinguish them

• Identified 7 “domains” they thought were Identified 7 “domains” they thought were important.important.

• Asked each of 12 US & Canadian Asked each of 12 US & Canadian “Evidence-Based Practice Center” “Evidence-Based Practice Center” Directors to nominate 6 trials:Directors to nominate 6 trials:

4 to exemplify Pragmatic trials4 to exemplify Pragmatic trials2 to exemplify Explanatory trials2 to exemplify Explanatory trials• Two blinded raters applied the 7 domain-Two blinded raters applied the 7 domain-

criteria and decided yes/no for eachcriteria and decided yes/no for each

Domain-criteriaDomain-criteria

1.1. Population was in primary carePopulation was in primary care

2.2. Less stringent eligibility criteriaLess stringent eligibility criteria

3.3. Health outcomes (function, QoL, mortality)Health outcomes (function, QoL, mortality)

4.4. Long study duration; clinically relevant Long study duration; clinically relevant treatment modalities (considered compliance treatment modalities (considered compliance an outcome)an outcome)

5.5. Assessment of adverse eventsAssessment of adverse events

6.6. Adequate sample size to assess a minimally Adequate sample size to assess a minimally important difference from a patient perspectiveimportant difference from a patient perspective

7.7. ITT analysisITT analysis

ResultsResults

Kappa for yes/no on the domains = 0.42Kappa for yes/no on the domains = 0.42

Decided best cut-point for a positive test Decided best cut-point for a positive test was the satisfaction of 6 of the 7 criteriawas the satisfaction of 6 of the 7 criteria

Sensitivity = 72%Sensitivity = 72%

Specificity = 83%Specificity = 83%

LR+ 4.3LR+ 4.3

LR- 0.3 LR- 0.3

Their ROC CurveTheir ROC Curve

ButBut

• Pragmatic vs. Explanatory is not an Pragmatic vs. Explanatory is not an either/or dichotomyeither/or dichotomy

• It is a It is a continuumcontinuum

• And individual methodological components And individual methodological components of a trial often vary in their “pragmatic-of a trial often vary in their “pragmatic-ness”ness”

And in SUPPORT we want And in SUPPORT we want to be able to describeto be able to describe

BOTHBOTHWhere a Where a trialtrial resides on that continuum … resides on that continuum …

ANDANDWhere a trial’s Where a trial’s individual componentsindividual components reside on that continuum.reside on that continuum.

That is, we want a summary That is, we want a summary or “or “precisprecis” of the trial and its ” of the trial and its individual methodological individual methodological componentscomponents

So a group of us have been So a group of us have been working on:working on:

PrPr = Pragmatic (to) = Pragmatic (to)

EE = Explanatory = Explanatory

CC = Continuum = Continuum

IsIs = Indicators = Indicators

There are 8 There are 8 PrECIsPrECIs elements elements (“spokes”) (“spokes”)

• Each is defined in terms of Each is defined in terms of restrictionsrestrictions on an otherwise totally on an otherwise totally pragmatic trialpragmatic trial

• The The more restrictionsmore restrictions in a trial, the in a trial, the higher its scorehigher its score, and the , and the smaller the smaller the populationpopulation to whom its results can to whom its results can be extrapolatedbe extrapolated

Spoke #1: Spoke #1: Participant Eligibility CriteriaParticipant Eligibility Criteria

• The extent to which restrictive eligibility The extent to which restrictive eligibility criteria were used in selecting study criteria were used in selecting study participants/patientsparticipants/patients

• Eg, age, risk, responsiveness, past Eg, age, risk, responsiveness, past compliancecompliance

Spoke #2: Spoke #2: Intervention FlexibilityIntervention Flexibility

• The extent to which restrictions were The extent to which restrictions were placed on how to apply the primary placed on how to apply the primary intervention and any co-interventionsintervention and any co-interventions

• Eg, inflexible protocols for how every bit of Eg, inflexible protocols for how every bit of the primary intervention was to be applied, the primary intervention was to be applied, and how many and which co-interventions and how many and which co-interventions were permittedwere permitted

Spoke #3: Spoke #3: Practitioner ExpertisePractitioner Expertise

• The extent to which restrictive demands The extent to which restrictive demands for ever-greater expertise were placed on for ever-greater expertise were placed on the practitioners who applied the the practitioners who applied the experimental maneuver.experimental maneuver.

• Eg, experience, certification, recognition, Eg, experience, certification, recognition, validation of expertise through validation of expertise through examination of past patients’ recordsexamination of past patients’ records

Spoke #4: Spoke #4: Follow-Up Follow-Up IntensityIntensity

• The restriction of “usual” follow-up by The restriction of “usual” follow-up by demands for increasing frequency and demands for increasing frequency and intensity of follow-up of trial participants.intensity of follow-up of trial participants.

• Eg, more frequent follow-up, and attempts Eg, more frequent follow-up, and attempts to track down and re-enlist trial to track down and re-enlist trial participants who drop-out.participants who drop-out.

Spoke #5: Spoke #5: Follow-Up DurationFollow-Up Duration

• The restriction of follow-up duration so that The restriction of follow-up duration so that it becomes too short to capture important it becomes too short to capture important health outcomeshealth outcomes

• Eg, too short to capture long-term efficacy Eg, too short to capture long-term efficacy and safety, restriction to surrogate and safety, restriction to surrogate mechanistic “biomarkers”mechanistic “biomarkers”

Spoke #6: Spoke #6: Participant ComplianceParticipant Compliance

• Restrictions on leaving trial participants Restrictions on leaving trial participants alone to follow/ not follow trial treatments alone to follow/ not follow trial treatments as they would in usual health care.as they would in usual health care.

• Eg, compliance measurements, feed-back, Eg, compliance measurements, feed-back, and the employment of compliance-and the employment of compliance-improving strategies.improving strategies.

Spoke #7: Spoke #7: Practitioner AdherencePractitioner Adherence

• Restrictions on leaving trial practitioners Restrictions on leaving trial practitioners alone to offer and apply trial treatments as alone to offer and apply trial treatments as they would in usual health care.they would in usual health care.

• Eg, adherence measurements, feed-back, Eg, adherence measurements, feed-back, and the employment of adherence-and the employment of adherence-improving strategies.improving strategies.

Spoke #8: Spoke #8: Primary AnalysisPrimary Analysis

• Restrictions (in the form of exclusions) on Restrictions (in the form of exclusions) on the data that are incorporated in the the data that are incorporated in the primary analysis.primary analysis.

• Eg, excluding drop-outs or non-compliant Eg, excluding drop-outs or non-compliant patients from the primary analysis (“per patients from the primary analysis (“per protocol”).protocol”).

The results can be The results can be displayed graphicallydisplayed graphically

The PrECIs SpokesThe PrECIs Spokes

And the graphic form can be used And the graphic form can be used to display agreement among to display agreement among readers/observers of the same trial readers/observers of the same trial reportreport

• For example, the latest group of Trout For example, the latest group of Trout Fellows read two low-dose aspirin trials for Fellows read two low-dose aspirin trials for preventing/treating pre-eclampsia preventing/treating pre-eclampsia

The CLASP Trial (Lancet ’94)The CLASP Trial (Lancet ’94)

The Caritis et al trial (NEJM ’98)The Caritis et al trial (NEJM ’98)

And the graphic form can be And the graphic form can be used to display an overall used to display an overall pattern in a trialpattern in a trial

Could “connect the dots” of Could “connect the dots” of greatest agreementgreatest agreement

• The resulting wheel could be informativeThe resulting wheel could be informative

• Small = applies to only a small proportion Small = applies to only a small proportion of the target population = Explanatoryof the target population = Explanatory

• Large = applies to a large proportion of the Large = applies to a large proportion of the target population = Pragmatictarget population = Pragmatic

• Lumpy-Bumpy = inconsistent/ ?confused Lumpy-Bumpy = inconsistent/ ?confused protocolprotocol

A highly Explanatory “expert” surgical trial:A highly Explanatory “expert” surgical trial:

The CLASP TrialThe CLASP Trial

The Caritis TrialThe Caritis Trial

Might want a summary numberMight want a summary number

• Advantage: To give an overall indicator of Advantage: To give an overall indicator of “Pragmatic-ness”“Pragmatic-ness”

• Disadvantage: Hides individual spoke Disadvantage: Hides individual spoke scores, which may have extreme valuesscores, which may have extreme values

• Constructed in terms of “restrictions” to Constructed in terms of “restrictions” to study participants, treatments, analyses, study participants, treatments, analyses, etc.etc.

• Few restrictions = low # = PragmaticFew restrictions = low # = Pragmatic• Many restrictions = high # = ExplanatoryMany restrictions = high # = Explanatory

Summary numberSummary number

Simply add the scores for the individual Simply add the scores for the individual spokesspokes

The NASCET trial scores 27 !The NASCET trial scores 27 !

The Caritis Trial scores 10The Caritis Trial scores 10

The CLASP Trial scores 6The CLASP Trial scores 6

Progress to dateProgress to date

1.1. Have agreed on the 8 domainsHave agreed on the 8 domains

2.2. Have developed 3Have developed 3rdrd drafts of criteria drafts of criteria for themfor them

3.3. Have demonstrated moderate to Have demonstrated moderate to good agreement in applying criteriagood agreement in applying criteria

Work yet to be done:Work yet to be done:

1.1. Further refinement of the criteria Further refinement of the criteria for (at least some) spokesfor (at least some) spokes

2.2. Decide how they should be Decide how they should be scoredscored

3.3. Do more face-validation studiesDo more face-validation studies

4.4. Get observer agreement up to Get observer agreement up to high levelshigh levels

1. Further develop the criteria 1. Further develop the criteria for (some) spokesfor (some) spokes

• Do some individual elements need to be Do some individual elements need to be added, altered, or eliminated?added, altered, or eliminated?

2. How should they be scored?.......2. How should they be scored?.......

• Independent of each other, and equal in Independent of each other, and equal in weight (1 point each, with their sum weight (1 point each, with their sum naturally limited to 4 points)?naturally limited to 4 points)?

• Independent of each other, but weighted Independent of each other, but weighted by their importance (1-4 points each, with by their importance (1-4 points each, with their sum truncated at 4)?their sum truncated at 4)?

• Mutually exclusive, and progressive Mutually exclusive, and progressive (maximum score of 4)? (maximum score of 4)?

3. More face-validation studies3. More face-validation studies

• A comparison to the Gartlehner et al set of A comparison to the Gartlehner et al set of trials is underwaytrials is underway

4. Get observer agreement 4. Get observer agreement up to high levelsup to high levels

• As part of our work in revising and As part of our work in revising and improving the spokes and improving the spokes and individual criteriaindividual criteria

Please help us !Please help us !

1.1. Please review the articles that Andy Please review the articles that Andy distributed distributed ((Rodrigo Salinas & Eduardo Bergel)Rodrigo Salinas & Eduardo Bergel)

2.2. Score themScore them

3.3. Suggest improvements in the Suggest improvements in the individual criteria individual criteria

4.4. Suggest how they should be scored Suggest how they should be scored at every level (individual, spoke & at every level (individual, spoke & overall)overall)

Last week at the Trout Centre

Results of our PrECIs exerciseResults of our PrECIs exercise

Thanks, everyone !Thanks, everyone !

Overall RatingsOverall Ratings

How long (How long (minutesminutes) did it take to apply these ) did it take to apply these PrECIs criteria?PrECIs criteria?

<10<10 x x

10-1410-14 xx

15-1915-19 xxxxxxxxxxxxxxxxxx

20-2420-24 xxxxxxxxxxxxxxxx

25-2925-29

30-3430-34 xxxxxxxxxxxxxx

35-3935-39 xx

40+40+ xxxx

How How difficultdifficult was it to apply these criteria? was it to apply these criteria?

Very easyVery easy 11 22 xxxx 33 xx 44 xxxxxxxx 55 xxxxxxxxxxxxxxxxxxxx 66 xxxx 77 xxxxxxxxxxxx 88 xxxxxxxxxx 99 xxxx 1010Very DifficultVery Difficult

How well were How well were important propertiesimportant properties captured? captured?

Not at allNot at all 11 22 33 xxxx 44 xxxxxx 55 xxxx 66 xxxxxx 77 xxxxxxxxxxxx 88 xxxxxxxxxxxxxxxx 99 xxxx 1010 xxxxExtremely wellExtremely well

How much How much funfun did you have ? did you have ?

No fun at allNo fun at all 11 xx 22 xx 33 xx 44 xxxx 55 xxxx 66 xxxx 77 xxxxxxxxxx 88 xxxxxxxxxxxxxxxxxxxxxx 99 xxxxxx 1010 xxGreat funGreat fun

Observer Variation Results Observer Variation Results

Magpie TrialMagpie Trial

Belfort TrialBelfort Trial

Your suggestions for revisionsYour suggestions for revisions

Keep them coming!Keep them coming!

Need to define when a Spoke might Need to define when a Spoke might be “Not Applicable” be “Not Applicable”

• Eg, Spoke 6: “Participant Compliance”Eg, Spoke 6: “Participant Compliance”

• When treatment is applied in a single When treatment is applied in a single session at the start of the trial (an session at the start of the trial (an operation, an immunization, etc.) operation, an immunization, etc.) participants can’t not comply.participants can’t not comply.

Further develop the criteria for Spoke 1:Further develop the criteria for Spoke 1:

Participant Eligibility CriteriaParticipant Eligibility Criteria

Don’t charge a restriction for a unisex Don’t charge a restriction for a unisex disorder.disorder.

Further develop the criteria for Spoke 5:Further develop the criteria for Spoke 5:

Follow-Up DurationFollow-Up Duration

1.1. Spoke 5 really isn’t about follow-up Spoke 5 really isn’t about follow-up duration, it’s about restrictions on duration, it’s about restrictions on the “events” chosen for the analysis.the “events” chosen for the analysis.

2.2. So rename it and make it more clearSo rename it and make it more clear

Further develop the criteria for Spoke 8:Further develop the criteria for Spoke 8:

Primary Analysis InclusionsPrimary Analysis Inclusions

• Need to distinguish between (or combine):Need to distinguish between (or combine):

Drop-outsDrop-outsNon-compliant participantsNon-compliant participants

Suggestions about scoringSuggestions about scoring

• Make criteria within a spoke independent Make criteria within a spoke independent of each other, but weighted by their of each other, but weighted by their importance importance

• Could be worth 1-4 points for each Could be worth 1-4 points for each restrictionrestriction

• Maximum score for a spoke = Maximum score for a spoke = >> 4 4

More face-validation studiesMore face-validation studies

• A comparison to the Gartlehner et al set of A comparison to the Gartlehner et al set of trials is underwaytrials is underway

Improve observer agreementImprove observer agreement

• As part of our work in revising and As part of our work in revising and improving the spokes and improving the spokes and individual criteriaindividual criteria

Explore other issuesExplore other issues

• How would PI ratings of their own trials How would PI ratings of their own trials compare with ours?compare with ours?

• Do trials with differing scores also have Do trials with differing scores also have differing effect-sizes?differing effect-sizes?

Results of our PrECIs exerciseResults of our PrECIs exercise

Thanks, everyone !Thanks, everyone !

Logo Breakfast ClubLogo Breakfast Club

FernandoFernando

MarianMarian

EdgardoEdgardo

Anna MariaAnna Maria

KilgoreKilgore