Secondary Aims Using Data Arising from a SMART

Secondary Aims Using Data Arising from a SMART

Module 6—Day 2

Getting SMART About Developing Individualized Adaptive Health Interventions

Methods Work, Chicago, Illinois, June 11-12Daniel Almirall & Susan A. Murphy

Secondary Aims, Outline

• Discuss what is a “more deeply-tailored ATS”• Review of auxiliary data typical of SMARTs• Q-Learning: Using backward-induction logic

– S(a): Analysis of time-varying tailoring variables of second-stage treatment

• There is one step to this: Step (i)

– S(b): Analysis of baseline tailoring variables of first-stage (assuming an optimal second-stage treatment)

• There are two steps to this: Steps (ii) and (iii)

What is a more deeply-tailored ATS?

• To understand this, we first review what are the “embedded ATSs” within the ADHD SMART study

• Recall there are 4 SMART-design embedded ATSs.

Recall the Prototypical SMART Design: ADHD SMART Example

Continue Medication Responders

Medication Increase Medication Dose

Add Behavioral InterventionR

Continue Behavioral

InterventionBehavioral Intervention Increase

Behavioral Intervention

Add Medication

Non-Responders R

Responders

Non-Responders R




Continue Behavioral



Add Medication

Non-Responders R

Responders

Non-Responders R

These are the 2 embedded ATS that employ the “add other treatment” tactic




Continue Behavioral



Add Medication

Non-Responders R

Responders

Non-Responders R

These are the 2 embedded ATSs that employ the “intensify initial treatment” tactic

So why consider a more deeply-tailored ATS?

• First, it may be that some participants (e.g., those who have used MED in the past) may benefit more from MED vs BMOD.

• Second, certain types of non-responders (e.g., those who do not adhere to initial treatment) may benefit more from INTENSIFY vs ADD.

What is a more deeply-tailored ATS?It may look like this…

• If the child used MED in prior year, then begin with MED; otherwise, begin with BMOD.

• If the child is non-responsive and non-adherent to either first-line treatment, then AUGMENT with the other treatment option.

• If the child is non-responsive but adherent to either first-line treatment, then it is better to INTENSIFY first-line treatment.

• If the child is responsive to first-line treatment, then CONTINUE first-line treatment.







Other Measures Collected in a SMART




Continue Behavioral



Add Medication

Non-Responders R

Responders

Non-Responders RO1 = Demog., Pre-txt Medication Hx, Pre-txt ADHD scores, Pre-txt school performance, ODD Dx, …

O2 = Month of non-response, adherence to first-stage txt, …

O1 A1 O2 / R Status A2 Y

How are O1 and O2 used?

• We can use the auxiliary data O1 to help decide which is best MED v BMOD for certain individuals

• We can use the auxiliary data O1 and O2 to help decide which is best INTENSIFY v ADD for certain individuals







Using Q-Learning to develop a more richly-individualized ATS

Q-Learning is an extension of regression to sequential treatments.

• Q-Learning results in a proposal for an adaptive treatment strategy with greater individualization.

• A subsequent trial would evaluate the proposed adaptive treatment strategy versus usual care.

Three Steps in Q-Learning RegressionWork backwards (as you would for a project time-line)

i. Do a regression to learn how to more deeply- tailor second-stage treatment using O1 and O2 (in ADHD SMART, this is only with non-responders)

ii. Assign each non-responder the value Ŷi , an estimate of the outcome under the second-line treatment that yields best outcome for that person i. Responders get their observed Yi.

iii.Using Ŷi do a regression to learn how to more deeply-tailor first-stage treatment using O1




Continue Behavioral



Add Medication

Non-Responders R

Responders

Non-Responders R

Step (i) of Q-Learning: Second-stage treatment tailoring?

Step (i)a.

Step (i)b.

Adherence to initial MED

Adherence to initial BMODO1 A1 O2 / R Status A2 Y

Step (i) of Q-Learning: Second-stage treatment tailoring?

To accomplish Step (i) of Q-Learning, you may fit a regression model such as this:

Y = b1 + b2 o11c + b3 o12c + b4 o13c + b5 o14c + b6 o21c + b7 a1 + b8 o22 + b9 a2*a1 + b10 a2*o22 + b11 a2 + error

From such a model we would learn if, for example, o22 (adherence to first stage txt) is a strong moderator of the effect of a2.

Step (i) of Q-Learning: Learn best second-stage treatment for non-responders

≈ -1.3 to -2.1

Among non-adherers to either first-stage treatment, better to add txt.

This analysis is with simulated data.

INT –ADD


≈ +1.02

Among adherers to MED, better to intensify

MED.


INT –ADD


≈ +0.25

Among adherers to

BMOD, better to intensify

but not by much (NS).


INT –ADD





Step (ii) of Q-Learning: Graphically


For patients who would initially begin MED and don’t respond, we assign them their optimal outcome under the treatment that is best for them based on adherence or non-adherence.

For those who would begin on BMOD and don’t respond, assign them their optimal outcome based on adherence or non-adherence.





Step (iii) of Q-Learning: First-stage treatment tailoring?

To accomplish Step (iii) of Q-Learning, you may fit a regression model such as this:

Ŷ = b1 + b2 o11c + b3 o12c + b4 o14c + b5 o13 + b6 o13*a1 + b7 a1 + error

From such a model we would learn if, for example, o13 (prior MED) is a strong moderator of the effect of a1 such that it would make a good tailoring variable.

Step (iii) Q-Learning: Learn optimal first-treatment for all given optimal future txt

-0.50

Among kids using MED in prior year, it is better to start with MED.

= BMOD – MED


Q-Learning Step (iii): Learn optimal first-treatment for all given optimal future txt

Among kids not using MED in prior year, it is better to start

with BMOD

+0.62= BMOD –

MED


SAS software to perform Q-Learning

• We next show you how to do • Step (i) using regression• Steps (ii) and (iii) using a SAS add-on known

as PROC QLEARN• We will use the two example regression models

shown previously.

SAS code for Step (i) of Q-Learning: Second-stage treatment tailoring

* use only non-responders;data dat10; set dat1; if R=0; run;

* comparison of add vs intensify given first line txt and adherence;proc genmod data = dat10; model y = o11c o12c o13c o14c a1 o21c o22 a2 a2*a1 a2*o22; * effect of add vs intensify given first-line = MED x ADH status; estimate 'INT vs ADD for NR MED ADH' a2 2 a2*a1 -2 a2*o22 2 ; estimate 'INT vs ADD for NR MED Non-ADH' a2 2 a2*a1 -2 a2*o22 0 ; * effect of add vs intensify given first-line = BMOD x ADH status; estimate 'INT vs ADD for NR BMOD ADH' a2 2 a2*a1 2 a2*o22 2 ; estimate 'INT vs ADD for NR BMOD Non-ADH' a2 2 a2*a1 2 a2*o22 0 ;run;

Contrast Estimate Results

95% Conf Limits

Label Estimate Lower Upper P-value

INT vs ADD for NR MED ADH 1.0240 0.4131 1.6350 <.0001

INT vs ADD for NR MED Non-ADH -1.3412 -1.9896 -0.6927 <.0001

INT vs ADD for NR BMOD ADH 0.2503 -0.3950 0.8956 0.4471

INT vs ADD for NR BMOD Non-ADH -2.1149 -2.7050 -1.5248 <.0001


Try it yourself in SAS• Go to the file:

sas_code_modules_4_5_and_6_ADHD.doc

• Copy the SAS code on Page 11• Paste into SAS Enhanced Editor window• Press F8 or click the Submit button (the little

running guy)

SAS code for Step (i) of Q-Learning: Second-stage treatment tailoring

* use only non-responders;data dat10; set dat1; if R=0; run;

* comparison of add vs intensify given first line txt and adherence;proc genmod data = dat10; model y = o11c o12c o13c o14c a1 o21c o22 a2 a2*a1 a2*o22; * effect of add vs intensify given first-line = MED x ADH status; estimate 'INT vs ADD for NR MED ADH' a2 2 a2*a1 -2 a2*o22 2 ; estimate 'INT vs ADD for NR MED Non-ADH' a2 2 a2*a1 -2 a2*o22 0 ; * effect of add vs intensify given first-line = BMOD x ADH status; estimate 'INT vs ADD for NR BMOD ADH' a2 2 a2*a1 2 a2*o22 2 ; estimate 'INT vs ADD for NR BMOD Non-ADH' a2 2 a2*a1 2 a2*o22 0 ;run;


95% Conf Limits


INT vs ADD for NR MED ADH 1.0240 0.4131 1.6350 <.0001

INT vs ADD for NR MED Non-ADH -1.3412 -1.9896 -0.6927 <.0001

INT vs ADD for NR BMOD ADH 0.2503 -0.3950 0.8956 0.4471

INT vs ADD for NR BMOD Non-ADH -2.1149 -2.7050 -1.5248 <.0001


SAS add-on: PROC QLEARNWhat do we provide PROC QLEARN?

1. Data set with O1, A1, R, O2, A2, Y2. The second stage regression model

• Y ~ O1, A1, O2, A2• Specify among who to do this regression

(e.g., non-responders in ADHD SMART)3. The first stage regression model

• Ŷ ~ O1, A1

SAS add-on: PROC QLEARNWhat does PROC QLEARN do?

1. It implements Step (i): Second-stage Regression • Provides us with second-stage regression

parameter estimates2. It implements Step (ii): Obtains Ŷ

• It assigns non-responders the outcome under the best second-stage treatment based on Step (i).

• It assigns responders their observed outcome.3. It implements Step (iii): First-stage Regression

• Provides first-stage regression parameter estimates• Provides appropriate confidence intervals for the

first-stage regression parameter estimates

SAS add-on: PROC QLEARNWhat does the syntax look like?

PROC QLEARN <options for input> ;MAIN1 variables ;TAILOR1 variables ;MAIN2 variables ;TAILOR2 variables ;RESPONSE variable ;STG1TRT variable ;STG2TRT variable ;STG2SAMPLE variable ;ALPHA value ;

RUN;

SAS add-on: PROC QLEARNModel Specification

PROC QLEARN <options for input> ;MAIN1 variables ;TAILOR1 variables ;MAIN2 variables ;TAILOR2 variables ;RESPONSE variable ;STG1TRT variable ;STG2TRT variable ;...

RUN;

STAGE2 REGRESSION RESULTS ARE OUTPUT LIKE THIS:INT + MAIN2 + TAILOR2 + TAILOR2*STG2TRT + STG2TRT

STAGE1 REGRESSION RESULTS ARE OUTPUT LIKE THIS:INT + MAIN1 + TAILOR1 + TAILOR1*STG1TRT + STG1TRT

Recall our second-stage regression modeldata dat10; set dat1; if R=0; run; * use only non-responders; proc genmod data = dat10; model y = o11c o12c o13c o14c a1 o21c o22 a2 a2*a1 a2*o22;run;

This is how that same model is specified in PROC QLEARN

data dat11; set dat1; S = 1-R; run; * use only non-responders;proc qlearn data=dat11; ...[next slide] main2 o11c o12c o13c o14c o21c; tailor2 a1 o22; stg2sample s; response y; stg2trt a2; ...[next slide]run;

Y = b1 + b2 o11c + b3 o12c + b4 o13c + b5 o14c + b6 o21c + b7 a1 + b8 o22 + b9 a1*a2 + b10 o22*a2 + b11 a2 + error

PROC QLEARN Full Example, ADHD Datadata dat11; set dat1; S = 1-R; run; proc qlearn data=dat11 contrasts1=contrasts1 deriveci; main1 o11c o12c o14c; tailor1 o13; main2 o11c o12c o13c o14c o21c; tailor2 a1 o22; stg2sample s; response y; stg1trt a1; stg2trt a2;run;

This will ask SAS to fit the following two regressions:

Stage 2: Y = b1 + b2 o11c + b3 o12c + b4 o13c + b5 o14c + b6 o21c + b7 a1 + b8 o22+ b9 a1*a2 + b10 o22*a2 +

b11 a2 + error (fit only with non-responders, S=1)

Stage 1: Ŷ = b1 + b2 o11c + b3 o12c + b4 o14c + b5 o13 + b6 o13*a1 + b7 a1 + error

This requests confidence intervals by Laber and Murphy (2012)

This requests contrasts of interest (in GENMOD they are call “estimates”. See next slide for details.

PROC QLEARN: Contrast Matrixdata contrasts1; input M1 M2 M3 M4 M5 M6 M7; *no. of mtrx cols = no. of stage 1 parameters ; * the cols correspond to the parameters in stage 1 model: ; * b1 + b2 o11c + b3 o12c + b4 o14c + b5 o13 + b6 a1 o13 + b7 a1 ; * each row corresponds to a different linear comb of the b's. some linear ; * combos can be used to obtain mean outcomes for different children under ; * initial BMOD vs initial MED. whereas other linear combos can be used to ; * compare mean outcomes between MED vs BMOD for different children. ; datalines; 1 0 0 0 1 1 1 1 0 0 0 1 -1 -1 0 0 0 0 0 2 2 1 0 0 0 0 0 1 1 0 0 0 0 0 -1 0 0 0 0 0 0 2 ; * taking into account optimal future decisions (to intensify vs augment) ; * contrast 1 = mean outcome under BMOD for children w/yes med in prior year; * contrast 2 = mean outcome under MED for children w/yes med in prior year ; * contrast 3 = mean diff (BMOD - MED) for children w/yes med in prior year ; * contrast 4 = mean outcome under BMOD for children w/no med in prior year ; * contrast 5 = mean outcome under MED for children w/no med in prior year ; * contrast 6 = mean diff (BMOD - MED) for children w/no med in prior year ;run;

Try it yourself in SAS• Go to the file:

sas_code_modules_4_5_and_6_ADHD.doc

• Copy the SAS code on Page 14– This code defines the contrast matrix and runs

PROC QLEARN

• Paste into SAS Enhanced Editor window• Press F8 or click the Submit button (the little

running guy)

PROC QLEARN: Results PROC QLEARN -- Q-Learning and Adaptive Confidence Intervals

Number of observations in dataset: 150Number of observations in stage 1: 150Number of observations in stage 2: 99

First Stage Regression Result-------------------------------------------------------------- Parameter Confidence IntervalVariable Estimates Upper Lowerintercept 3.4575 3.7119 3.2135o11c -0.4407 -0.0777 -0.7903o12c -0.3366 -0.1552 -0.5061o14c 0.5650 1.0026 0.1586o13 -0.0418 0.3439 -0.4235o13 :a1 -0.5610 -0.2836 -0.8292a1 0.3104 0.4992 0.0993

Parameter Confidence IntervalContrasts Estimates Upper LowerContrast 1 3.1651 3.6676 2.6437Contrast 2 3.6663 4.0535 3.3291Contrast 3 -0.5012 -0.0032 -1.0399Contrast 4 3.7679 4.0726 3.4328Contrast 5 3.1471 3.4739 2.8384Contrast 6 0.6208 0.9984 0.1986

What did we learn with Q-learning?Adaptive Treatment Strategy Proposal

• If the child used MED in prior year, then begin with MED; otherwise, begin with BMOD.

• If the child is non-responsive and non-adherent to either first-line treatment, then AUGMENT with the other treatment option.

• If the child is non-responsive but adherent to either first-line treatment, then it is better to INTENSIFY first-line treatment.

• If the child is responsive to first-line treatment, then CONTINUE first-line treatment.

This Q-learning analysis was done with simulated/altered data.

What did we learn with Q-learning?Adaptive Treatment Strategy Proposal

• The mean Y, school performance, under the more deeply individualized ATS obtained via Q-learning is estimated to be 3.72.

• As expected, this is larger than the value of the ATS which started with BMOD and augmented with MED for non-responders (mean = 3.51)

• Recall (BMOD, MED) was the ATS with the largest mean among the 4 embedded ATSs.

This Q-learning analysis was done with simulated/altered data.

Citations to Technical Reports

• Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G., Waxmonsky, J., et al. (2010). Q-Learning: A data analysis method for constructing adaptive interventions . Technical Report, The Methodology Center, Penn State University.

• Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, B., Fabiano, G., Waxmonsky, J., Yu, J., & Murphy, S. (2010). Experimental design and primary data analysis for informing sequential decision making processes. Technical Report, The Methodology Center, Penn State University.

Practicum

Autism Exercises: As before, we will go through the Autism Starter File to continue practicing/working through these primary data analyses using the Autism data set.

Extra Slides: For Statistical Aficionados!

In the next two slides, we actually do Steps (ii) and (iii) of Q-Learning by hand. This is

what the PROC QLEARN software automatically does. The issue, however, is that the standard errors here are incorrect.

PROC QLEARN calculates the appropriate standard errors.

SAS code for Step (ii) of Q-Learning: Assign optimal outcome based on Step (i)

data dat11; set dat1; * first, everyone gets their observed outcome; yhat = y; * second, re-assign the outcome for non-responders; if R=0 then

yhat = 3.0039 - 0.2462*o11c - 0.2961*o12c + 0.0391*o13c + 0.4868*o14c + 0.0758*a1 - 0.0097*o21c - 0.0980*o22 + abs(-0.8640*a2 - 0.1934*a1*a2 + 1.1826*o22*a2) ;run;proc means data=dat11; var y yhat; run;

The MEANS Procedure

Variable N Mean Std Dev

-------------------------------------------------

Y 150 2.9533333 1.2814456

yhat 150 3.4107823 0.9385790

-------------------------------------------------


SAS code for Step (iii) of Q-Learning: First-stage treatment tailoring

* Step (iii) regression using the new outcome;proc genmod data=dat11; model yhat = o11c o12c o14c o13 a1*o13 a1; estimate 'BMOD vs MED given MED prior yr ' a1*o13 2 a1 2; estimate 'BMOD vs MED given NO MED prior yr' a1*o13 0 a1 2;run;* medication in the year prior appears to be a tailoring variable ; * however, statistical inferences (p-values, confidence intervals) ;* should not be based on this output. ;


Naïve

95% Conf Limits


BMOD vs MED given MED prior yr -0.5012 -0.9423 -0.0601 0.0259

BMOD vs MED given MED prior yr 0.6208 0.3266 0.9150 <0.001


Documents

Secondary Aims Using Data Arising from a SMART