Objective and Design Issues - British Columbia

Operational Trials:

Objective and Design Issues

Wendy BergerudResearch Branch

BC Min. of ForestsMarch 2003

WAB

Adaptive Management Cycle

Assess

Evaluate

Adjust

Implement

Design

Monitor

WAB

Research trials vs Operational trials

�Operational trials must be designed andimplemented with great care andforethought if results are to be accurateand useful in making better managementdecisions.

�Operational trials are not the researchtrials’ “poor cousin”.

WAB

Research trials vs Operational trials

�Operational trials are designed to test iftreatments “work” when applied underoperational conditions.

�Research trials are designed to findtreatment differences while controlling oraccounting for as many other sources ofvariation as possible.

WAB

“Statistical analysis and interpretation arethe least critical aspects ofexperimentation, in that if purelystatistical or interpretative errors aremade, the data can be reanalyzed. Onthe other hand, the only complete remedyfor design or execution errors isrepetition of the experiment.”

(Hurlbert, 1984, p 189)

WAB

Trial Life Cycle� Steps:� 1) Identify need for study� 2) Design� 3) Establish� 4) Collect data and

maintain site(s)� 5) Analyse data� 6) Communicate results� 7) Wrap-up

� Documentation:�Problem Analysis�Working Plan�Establishment

Report�Project Diary

�Progress and/orFinal Report(s)and Extension

WAB

The Role of Statistics andStatisticians� The statistician’s expertise is particularly

relevant during:� Design of the operational trial (step 2)� Analysis and interpretation of collected data

(steps 5 and 6)

� Research scientists can often provide thisexpertise at a basic level.

WAB

Questions we ask about design:�What questions are we trying to answer

with this trial? What is the objective?� How can we design this trial to answer

these questions and yet do so within thegiven logical and resource restraints?

� What internal and external threats couldundermine our confidence in the finalresults? How can we mitigate against thesethreats?

WAB

Questions we ask about analysis and interpretation:

� How do we analyse this data to answer thequestions?

� What assumptions are necessary to do thisanalysis and make these conclusions?

� And how well do we know theseconclusions? (e.g. confidence limits arounda mean).

WAB

“Statistical designs always involve compromisesbetween the desirable and the possible”.

- Leslie Kish

� Some assumptions and simplificationsmust always be made.

� We must understand their consequences:� Will others agree with them?� How do they weaken our ability to

generalize the results?� How do they weaken any cause and effect

statements that we might like to make?

WAB

Trial Objective� Should be specific and detailed.� Should describe the population under

study.� If looking for differences, should

describe the minimum difference ofpractical importance.

� Consider a range of possible outcomesand what that means for the design anddata analysis and interpretation.

Q1

WAB

Population Definition

� To what population do we want our trialresults to be relevant?� That is, what material do we want to study?

� What population CAN we study? Howdoes this limit our study results?

Q2

WAB

Populations

WAB

What are the treatments andwhat controls will we have?

� Controls are very important fordetermining if treatments have had aneffect.� Consider if controls for spatial and/or

temporal variation are required.

Q3

WAB

Effect Size

� How big must the difference between thetreatments or treatment and control be inorder for us to change our managementpractice?� If our trial can’t detect a difference this small

then our study can’t provide the requiredevidence to change management practice[sample size requirements and power].

Q4

WAB

Consider possible outcomes

� What are some of the possible outcomesand how do they affect the design andpossible data analysis and interpretation?

Q5

WAB

Nuisance variables� What are some of the ‘nuisance’ variables

that will affect our results? For each one,we need to determine whether to:� randomize it out; or� include it in the design as a factor in the study;

or� include it as an independent continuous

variable in the statistical model.

Q6

WAB

Study Unit Definition and Selection� How are we going to select members

from the population to study? Thesemembers are our study units.� Can we select them in a random and/or

unbiased manner??

Q7

WAB

Replication

� How many study units will we use? Doour resources limit the amount ofreplication possible?

� Know how to recognize pseudo-replication.

Q8

WAB

Treatment Assignment

�How will the treatment and ‘nuisance’factors be assigned to the study units?� Can we assign them in a random and

unbiased manner? or� Are they characteristics of the study units

that can only be observed (observationalfactors)?

Q9

WAB

Response Variables

�What variables measure the response weare interested in and how will we measurethem� Can the variables we are interested in be

directly measured or must we use a proxy orsurrogate measure?

Q10

WAB

Subsampling of study units

� Must we subsample the study units toobtain a study unit response or can wedirectly get a response for the whole unit?� If subsampling is required can we use simple

random sampling within each study unit? Ordo we need some more complicatedsampling scheme?

Q11

WAB

Proposed Analysis

� What form of analysis do we expect torun on the data?� What design requirements does this analysis

have?

Q12

WAB

Example: Objective

� Objective is to compare site preparationwith duff-planting.

�Seems like a simple and straightforwardobjective, but is it?

WAB

Example: Setting

� Forest company must manage a largenumber of cutblocks that need to beplanted each year (the population).

� Duff-planting is cheaper than site-prep.� Company would prefer to use site-prep

only when necessary.

WAB

Example: Setting

� Various attributes of cutblocks in thepopulation are known (Factor C).

� Based on these attributes, we know thatsome cutblocks will do just fine withduff-planting and that others will needsite-prep.

� The question applies to those cutblocksin the ‘gray area’.

WAB

Possible Outcomes: for C = 1

C =1 C=2 C=3 C=4 C=5 Factor C

Growth orSurvival � Duff-planting

� Site-prep No real difference in response

WAB

Possible Outcomes: for C = 5


Growth orSurvival � Duff-planting

� Site-prep

Gain usingsite-prep

�

�

WAB

Possible Outcomes: for other C


Growth orSurvival �

�

Question is:

What happensin this grayarea?

�

�

WAB

Example: Objective� How large must the gain be for

management to switch practice fromduff-planting to site-prep?

� You must determine threshold values.�The question we are really interested in

is: Can we use some cutblock attributes topredict when the difference in responsebetween duff-planting and site-prep islarge enough that site-prep is preferred?

WAB

Example: Decision Risks

� Study can help increase percent of ‘right’decisions, but 100% ‘right’ decisions areunattainable.

� Industry might prefer to err by duff-planting unless really necessary.

� Ministry might prefer to err by requiringsite-prep unless loss due to duff-plantingis demonstrated to be negligible.

WAB

Abbreviations:

� RB - Randomized Block

� CR - Completely Randomized

� FRD - Factor Relationship Diagram

� df - Degrees of freedom

WAB

RB Design - 5 “Different” Sites

Factor C:

C=1 C=2 C=3 C=4 C=5

Site 1 Site 2 Site 3 Site 4 Site 5

Design 1

WAB

RB Design - 5 “Different” Sites

� Treatment is replicated.� Factor C is not replicated and is

confounded with site/blocks.� A good “screening” design if nothing is

known about the levels of Factor C.

1

1

A

1

B

2

2

2

A

3

B

4

3

3

A

5

B

6

4

4

A

7

B

8

5

5

A

9

B

10Plot

Treatment

Site/Block

Factor C

Design 1

WAB

Source df Error Source df Test?Factor C 0 B(C)Site/Block B(C) 4 Plots(CBT) Block B 4 --Treatment T 1 T x B(C) Treatment T 1 YesT x C 0 T x B(C)T x B(C) 4 Plots(CBT) T x B 4 --Plots(CBT) 0 --

Analysis of Variance Table

Design 1

� Many sources cannot be estimated butthe most interesting test is available.

Matches the FRD Final Report

WAB

RB Design - 5 “Similar” Sites

Factor C: C=3


Design 2

WAB

RB Design - 5 “Similar” sites

� Treatments are replicated within onelevel of Factor C.

� No information on different levels ofFactor C - inference is limited.

3

1

A

1

B

2

2

A

3

B

4

3

A

5

B

6

4

A

7

B

8

5

A

9

B

10Plot

Treatment

Site/Block

Factor C

Design 2

WAB

Source df Error Source df Test?Factor C 0 B(C)Site/Block B(C) 4 Plots(CBT) Block B 4 --Treatment T 1 T x B(C) Treatment T 1 YesT x C 0 T x B(C)T x B(C) 4 Plots(CBT) T x B 4 --Plots(CBT) 0 --


Design 2

� Looks like Design 1 -- but inferencesapply to only one level of Factor C.

Matches the FRD Final Report

WAB

RB Design - one site

Factor C: C=3

Block 2

Block 1

Block 4 Block 5

Block 3

Design 3

WAB

RB Design - one site

� Pseudo-replicated - treatments arerepeated only within one site.

� Treatment application is replicated butlittle else.

3

1

1

A

1

B

2

2

A

3

B

4

3

A

5

B

6

4

A

7

B

8

5

A

9

B

10Plot

Treatment

Block

Site

Factor C

Design 3

WAB

Source df Error Source df Test?Factor C 0 S(C)Site S(C) 0 B(SC)Block B(SC) 4 Plots(CSBT) Block B 4 --Treatment T 1 T x S(C) Treatment T 1 ?Yes?T x C 0 T x S(C)T x S(C) 0 T x B(SC)T x B(SC) 4 Plots(CSBT) T x B 4 --Plots(CSBT) 0 --


Design 3

� Looks like Designs 1 & 2 -- butinferences are much more limited!

Final ReportMatches the FRD

WAB

Split-plot Design (CR)

Factor C: C=2 C=3


Design 4

WAB

Split-plot Design (CR)

� Factor C and treatment are replicated, buthave different study units.

� Equal replication is ‘nice’ but notessential.

2

1

A

1

B

2

2

A

3

B

4

3

A

5

B

6

3

4

A

7

B

8

5

A

9

B

10Plot

Treatment

Site/Block

Factor C

Design 4

WAB

Source df Error Source df Test?Factor C 1 B(C) Factor C 1 YesSite/Block B(C) 3 Plots(CBT) Block B(C) 3 --Treatment T 1 T x B(C) Treatment T 1 YesT x C 1 T x B(C) T x C 1 YesT x B(C) 3 Plots(CBT) T x B(C) 3 --Plots(CBT) 0 --


Design 4

� More interesting tests are now available.

Final ReportMatching FRD

WAB

Blocked Study Units

� Previous designs have arranged thetreatments together into ‘blocks’.

� This is desirable if:� Treatments can be applied to portions of

cutblocks; and� Within cutblock variability is less than

between cutblock variability so thattreatment comparisons will be more precise.

WAB

Separate Study Units

� Treatments may be applied to wholecutblocks if:� Operationally, treatments can only be

applied to large areas; and/or� Within cutblock variability is at least as

great as between cutblock variability(not a common assumption!).

� This leads us to Completely Randomized(CR) Designs.

WAB

CR Design - 10 “Different” Sites

C=1 C=2 C=3 C=4 C=5

Sites 3 & 4 Sites 7 & 8 Sites 1 & 2 Sites 5 & 6

Sites 9 & 10

Factor C:

Design 5

WAB

CR Design - 10 “Different” sites

� Treatments are replicated.� Factor C is minimally replicated.� Analysis may need to assume no

interaction between the treatment and CDesign 5

1

A

1

1

B

2

2

2

A

3

3

B

4

4

3

A

5

5

B

6

6

4

A

7

7

B

8

8

5

A

9

9

B

10

10Plot

Site

Treatment

Factor C

WAB

Source df Error Source df Test?Factor C 4 S(CT) Factor C 4 Yes?Treatment T 1 S(CT) Treatment T 1 Yes?T x C 0 S(CT)Site S(CT) 4 Plots(CBT) Site S(CT) 4 --Plots(CBT) 0 --


Design 5

� Test for Factor C assumes no interactionbetween the treatment and C.

� Test for treatment okay if Factor C is‘random’.


WAB

CR Design - 10 “Similar” Sites

Factor C: C=3

Sites 3, 4 & 5 Sites 8, 9 & 10 Sites 1 & 2 Sites 6 & 7

Design 6

WAB

CR Design - 10 “Similar” Sites

� Treatment is replicated.� No information on different levels of

Factor C - inference is limited.

Design 6

3

A

1

1

2

2

3

3

4

4

5

5

B

6

6

7

7

8

8

9

9

10

10Plot

Site

Treatment

Factor C

WAB

Source df Error Source df Test?Factor C 0 S(CT)Treatment T 1 S(CT) Treatment T 1 YesT x C 0 S(CT)Site S(CT) 8 Plots(CBT) Site S(CT) 8 --Plots(CBT) 0 --


Design 6

� Simple One-way Design, but inferencesapply to only one level of Factor C.


WAB

CR Design - one siteFactor C: C=3

Design 7

WAB

CR Design - one site

� Pseudo-replicated - treatments arerepeated only within one site.

� Treatment application is replicated butlittle else.

3

1

A

1 2 3 4 5

B

6 7 8 9 10Plot

Treatment

Site/Block

Factor C

Design 7

WAB


Design 7

� Appears to be a simple One-way Designbut uses pseudo-replication.

Final ReportMatching FRDSource df Error Source df Test?Factor C 0 S(C)Site S(C) 0 Plots(CBT)Treatment T 1 T x S(C) Treatment T 1 YesT x C 0 T x S(C)T x S(C) 8 Plots(CBT) Site S(CT) 8 --Plots(CBT) 0 --

WAB

CR Design

Sites 3, 4, 5 & 6 Sites 7, 8, 9 & 10 Sites 1 & 2

Factor C: C=2 C=3

Design 8

WAB

CR Design

� Factor C and treatment are replicated.� Equal replication is ‘nice’ but not

essential.Design 8

2

A

1

1

2

2

3

3

B

4

4

5

5

6

6

3

A

7

7

8

8

9

9

B

10

10Plot

Site

Treatment

Factor C

WAB


Design 8

� Factorial Two-way Design with realreplication.

� Many interesting tests.

Final ReportMatching FRDSource df Error Source df Test?Factor C 1 S(CT) Factor C 1 YesTreatment T 1 S(CT) Treatment T 1 YesT x C 1 S(CT) T x C 1 YesSite S(CT) 6 Plots(CST) Site S(CT) 6 --Plots(CST) 0 --

WAB

Conclusions

� The objective and design provide thefoundation for the trial - if poorly done,the trial has little chance of succeeding.

� There is no one design that is always“best”. The “best” design depends uponthe objective.

WAB

Conclusions

�Get good statistical advice early, duringthe design.

� Consider how the assumptions andsimplifications you’ve had to make affectthe strength and applicability of yourtrial’s conclusions.

WAB

Documents

Objective and Design Issues - British Columbia