Operational Trials:
Objective and Design Issues
Wendy BergerudResearch Branch
BC Min. of ForestsMarch 2003
WAB
Research trials vs Operational trials
�Operational trials must be designed andimplemented with great care andforethought if results are to be accurateand useful in making better managementdecisions.
�Operational trials are not the researchtrials’ “poor cousin”.
WAB
Research trials vs Operational trials
�Operational trials are designed to test iftreatments “work” when applied underoperational conditions.
�Research trials are designed to findtreatment differences while controlling oraccounting for as many other sources ofvariation as possible.
WAB
“Statistical analysis and interpretation arethe least critical aspects ofexperimentation, in that if purelystatistical or interpretative errors aremade, the data can be reanalyzed. Onthe other hand, the only complete remedyfor design or execution errors isrepetition of the experiment.”
(Hurlbert, 1984, p 189)
WAB
Trial Life Cycle� Steps:� 1) Identify need for study� 2) Design� 3) Establish� 4) Collect data and
maintain site(s)� 5) Analyse data� 6) Communicate results� 7) Wrap-up
� Documentation:�Problem Analysis�Working Plan�Establishment
Report�Project Diary
�Progress and/orFinal Report(s)and Extension
WAB
The Role of Statistics andStatisticians� The statistician’s expertise is particularly
relevant during:� Design of the operational trial (step 2)� Analysis and interpretation of collected data
(steps 5 and 6)
� Research scientists can often provide thisexpertise at a basic level.
WAB
Questions we ask about design:�What questions are we trying to answer
with this trial? What is the objective?� How can we design this trial to answer
these questions and yet do so within thegiven logical and resource restraints?
� What internal and external threats couldundermine our confidence in the finalresults? How can we mitigate against thesethreats?
WAB
Questions we ask about analysis and interpretation:
� How do we analyse this data to answer thequestions?
� What assumptions are necessary to do thisanalysis and make these conclusions?
� And how well do we know theseconclusions? (e.g. confidence limits arounda mean).
WAB
“Statistical designs always involve compromisesbetween the desirable and the possible”.
- Leslie Kish
� Some assumptions and simplificationsmust always be made.
� We must understand their consequences:� Will others agree with them?� How do they weaken our ability to
generalize the results?� How do they weaken any cause and effect
statements that we might like to make?
WAB
Trial Objective� Should be specific and detailed.� Should describe the population under
study.� If looking for differences, should
describe the minimum difference ofpractical importance.
� Consider a range of possible outcomesand what that means for the design anddata analysis and interpretation.
Q1
WAB
Population Definition
� To what population do we want our trialresults to be relevant?� That is, what material do we want to study?
� What population CAN we study? Howdoes this limit our study results?
Q2
WAB
What are the treatments andwhat controls will we have?
� Controls are very important fordetermining if treatments have had aneffect.� Consider if controls for spatial and/or
temporal variation are required.
Q3
WAB
Effect Size
� How big must the difference between thetreatments or treatment and control be inorder for us to change our managementpractice?� If our trial can’t detect a difference this small
then our study can’t provide the requiredevidence to change management practice[sample size requirements and power].
Q4
WAB
Consider possible outcomes
� What are some of the possible outcomesand how do they affect the design andpossible data analysis and interpretation?
Q5
WAB
Nuisance variables� What are some of the ‘nuisance’ variables
that will affect our results? For each one,we need to determine whether to:� randomize it out; or� include it in the design as a factor in the study;
or� include it as an independent continuous
variable in the statistical model.
Q6
WAB
Study Unit Definition and Selection� How are we going to select members
from the population to study? Thesemembers are our study units.� Can we select them in a random and/or
unbiased manner??
Q7
WAB
Replication
� How many study units will we use? Doour resources limit the amount ofreplication possible?
� Know how to recognize pseudo-replication.
Q8
WAB
Treatment Assignment
�How will the treatment and ‘nuisance’factors be assigned to the study units?� Can we assign them in a random and
unbiased manner? or� Are they characteristics of the study units
that can only be observed (observationalfactors)?
Q9
WAB
Response Variables
�What variables measure the response weare interested in and how will we measurethem� Can the variables we are interested in be
directly measured or must we use a proxy orsurrogate measure?
Q10
WAB
Subsampling of study units
� Must we subsample the study units toobtain a study unit response or can wedirectly get a response for the whole unit?� If subsampling is required can we use simple
random sampling within each study unit? Ordo we need some more complicatedsampling scheme?
Q11
WAB
Proposed Analysis
� What form of analysis do we expect torun on the data?� What design requirements does this analysis
have?
Q12
WAB
Example: Objective
� Objective is to compare site preparationwith duff-planting.
�Seems like a simple and straightforwardobjective, but is it?
WAB
Example: Setting
� Forest company must manage a largenumber of cutblocks that need to beplanted each year (the population).
� Duff-planting is cheaper than site-prep.� Company would prefer to use site-prep
only when necessary.
WAB
Example: Setting
� Various attributes of cutblocks in thepopulation are known (Factor C).
� Based on these attributes, we know thatsome cutblocks will do just fine withduff-planting and that others will needsite-prep.
� The question applies to those cutblocksin the ‘gray area’.
WAB
Possible Outcomes: for C = 1
C =1 C=2 C=3 C=4 C=5 Factor C
Growth orSurvival � Duff-planting
� Site-prep No real difference in response
WAB
Possible Outcomes: for C = 5
C =1 C=2 C=3 C=4 C=5 Factor C
Growth orSurvival � Duff-planting
� Site-prep
Gain usingsite-prep
�
�
WAB
Possible Outcomes: for other C
C =1 C=2 C=3 C=4 C=5 Factor C
Growth orSurvival �
�
Question is:
What happensin this grayarea?
�
�
WAB
Example: Objective� How large must the gain be for
management to switch practice fromduff-planting to site-prep?
� You must determine threshold values.�The question we are really interested in
is: Can we use some cutblock attributes topredict when the difference in responsebetween duff-planting and site-prep islarge enough that site-prep is preferred?
WAB
Example: Decision Risks
� Study can help increase percent of ‘right’decisions, but 100% ‘right’ decisions areunattainable.
� Industry might prefer to err by duff-planting unless really necessary.
� Ministry might prefer to err by requiringsite-prep unless loss due to duff-plantingis demonstrated to be negligible.
WAB
Abbreviations:
� RB - Randomized Block
� CR - Completely Randomized
� FRD - Factor Relationship Diagram
� df - Degrees of freedom
WAB
RB Design - 5 “Different” Sites
Factor C:
C=1 C=2 C=3 C=4 C=5
Site 1 Site 2 Site 3 Site 4 Site 5
Design 1
WAB
RB Design - 5 “Different” Sites
� Treatment is replicated.� Factor C is not replicated and is
confounded with site/blocks.� A good “screening” design if nothing is
known about the levels of Factor C.
1
1
A
1
B
2
2
2
A
3
B
4
3
3
A
5
B
6
4
4
A
7
B
8
5
5
A
9
B
10Plot
Treatment
Site/Block
Factor C
Design 1
WAB
Source df Error Source df Test?Factor C 0 B(C)Site/Block B(C) 4 Plots(CBT) Block B 4 --Treatment T 1 T x B(C) Treatment T 1 YesT x C 0 T x B(C)T x B(C) 4 Plots(CBT) T x B 4 --Plots(CBT) 0 --
Analysis of Variance Table
Design 1
� Many sources cannot be estimated butthe most interesting test is available.
Matches the FRD Final Report
WAB
RB Design - 5 “Similar” sites
� Treatments are replicated within onelevel of Factor C.
� No information on different levels ofFactor C - inference is limited.
3
1
A
1
B
2
2
A
3
B
4
3
A
5
B
6
4
A
7
B
8
5
A
9
B
10Plot
Treatment
Site/Block
Factor C
Design 2
WAB
Source df Error Source df Test?Factor C 0 B(C)Site/Block B(C) 4 Plots(CBT) Block B 4 --Treatment T 1 T x B(C) Treatment T 1 YesT x C 0 T x B(C)T x B(C) 4 Plots(CBT) T x B 4 --Plots(CBT) 0 --
Analysis of Variance Table
Design 2
� Looks like Design 1 -- but inferencesapply to only one level of Factor C.
Matches the FRD Final Report
WAB
RB Design - one site
� Pseudo-replicated - treatments arerepeated only within one site.
� Treatment application is replicated butlittle else.
3
1
1
A
1
B
2
2
A
3
B
4
3
A
5
B
6
4
A
7
B
8
5
A
9
B
10Plot
Treatment
Block
Site
Factor C
Design 3
WAB
Source df Error Source df Test?Factor C 0 S(C)Site S(C) 0 B(SC)Block B(SC) 4 Plots(CSBT) Block B 4 --Treatment T 1 T x S(C) Treatment T 1 ?Yes?T x C 0 T x S(C)T x S(C) 0 T x B(SC)T x B(SC) 4 Plots(CSBT) T x B 4 --Plots(CSBT) 0 --
Analysis of Variance Table
Design 3
� Looks like Designs 1 & 2 -- butinferences are much more limited!
Final ReportMatches the FRD
WAB
Split-plot Design (CR)
� Factor C and treatment are replicated, buthave different study units.
� Equal replication is ‘nice’ but notessential.
2
1
A
1
B
2
2
A
3
B
4
3
A
5
B
6
3
4
A
7
B
8
5
A
9
B
10Plot
Treatment
Site/Block
Factor C
Design 4
WAB
Source df Error Source df Test?Factor C 1 B(C) Factor C 1 YesSite/Block B(C) 3 Plots(CBT) Block B(C) 3 --Treatment T 1 T x B(C) Treatment T 1 YesT x C 1 T x B(C) T x C 1 YesT x B(C) 3 Plots(CBT) T x B(C) 3 --Plots(CBT) 0 --
Analysis of Variance Table
Design 4
� More interesting tests are now available.
Final ReportMatching FRD
WAB
Blocked Study Units
� Previous designs have arranged thetreatments together into ‘blocks’.
� This is desirable if:� Treatments can be applied to portions of
cutblocks; and� Within cutblock variability is less than
between cutblock variability so thattreatment comparisons will be more precise.
WAB
Separate Study Units
� Treatments may be applied to wholecutblocks if:� Operationally, treatments can only be
applied to large areas; and/or� Within cutblock variability is at least as
great as between cutblock variability(not a common assumption!).
� This leads us to Completely Randomized(CR) Designs.
WAB
CR Design - 10 “Different” Sites
C=1 C=2 C=3 C=4 C=5
Sites 3 & 4 Sites 7 & 8 Sites 1 & 2 Sites 5 & 6
Sites 9 & 10
Factor C:
Design 5
WAB
CR Design - 10 “Different” sites
� Treatments are replicated.� Factor C is minimally replicated.� Analysis may need to assume no
interaction between the treatment and CDesign 5
1
A
1
1
B
2
2
2
A
3
3
B
4
4
3
A
5
5
B
6
6
4
A
7
7
B
8
8
5
A
9
9
B
10
10Plot
Site
Treatment
Factor C
WAB
Source df Error Source df Test?Factor C 4 S(CT) Factor C 4 Yes?Treatment T 1 S(CT) Treatment T 1 Yes?T x C 0 S(CT)Site S(CT) 4 Plots(CBT) Site S(CT) 4 --Plots(CBT) 0 --
Analysis of Variance Table
Design 5
� Test for Factor C assumes no interactionbetween the treatment and C.
� Test for treatment okay if Factor C is‘random’.
Final ReportMatching FRD
WAB
CR Design - 10 “Similar” Sites
Factor C: C=3
Sites 3, 4 & 5 Sites 8, 9 & 10 Sites 1 & 2 Sites 6 & 7
Design 6
WAB
CR Design - 10 “Similar” Sites
� Treatment is replicated.� No information on different levels of
Factor C - inference is limited.
Design 6
3
A
1
1
2
2
3
3
4
4
5
5
B
6
6
7
7
8
8
9
9
10
10Plot
Site
Treatment
Factor C
WAB
Source df Error Source df Test?Factor C 0 S(CT)Treatment T 1 S(CT) Treatment T 1 YesT x C 0 S(CT)Site S(CT) 8 Plots(CBT) Site S(CT) 8 --Plots(CBT) 0 --
Analysis of Variance Table
Design 6
� Simple One-way Design, but inferencesapply to only one level of Factor C.
Final ReportMatching FRD
WAB
CR Design - one site
� Pseudo-replicated - treatments arerepeated only within one site.
� Treatment application is replicated butlittle else.
3
1
A
1 2 3 4 5
B
6 7 8 9 10Plot
Treatment
Site/Block
Factor C
Design 7
WAB
Analysis of Variance Table
Design 7
� Appears to be a simple One-way Designbut uses pseudo-replication.
Final ReportMatching FRDSource df Error Source df Test?Factor C 0 S(C)Site S(C) 0 Plots(CBT)Treatment T 1 T x S(C) Treatment T 1 YesT x C 0 T x S(C)T x S(C) 8 Plots(CBT) Site S(CT) 8 --Plots(CBT) 0 --
WAB
CR Design
� Factor C and treatment are replicated.� Equal replication is ‘nice’ but not
essential.Design 8
2
A
1
1
2
2
3
3
B
4
4
5
5
6
6
3
A
7
7
8
8
9
9
B
10
10Plot
Site
Treatment
Factor C
WAB
Analysis of Variance Table
Design 8
� Factorial Two-way Design with realreplication.
� Many interesting tests.
Final ReportMatching FRDSource df Error Source df Test?Factor C 1 S(CT) Factor C 1 YesTreatment T 1 S(CT) Treatment T 1 YesT x C 1 S(CT) T x C 1 YesSite S(CT) 6 Plots(CST) Site S(CT) 6 --Plots(CST) 0 --
WAB
Conclusions
� The objective and design provide thefoundation for the trial - if poorly done,the trial has little chance of succeeding.
� There is no one design that is always“best”. The “best” design depends uponthe objective.
WAB
Conclusions
�Get good statistical advice early, duringthe design.
� Consider how the assumptions andsimplifications you’ve had to make affectthe strength and applicability of yourtrial’s conclusions.