Upload
anna-liddle
View
127
Download
0
Embed Size (px)
Citation preview
Brunel University London
2nd International Symposium on
Factorial Survey Methods in Health, Social and
Ageing Research
Wednesday 15th June 2016
Darwin Room, Hamilton Centre,
Brunel University London
Hosted by:
Professor Ken Gilhooly, Brunel University London
Brunel University London
Contents
1. Using the factorial survey design to study individual judgement
Prof Brian Taylor, Ulster University
2. Using a fractional factorial survey to develop an effective training tool to
improve fitness-to-drive decisions
Prof Cilla Harries, Brunel University London
3. Using a fractional factorial survey to develop an effective training tool to
improve dietetic referral prioritisations
Dr Hulya Gokalp, Brunel University London
4. The importance of work willingness in social assistance eligibility
assessments
Dr Marjolijn de Wilde, University of Antwerp
5. Administration of an online factorial survey using Qualtrics software
Helena McElhinney, Ulster University (contact details only)
6. Synthesis of three data sources to determine independent variables in an online
factorial survey of implantable cardioverter defibrillator deactivation decisions
Loreena Hill, Ulster University
7. A tool that produces judgment study stimulus case sets with specified cue-cue
correlations
Dr Robert Hamm, University of Oklahoma Health Sciences Center
Using the factorial survey
design to study judgement
Brian Taylor
Ulster University
Northern Ireland
Factorial Survey - Essence
Present a random sample of respondents
with a random selection of vignettes
with randomised characteristics
that represent familiar type of situations
to make a familiar type of judgement;
analyse results to measure effect of
vignette factors on the judgements
of this type of respondent
Juni 2016 ISFS 4
Judgements vs decisions
Judgment = the considered evaluation of
evidence by an individual using their cognitive
faculties so as to reach an opinion on a preferred
course of action based on available information,
knowledge and values. (Taylor, 2013, p179)
Decision = a conscious process (individually or
with one or more others) leading to the selection
of a course of action from among two or more
alternatives (Taylor, 2013, p178)
Taylor BJ (2013) Professional Decision Making
and Risk in Social Work (2nd ed) London: Sage
Juni 2016 ISFS 5
Approaches to Studying Judgements
Normative
= how people OUGHT to make judgements (to be ‘rational’)
Descriptive
= how people DO make judgements in reality - FS
Juni 2016 ISFS 6
FS=studying decision inputs & output
IN = factors considered, context, etc
Box = respondent cognitive process
OUT = judgement to be made
NB can incorporate respondent
characteristics in FS but this is not a strength
Juni 2016 ISFS 7
JudgementIN OUT
Basic factorial survey research
question:
What is the size of effect of factors
<X, Y, Z> [IVs]
on judgements by
<type of respondent>
regarding
<judgement to be made>? [DV]
Juni 2016 ISFS 8
Vignettes, Case Scenarios, Paper Cases
BASIC
QUESTION
for
respondent:
What would
you do in this
(familiar type
of) situation?
Juni 2016 ISFS 9
What factors to use within vignettes?
Identify factors from
previous research AND/OR
preliminary quaLitative study AND/OR
an expert group
> Factors of most interest YET
realistic vignettes for respondents
> Select for hypotheses in analysis
Juni 2016 ISFS 10
What levels of each vignette factor?
Interval variables
Ordinal variables
Nominal variables
Measurable effect on this judgement
Realistic levels for respondents
Meaningful anchors at end of scales
Pilot variables & levels
Juni 2016 ISFS 11
Vignette structure – IVs- example
Create one standard series of sentences to
encapsulate the dimensions of interest
Include one level of each element in the
sentence in each vignette e.g.
Mrs Smith is <70, 75, 80, 85, 90, 95>. Her
mobility level is <bedfast, uses a zimmer
frame, ‘furniture walks’, needs supervision,
fully mobile unaided>. etc etc
Juni 2016 ISFS 12
Client, Family AND Service Variables
Son visits <daily, weekly, monthly, never>
Home care services are currently provided for
<1,2,3,4,5,6,7> days per week
The Day Centre staff are <very stressed,
stressed, able for new admissions, eager to
redress falling enrolment>
NB variables must be familiar to
respondents and realistic to them
Juni 2016 ISFS 13
Juni 2016 ISFS 14
Independent Variable Levels No.
Levels
Source
Age Under 16 yrs. old
17-20 yrs. old
In her late 20’s
3 Focus Group 1,
FG 2, FG3, FG4,
article A
Drug Use Has never taken illegal drugs
Takes prescribed methadone
Takes illegal drugs
3 FG1, FG2, FG3,
FG4
A, P
Domestic Violence Has a supportive partner
Feels afraid at home
2 FG1, FG2, FG3,
FG4
W, A, P, M
Alcohol use Does not drink alcohol
Drinks 1-2 units per week
Drinks 5-6 units per week
3 FG1, FG3, FG4
P
Demonstrating source of IVs & Levels
(extract: Helena Mc Elhinney)
Vignette structure
– Dependent Variables Create 1, 2, or 3 standard judgements to be
made at the end of each vignette:
E.g. choose one of: “admit to nursing or
residential home”, “provide homecare”, “refer
for further assessment”, “send letter
regretting refusal of service”
E.g. rate likelihood of admission to
institutional care during next 12 months on a
scale of 1 to 100
Juni 2016 ISFS 15
The Question(s) = DVs
Juni 2016 ISFS 16
To what extent do you perceive there to be a risk of harm to the unborn child?No Harm 0 1 2 3 4 5 6 7 8 9 Significant Harm
To what extent do you think a child safeguarding referral should be made?No 0 1 2 3 4 5 6 7 8 9 Yes
If referred, how confident are you that a there will be a good outcome to this case?Not Confident 0 1 2 3 4 5 6 7 8 9 Very Confident
Categorical, Ordinal and Interval
Independent Variables
Nominal (categories)
Co-morbidity;
Ordinal (ordered)
Activities of Daily Living - level;
Interval (measured)
Number of ICD shocks; score on MMSE;
number of hospital admissions
Juni 2016 ISFS 17
Categorical, Ordinal and Interval
Dependent Variables
Nominal (categories)
Refer for investigation or not
Ordinal (ordered)
Outcomes in accepted order of severity
Interval (measured)
Likelihood to discuss de-activation of ICD;
estimate of likelihood of X occurring in
time period Y
Juni 2016 ISFS 18
How many vignettes per respondent
Typically 15 – 20
May be 6 to 40
Consider
respondent fatigue
Will they still be
taking vignettes
seriously at the
end?!
Check in piloting
Juni 2016 ISFS 19
Factorial DesignS Research methods textbooks that include
factorial designs normally include only:
what I call Factorial Experiments
(called Factorial Designs)
DO NOT YET USUALLY INCLUDE:
Factorial Surveys
= another type of Factorial Design
NB neither of these are factor analysis!
Juni 2016 ISFS 20
Use Factorial Design (Factorial
Survey OR Factorial Experiment)
Interested in two or more variables
Context means variables cannot be separated
We are interested in interactions
E.g. chronic illness (IV) and age (IV) on need for health & social care service (DV)
E.g. age at admission of child to state care (IV) and length of time in care (IV) on educational attainment (DV)
Factorial Design (Factorial Survey OR
Factorial Experiment)
Juni 2016 ISFS 22
Mobility limited A B
Mobility not limited C D
Cognitive functionMobility
Alert Confused
DV: Would you recommend client as able
to live in sheltered housing arrangement?
Independent Variables
More Complex: 3*4 Factorial Design
Juni 2016 ISFS 23
Shaken vigorously A B C
Struck with stick D E F
Banged on wall while shaking G H J
Slapped on face K L M
Child’s AgeAbuse
0 - 5 years 6 - 10 years 11 - 15 years
DV: Is a child protection investigation appropriate? -
adapted from O'Toole R, Webster SW, O'Toole AW & Lucal B
(1999) Teachers' recognition and reporting of child abuse: A
factorial survey. Child Abuse and Neglect, 23(11), 1083-101
Factorial Experiments compared to Factorial Surveys
In a factorial experiment, the number of
vignette factors in EACH cell of the above
tables must be sufficient to allow us to
measure the effect of each variable on every
other variable
HENCE in a factorial experiment
vignette must be (unrealistically) simple to
give enough responses for each combination
Juni 2016 ISFS 24
Factorial Survey - external validity
through randomisation of factors
The factorial survey, in contrast, randomises
the selection of factors within the vignettes
HENCE
It is not necessary to have large numbers for
every possible combination
HENCE
Vignettes may be made more realistic by
including more factors & more levels
Juni 2016 ISFS 25
Eg. IVs in Factorial Experiment
Social workers’ support for older person’s
autonomy (DV) in relation to 3 factors (IVs):
Cognitive status [2 levels]
Care status [2 levels]
Caregiver burden [2 levels]
NB factorial experiments allow modelling
of the judgements of individuals, unlike FSTara Healy (1998) ‘The complexity of everyday ethics in home health
are: An analysis of social workers’ decisions regarding frail elders’
autonomy’, Social Work in Health Care, 27(4): 19-37
Juni 2016 ISFS 26
Eg. IVs in Factorial Survey
DV: Long term care plan in older age
3 factors with 2 levels
15 factors with 3 levels
3 factors with 4 levels
2 factors with 5 levels
1 factor with 6 levels
> 1,099,496,032,600 possible vignettes!
Catherine Hagan Hennessy (1993) ‘Modeling case
management decision-making in a consolidated long-term
care program’, The Gerontologist, 33(3): 333-41
Juni 2016 ISFS 27
Basic Analysis Regression measures effect of significant IVs
Need measure independent of sample size
Pearson’s R2 = amount of variance in the
decision that is attributable to that factor
One-way ANOVA to measure size of
relationship between each IV & DV in turn
Can use pairs of two-way ANOVA to establish
if there are first-order interaction effects
>Taylor BJ & Zeller RA (2007) Getting robust and valid data
on decision policies: The factorial survey. Irish Journal of
Psychology, 28(1-2), 27-42
Juni 2016 ISFS 28
Refinements
Initial 2-3 standard vignettes to acclimatise
respondent and to control for tendency to
score high or low before randomised ones
These may be chosen as ~low/medium/high
Repeat standard vignette - consistency
Ask to describe ‘a recent case’ to validate
external validity of vignettes
Questionnaire to give demographic data on
respondents – can be combined with vignette
data
Juni 2016 ISFS 29
Can combine levels of data:
Juni 2016 ISFS 30
Organisation Data
Respondent Data
Vignette Data
BUT hierarchical regression may lose more than is
gained! Recommend standard regression >
Taylor BJ & Zeller RA (2007) Getting robust and valid
data on decision policies: The factorial survey. Irish
Journal of Psychology, 28(1-2), 27-42
Independence of Decisions
If we can assume for statistical purposes that
the vignette decisions may all be treated as
independent even though each respondent
has completed several
Then can use standard regression, giving
Much greater scope to combine the results
from different studies
Can demonstrate error with Monte Carlo Analysis
Juni 2016 ISFS 31
Factorial Survey – Advantages
High internal validity of experimental method
because vignette variables are random and
are controlled by the researcher
High external validity of survey method
because vignettes may be realistic in terms
of complexity of factors
Efficiency of survey method to access large
numbers giving robust results
Juni 2016 ISFS 32
Prospects for Factorial Surveys
Computers greatly simplify creation of
vignettes with randomised characteristics
Need to develop more efficient process of
data extraction & web-based administration
Developing alternative presentations
visual, auditory, referral form, typical presentation
Develop time-sequence judgements?
Juni 2016 ISFS 33
34
Theory: What is Judgement Analysis?
(its) “.. use as a common descriptor for any paradigm employing multiple regression equations to model human judgement is now widespread.”*
Tenets:
human judgement is probabilistic &
can be expressed as a linear model**
*Ray Cooksey (1996) Judgment Analysis: Theory, Methods &
Applications. San Diego, CA: Academic Press (p. xi)
**Robert S Wigton (1988) Use of linear models to analyze
physicians’ decisions. Medical Decision Making, 8, 241-252
35
Applying Brunswik Lens ModelBrunswik E (1952) The Conceptual Framework of Psychology, Chicago:
University of Chicago
REALITY
CUE 1
CUE 2
CUE 3
JUDGE
Observed Cues
Cues can be modelled egFS study?!
Cues can be
modelled
eg risk factors
Factorial Surveys in relation to
Brunswik’s key points
1. Probabilistic view of the world
vs then dominant deterministic paradigm
2. Representative design
= real life not dominant laboratory focus
3. Focus on individuals
Factorial Surveys fit the first two
points but not the third
36
37
What is missing from JA?
Judgement Analysis does not include explicitly
the concept of modelling (with multiple
regression) dispersed ‘groups’ (= types) of
people such as in Factorial Surveys
JA either samples the whole universe of
possible vignettes (full factorial) or cases
selected from reality (‘representative’ in sense
of real-life, not laboratory-based)
JA currently lacks the concept of random
selection from the full factor(ial) universe
References Origins:
Rossi PH & Nock SL (1982) Measuring Social
Judgments – The Factorial Survey Approach.
Beverley Hills, California, Sage
A straightforward ‘how to’ description:
Taylor BJ (2006) Factorial surveys: Using
vignettes to study professional judgement. British
Journal of Social Work, 36(7), 1187-1207
Detailed recent text by Katrin & Thomas:
Auspurg K & Hinz T (2014) Factorial Survey
Experiments. London: Sage [with web resources]
Juni 2016 ISFS 38THE END
Professor Priscilla Harries
Professor Carolyn Unsworth
Dr. Hulya Gokalp
Dr. Miranda Davies
Educating Novice
Occupational Therapy Driver
Assessors To Make Fitness-
to-drive Recommendations
For Older and / or Disabled
Clients
Driving is an important and valued
IADL. It promotes independence, mobility & freedom
Health conditions and disability can impair driving
Occupational therapists are well placed to assess fitness-to-drive, and usually conduct off & on-road assessments.
It is not clear what judgement policies OTDAs are using when making fitness-to-drive recommendations
Evidence based training is needed to enhance workforce capacity (Classen, 2010; Kortling & Kaptein 1996; Unsworth, 2007)
Background
Project advisors:
Service user group
Tamalina Al-Dakkak
Ann Bunce
Jon and Rodney Hutchings
Judith Sinclair
Hilary Strickland
Project advisory group
Professor Mary Gilhooly (Brunel University London)
Professor Mark Williams (Brunel University London)
Professor Peter Ayton (City University)
Professor Richard Tay (La Trobe University)
Dr Catherine Lant (Rosemead GP surgery)
Experienced occupational therapy driver
assessors
Meg Marmo (Australia)
Jenny Wise (Australia)
Kirsty Flint (UK)
Objectives 1-3
Develop a consensus policy on fitness-to-drive from experts.
Method
Results
Objectives 3-6
Use consensus policy in RCT to determine if we can train novices and then develop decision aid.
Method
Results
Structure of the presentation
Task 1Case scenarios
Task 3Training Package
Task 2Policy Capturing
Study
Decision
Training
Aid
Study Design
Task 4RCT
Task 5Training aid Website
Task 5Training aid Website
1 Create ‘driving’ case scenarios of older people and people with disabilities who want to learn to drive, or resume driving for assessment;
2 Identify what fitness-to-drive (fit, not fit) decisions are made;
3 Statistically model and obtain consensus on how experienced occupational therapy driver assessors make optimal fitness-to-drive decisions for older people and people with disabilities who want to learn to drive, or resume driving;
Objectives
In relation to Objectives 1-3
1.
Case background (same for each case)
The client you are assessing, who is a private car driver, lives on
the outskirts of the city / metropolitan area. Their doctor, who
would like them to have a comprehensive driver assessment, has
referred them to you. They are suitable for on-road assessment as
they are medically fit to drive, they do not have unilateral spatial
neglect (lack of awareness of one side) and they meet the legal
vision standards for driving. Initially you undertake a client-based
assessment using a battery of tests in the clinic e.g. range of
movement testing, pain questionnaire and medication screen, etc.
You decide to proceed to an on-road assessment, which involves
you taking the client out, with a driving instructor in a dual-
controlled car, in an area not familiar to them. You sit in the back to
observe their actual driving behaviour and are going to make your
final recommendation from the findings of the on-road assessment.
The results of the on-road assessment are as shown below.
Cue Client Cue Level
Age 60 years old
Driving experience Client has been driving 3-7 years
Driving history Client has had a few minor scrapes in the last 12-months
Current driving needs Client drives predominantly in the local / familiar area
Physical Skills Physical skills support safe driving
Cognitive & / or perceptual skills
Minor cognitive &/or perceptual problems identified but demonstrates capacity for learning & improvement
Sensory functions Sensory functions support safe driving
Driver behaviour Some behaviour problems identified
Road law knowledge & / or road craft
Road law knowledge and / or road craft support safe driving
Vehicle handling skills Vehicle handling supports safe driving
Driving instructor interventions Driving instructor provides one physical intervention
Medical prognosis Medically stable
Fit-to-drive– Unrestricted licence
Fit-to-drive– With conditions. For example, using an automatic car.
Not fit-to-drive– Driver rehabilitation to be completed (may require reassessment)
Not fit-to-drive– Suspend or cancel licence.
What is your recommendation for this client?
Please click on one of the boxes below to make
your recommendation:
Social Judgment Theory method
45 experienced occupational therapy driver assessors from the UK, Australasia
Mean number of years they worked as an occupational therapist was 21, with a mean of 11 years’ experience working as a driving assessor.
Methods
Fitness-to-drive recommendations
made for a series of 64 case scenarios, on-line;
Data were analysed using discriminant function analysis and an intraclass correlation coefficient;
ICC (type 2,1) was used to determine the level of agreement between decisions made by OTDAs.
Results
Recommendation Number of cases
Not fit-to-drive: Suspend or cancel licence
569 (20%)
Not fit-to-drive – driver rehabilitation to be completed
1529 (53%)
Fit-to-drive: With conditions
415 (14%)
Fit-to-drive: Unrestricted licence
367 (13%)
Total 64 scenarios x 45 OTDAs
2880
Discriminant function analysis.
Information central to fitness-to-drive recommendations are:
physical skills,
cognitive and perceptual skills,
road law craft skills,
vehicle handling skills
the number of driving instructor interventions.
Results How do OTDAs weight different types of information when making fitness-to-drive recommendations?
The three functions produced by the discriminant function analysis, showing the correlations between cues and the fitness-to-drive recommendation (Structure Matrix).
Functions
Cues 1Discriminated clients who were Fit-to-drive from Not fit-to-drive
2Discriminated clients who were Not fit-to-drive - require rehab, from the other three
outcomes
3Discriminated clients who were Fit-to-drive
or Not fit-to-drive from the middle two
outcomes of Fit to drive with conditions,
or Not fit-to-drive -require rehab.
Physical skills .48* .43 .07
Instructor interventions .44* -.38 -.42
Road law/road craft .39* -.03 -.03
Cognitive & perceptual skills .39* .37 -.09
Sensory functions .33* .07 .20
Driving experience .14* .05 .06
Medical prognosis .12 .48* -.27
Driving Need .07 -.37* .10
Driving History .04 -.19* .10
Vehicle handling skills .39 -.29 .53*
Age .07 .15 .41*
Driver behaviour .32 -.08 -.38*
Agreement (consensus) between fitness-to-drive recommendations was very high ICC= .97(95% CI .96-.98).
Results Do experienced OTDAs agree (level of
consensus) about the fitness-to-drive
recommendations made?
4. Produce a training package and develop an experimental website to test the effectiveness of the web-based decision aid;
5. Test the effectiveness of this aid on novice occupational therapists’ capacity to make fitness-to-drive decisions;
6. Host open access web-based decision aid designed to promote optimal occupational therapy assessment for use by the profession internationally.
Objectives
Randomised Control Design
Case scenario
decisions
Baseline
Case scenario
decisions
Post training
Training
(Intervention)
No training
(Control)
Eligibile for invitation (n=XX )
Consented to participate
(n=289)
Analysed at the baseline (n=70)
Analysed at immediate post-test (n=70)
Completed immediate post-training test
(n=70 )
Allocated to intervention (n=120)
completed the task (n=70)
received training intervention
(n=70)
Completed immediate post-training test
(n=96 )
Allocated to control (n=169)
completed the task (n=96)
received no training (n=96)
Analysed at the baseline (n=96)
Analysed at immediate post-test (n=96)
Allocation
Follow-Up
Randomized (n=289)
.
Results
Bland-Altman test was used to measure agreement between decisions made by a novice and experts’ consensus.
The equality graphs below demonstrate this trend
Control Group
Experimental Group
The effect size is calculated using the mean BA statistics from the post-training for control (.31, sd=.41) and for intervention (-.02, sd=.56); this demonstrated a moderate effect (d=.69, r=.32).
Results
Signal detection theory (SDT) was used to predict accuracy of decisions made by novices and to identify any change in decision strategy.
Although not all novices were able to align their recommendations to align with the expert consensus, their recommendation strategy changed and they adopted a lower risk strategy (ie recommended more clients to receive rehabilitation or not drive) which was more aligned with the experts (same as viewed in Equality Graphs above)
Web link: https://cisbic.bioinformatics.ic.ac.uk/fitness_to_drive_decision_aid/
(About to be launched!)
This tool is a training aid only. The developers are not liable for practice based decision making in the workplace.
Share expert capacity to optimize decision making
Can increase workforce capacity among novices in enhancing skilled fitness-to-drive recommendations
Can lead to a reduction in the number of unsafe drivers
Can lead to maintenance quality of life for those drivers who are fit-to-drive
Conclusion &
Contribution to practice
Unsworth, CU. , Harries, PA. and Davies, M. (2015) 'Using Social Judgment Theory method to examine how experienced occupational therapy driver assessors use information to make fitness-to-drive'. British Journal of Occupational Therapy, 2 pp. 71 - 72.
doi: 10.1177/0308022614562396
Publications
Professor Priscilla Harries
Professor Mary Hickson
Dr. Miranda Davies
Dr. Hulya Gokalp
Using fractional factorial
survey to develop an
effective training tool to
improve dietetic referral
prioritisations
Background
Study design
Expert policy capturing study →Training materials
Randomised Controlled Trial (RCT)
Final decision training aid
Content
Nutrition condition affects Effectiveness of medical treatment, quality and cost of care
Large numbers of referrals in adult acute care settings ; they need to be prioritised
Novice dietitians may struggle to prioritise referrals
An evidence based decision training tool can be developed and used to train the novices
Background
Case scenarios
Training Package
Policy Capturing
Study stages
RCTTraining aid
Website
i) What type of information is available to dietitians in dietetic referrals
ii) What kind of prioritization decisions are made?
iii) How do experienced dietitians weight the different information provided in dietetic referrals in order to prioritize them?
Phase I – Expert policy capturing
Questions to answer:
Case scenarios
Training Package
Policy Capturing
Cues / Levels
Cues / Cue Levels for dietetic referrals
Cue Cue Levels
Presenting complaint COPD*; Gastrointestinal C; Dementia; Pneumonia; UTI; Falls; Stroke; Dysphagia; Pressure ulcers
Nutrition status high-risk, at-risk or no-risk* of malnutrition
Reason for referral oral nutrition support*;specialist diet; enteral tube feeding; parenteral nutrition;dietary education
Previous food intake not eating; poor food intake; eating well*
Biochemistry picture Normal biochemistry; liver impairment; abnormal K+; re-feeding syndrome
Weight history Lost weight; gained weight; stableweight*
• In SPSS
• Unlikely cases were excluded• E.g. eating well, stable weight, BUT biochemistry picture indicating
refeeding syndrome
• 60 test cases
Fractional factorial design of test
cases
Sample case scenario
You have received a referral for a 65-year-old patient who may require dietetic assessment. The patient’s presenting complaint is Dysphagia; and they have screened as “High risk of malnutrition”. They have been referred for enteral tube feeding. The referrer reports that the patient is not eating and has stable weight. The biochemistry results
show normal biochemistry.
Non urgent – refer on to community dietetics
Non urgent – assess before discharge
Non urgent – assess within two working days
Urgent – assess on next working day
Urgent – assess today
When does this patient need to be seen for
assessment?:
Prioritisation options
Policy capturing study website
50 experienced dietitians to prioritise the 60 dietetic referral scenarios Inclusion criteria: min 6 months of experience as a dietitian
Multiple regression analysis was used to estimate weight of each cue level on the prioritisation decisions Dummy coding → Full regression model and reduced models
Analysis of experts’ decisions
• 50 dietitians : mean age 32 years, mean experience 7 years
• Agreement (consensus) between prioritisation decisions was very high: ICC= .98 (95% CI .97-.99).
Results
Results Full-regression results for influential cue levels
Cue Cue level B sr2
Presenting complaint Dysphagia .48 0.02
Nutrition status High risk .51 0.07
Reason for referral
Enteral tube feeding 1.01 0.28
Parenteral nutrition 1.18 0.29
Previous intake Not eating .39 0.04
Biochemistry picture
Re-feeding syndrome .74 0.13
Abnormal K+ .32 0.03
Weight history Lost W .23 0.02
model R2=0.957
Results
0
0.1
0.2
0.3
0.4
0.5
0.6
Reason Biochemistrypicture
Nutritionstatus
Presentingcomplaint
Previousintake
Weight history
Squ
ared
sem
i-p
arti
al c
orr
elat
ion
*
Cue
Results Importance of cue levels for the two most influential cues
Cue Cue Levels
Reason for referral 1. enteral tube feeding ≈ parenteral nutrition;2. specialist diet ≈ dietary education ≈ oral nutrition
support
Biochemistry picture 1. re-feeding syndrome2. liver impairment ≈ abnormal K+; 3. Normal biochemistry;
Phase 2
Training Package
RCT Training aid Website
• Development of study website
• Randomised Controlled Trial and analysis of data
• Development of final decision training website
Do the training materials improve capacity of novices
for making dietetic prioritisation decisions?
Randomised Control Design
Case scenario
decisions
Baseline40 scenarios
Case scenario
decisions
Post training20 scenarios
TrainingCase scenario
decisions
Case scenario
decisions
part
icip
ants
N=
197
115
82
intervention
control
82 82
69 69
Participants: pre-registration dietitians and recent graduates (1 year)
Results – Agreement between experts’ and novices’ decisions
ICC(2,1) between each novice’s decisions and experts’ consensus decisions
mean ICC(2,1) values for intervention and control at the two time points
Results
Pre-training
Post-training
Mean (pre –post)
Control 0.58 ↓ 0.53 -0.049
Intervention 0.59 ↑ 0.63 0.037
Cohen’s d (r) 0.54
Mean of ICC(2,1) values
• Significant main effect of group factor: F(1,149)=4.7, p<.01
• Significant interaction between time-point and group: F(1,149)=8.30, p<.01
• Simple effects: significant effect of group at post-training: F(1,149)=9.5,
p<0.01
ANOVA
Significant differences between the study groups at the post-training could be
attributed to the training intervention.
Decision training website
Dedicated website for training on dietetic referral prioritisation
Available since Jan 2016
TrainingPost
TrainingBaseline Feedback
https://cisbic.bioinformatics.ic.ac.uk/dietitian_training_aid/
• Evidence-based training materials are effective
• An open-access website for translation of the study into practice and for sharing best practice among novices
Conclusion
ulster.ac.uk
The Administration of an Online Factorial Survey Using Qualtrics
SoftwareHelena Mc Elhinney (email: [email protected])
Professor Marlene Sinclair
Professor Brian Taylor
SYNTHESIS OF THREE DATA
SOURCES TO DETERMINE
INDEPENDENT VARIABLES IN AN
ONLINE FACTORIAL SURVEY OF
IMPLANTABLE CARDIOVERTER
DEFIBRILLATOR DEACTIVATION
DECISIONSPresenter: Loreena Hill
Supervision team: Prof. B Taylor, Prof. S McIlfatrick & Prof. D Fitzsimons
Venue: Factorial Survey Methods in Health, Social and Ageing Research, London
15th June 2016
Outline
Clinical Context
Rationale for factorial survey design
Development of vignettes
Electronic dissemination of the survey
Results to date
Implications for clinical practice & methodology
Conclusions
Clinical Context
Implantable Cardioverter Defibrillator:
Implantations rates
Portrayed as ‘life-saving’ & ‘life-transforming’
Therapeutic role at end-of-life is contentious
Aging society: Multiple comorbidities such as renal failure or cancer
ICD prevents natural death
Decision-making: Patients’ choice limited due to lack of knowledge
Aim & Objectives
Aim:
To explore the perspectives of patients’, carers’ and professionals’ regarding ICD
deactivation and to examine the impact these have on clinical judgments about end-
of-life management.
Objectives:
1. To describe the understanding of heart failure patients with an ICD and carers
have regarding its deactivation at end-of-life.
2. To explore healthcare professionals’ practice and attitude towards ICD
deactivation.
3. To use this qualitative analysis to define the salient factors in ICD deactivation.
4. To evaluate the impact of these factors on professional judgement regarding
deactivation of an ICD at end-of-life.
Sequential Mixed Methods Design
Objective 3
9 Independent variables
Objective 1Systematic review of the literature (n=19)
Semi-structured interviews: patients (n=10) and carers (n=10)
Objective 2Retrospective case note review (n=44)
Focus Groups (n=9)
Objective 4
UK & Ireland factorial Survey
Rationale for Methodology
Ideal methodology
40+ years within research (Jasso & Rossi 1977, Ludwick et al. 2004, Taylor & Killick 2012)
Judgements are assumed to be “socially and individually structured”,
composed of guiding principles and values (Wallander 2009).
Sensitive research question
Factorial Study Design:
Novel & innovative to enhance recruitment
Manipulation of patient factors would be unethical & impossible within real-
life
Robust quantitative data
Complete anonymity ensuring authentic data collection
Electronic dissemination
Project Implicit:
US non-profit organisation, founded through by University of Washington,
Harvard & Virginia (1988)
Responsible for development of web-link and in real-time randomise IVs to
the stem vignette
Study instrument developed by the research team & uploaded onto
platform
200 vignettes were checked prior to dissemination
Online platform
Advantages:
Ease of access for busy healthcare professionals
Diverse sample in terms of geography disciplines
Minimal time to complete (7-10 mins), facilitated by “drop-down” menus
(Murray 2009)
Disadvantages:
Cost associated with using Project Implicit
Legal contract between Ulster University & Project Implicit
Vignette development
Independent variables (IV):
Seven (+/- two) within each vignette
Derived from practice knowledge, previous research & qualitative study (Ludwick 1999, Taylor 2006, Rattray et al. 2011)
Orthogonal (Wallander 2009) and clinically relevant
Each IV had between 2 - 5 levels, enhancing external validity.
(Excessive numbers can add unnecessary complexity and reduce statistical
power of analysis (Taylor & Zeller 2007))
Nine IVs in mathematical terms (5x3x3x3x4x3x3x3x3) equated to a potential
of 43,740 different vignettes= vignette universe
Vignette development (2)
Dependent variables (DV):
All participants answered 2 questions or DV on the clinical scenario
presented
Questions addressed both “likelihood” to make a decision and the
“confidence” in the decision just made
Corresponding responses were recorded on an ordered Likert scale with
scores ranging from zero to ten, anchored by descriptive terms (Lauder 2002)
Preliminary Analysis
Possible
variables
Levels Values Evidence from:
Literature Retrospective Case studies
case note review
Age 4 29-49
50-69
70-79
80-90
Null
Thylen, et al. 2013,
Fluur,, et al. 2013,
Groarke, et al. 2012,
Tajouri, et al. 2012,
Strachan, et al. 2011
Mean age 73
years
Focus Group 2 Dr:
“She is a bit
younger so it
would be difficult”
49 years is too
young
Co-Morbidities 4 Bowel cancer
Advanced renal disease
Dementia
Null
Stewart et al 2010,
Dodson et al 2013,
Buchalter et al 2014
Tajouri, et al 2014;
Kelley, et al 2009;
Marinski, et al 2010
12 patients had a
malignancy
Patient 4: Suffering
equated with
cancer and pain
with no mention of
HF symptoms
(NYHA IV)
Number of shocks 3 None
1-5
>10
Raphael et al. 2011,
Goldstein et al. 2008,
Stewart et al 2010
Ten patients had a
documented shock
and 4 multiple
shocks. Patients
who had never
experienced a
shock were more
likely (45% :15/33)
to undergo
deactivation.
Focus Group 5
Nurse: “ he thinks
he is getting
benefit from it and
he would like it left
on”. Carer 2:
linked shocks at
EOL to suffering.
Patient 3: “I would
hate to die
prematurely over
the sake of
Example of a vignette
You review a 59 year old female with moderate heart failure (NYHA III),
advanced renal failure. She has had 1 admission over the past year and
has experienced more than 1 shock. Medical records show no previous
discussion about deactivation with documented management plan to be
continue present treatment. The patient lives alone with no family or
friends.
1. What is the likelihood that you would discuss ICD deactivation with this
patient?
Not at all likely 0 1 2 3 4 5 6 7 8 9 10 very likely
2. How confident are you in the decision you have just made?
Not at all confident 0 1 2 3 4 5 6 7 8 9 10 very confident
Study instrument
Study instrument:
Short demographic questionnaire
One standard vignette (monitor the range of participants’ responses,
increasing internal & external validity; Lauder 2002)
Six vignettes with randomly assigned nine independent variables
Last case scenario (evaluate clinical validity of the study’s vignettes)
Sample of Questionnaire
Vignette & DV One within survey
Vignette & DV Two within survey
Analysis
Each vignette is a unit of analysis
Software: SPSS (version 21), significance was p=<0.05
Two levels of analysis:
Patient factors (vignette data)
Participant factors ( questionnaire data)
Multivariate regression analysis (Wallander 2009)
i.e. variance in the DV that can be explained by a IV
ANOVA with Least Significant Difference (LSD) posthoc
i.e. strength & direction of significance
Implications for methodology
To my knowledge 1st FS to synthesis 3 sources of evidence to develop
Independent variables
Systematic, rigorous and transparent process
Outcome: robust quantitative data on professional attitudes to
deactivation
International interest, with post-doc data to enable comparisons across
countries
A Tool That Produces Judgment Study Stimulus Case Sets
With Specified Cue-Cue Correlations.
Robert M. Hamm, PhD
Presentation at Factorial Survey Methods Conference, Brunel University, Uxbridge UK, June 15 2016
Department of Family and Preventive MedicineUniversity of Oklahoma Health Sciences Center
Overview
• Judgment studies
– We model judgments of sets of stimulus cases
• Stimulus sets
– Independent or correlated cues?
• Construction of stimulus sets with specified cue correlations
– Demonstration of spreadsheet
Judgment studies• Policy capturing
– Describe subject’s/judge’s use of cues– Possibly compare to “right way” to use cues
• Brunswik Lens Model – Describe cue use – Relate judge’s cue use to cue usefulness measured in
model of the ecology – The ecology model needs a criterion
• The “right answer” for each case
Stimulus sets
• Select appropriate cues– Cues relevant and important for task
– Vary over realistic ranges
– Number of levels for each cue
• Optional: definition of correct judgment– Statement of principles of good judgment
– Observation of true answer (“criterion”) • allows cue-criterion correlations (needed for lens model studies)
• Cue-cue correlations– Independent, orthogonal – factorial design?
– Or correlated?
Cue-Cue Correlations
• Efficiency
– Use of uncorrelated cues is more efficient
• “Design efficiency: (Glossary, Auspurg and Hinz, 2015)– Measure of the goodness of an experimental design.
– Good designs provide maximum information (minimum variance and covariance) for intended parameter estimates.”
• Easier to fit models– Factorial and “fractional factorial” ANOVA
• Smaller N of cases needed for a good fit
Cue-Cue Correlations
• Representative of the task ecology– Cue-cue correlations in stimulus set should be similar
to cue-cue correlations in the ecology.• Orthogonal design may produce unrealistic cases.
• “Vicarious functioning” – when cues are correlated, does not matter which one is attended.
• Judge may experience stimuli from orthogonal stimulus set as anomalous
– unusual cue combinations –
and adopt a different judgment strategy than usual.
• The descriptive analysis many not be able to reveal successful judgment strategies.
Analysis of Judgment of Stimulus Sets with Representative Cue-Cue Correlations
• Typically uses multiple linear regression.
– Could also use ANOVA
• But regression specifies relative weights
– Could also use multiple logistic regression (if judgment is dichotomous)
Example: Risk that ER patient will deteriorate
• Patient very ill, in emergency room– What is risk he or she will need resuscitation?
• Guidelines name the key predictors– Heart rate, respiration rate, temperature, level of
consciousness, systolic blood pressure– Could make up the cases, varying cue values
independently• Easy analysis
• Set of cases available (Subbe, 2001)– Could sample from them
• Representative of ecology
Example (continued)
• A blend between the “making up the cases with desirable cue correlations” and “sampling from real cases” was chosen:– For efficiency, 50% of the stimulus cases had a bad
outcome– Sample within outcome to get the cases
• What do you think of this strategy? How do you think it worked?– With respect to efficiency?– With respect to representativeness of cue
intercorrelations?
Cue Correlations in the EcologyData set – 555 patients, 57 “at risk”
• Cue correlations in data setCues Systolic
BP
Heart
rate
Resp.
rate
Temper
- ature
Conscious-
ness
Ecology
Criterion
Systolic BP ~
Heart rate .004 ~
Respiration
rate
.014 .284*** ~
Temperature -.047 .223*** .126** ~
Consciousness -.145** .069 .116** .088* ~
Ecology Criter. -.171*** .124** .179*** -.031 .161*** ~
Cue Correlations in Sample20 cases, 10 “at risk”
• Tolerance is 1 – R2, the complement of the proportion of variance explained in a multiple linear regression predicting the cue from all the other cues.
Cues Systolic
BP
Heart
rate
Resp.
rate
Temper
- ature
Conscious-
ness
Ecology
Criterion
Tolerance
Systolic BP ~ .072 .032 .558* -.616** -.147 .25
Heart rate .004 ~ .507* .493* .485* .455* .47
Respiration
rate
.014 .284*** ~ .353 .542* .594** .45
Temperature -.047 .223*** .126** ~ .007 .023 .43
Consciousness -.145** .069 .116** .088* ~ .523* .18
Ecology Criter. -.171*** .124** .179*** -.031 .161*** ~
Tolerance .98 .88 .91 .94 .96
Strategy did not reach either goal
• Failed to have efficiency
– very high cue intercorrelation
– Some subjects’ dichotomous judgments could not be modeled with logistic regression
• Failed to have representativeness
– More “at risk” cases than would typically be seen
Why were the cue-cue correlations so high?
• When we adjusted the prevalence of the criterion
– from 10% to 50%
it increased the cue-criterion correlations.
• When cues are correlated with a common variable (the criterion), we can expect the cues to be correlated with each other.
Sample or Construct the Stimulus Set?
• Barriers to using stimulus sets with correlated cues– Sampling: You need data on cues in the ecology
• Who has such data?
• If sample from data set, check the sample’s correlations.
• Look at the cue–cue correlations in the sample (and the cue-criterion correlations, if it is to be a lens model study),
• Make sure they are close to the correlations in the whole data set; if not, sample again.
Barriers to constructing a stimulus set with correlated cues
• Construction
– Who knows what the cue-cue correlations should be, without data?
• Ask experts? Refer to publications?
– Who knows how to construct a set of cases with the specified correlations?
• I developed an Excel spreadsheet to do it.
Spreadsheet strategy
• User specifies stimulus set features
– Including cue-criterion correlations
• Excel projects cue-cue correlations
– User can specify adjustments to cue-cue correlations
• Excel constructs stimulus set, random generation
– Computes cue-criterion and cue-cue correlations
– Compares with expected and projected correlations
• User inspects and uses it if “close enough”
Specify stimulus set
– N cases
• # with each level of criterion
– # cues
• # levels per cue
– Cue-criterion correlations
– Cue-cue correlations
Basics for any stimulus set
• Features
• Correlations with criterion
– “Cue 1” is the criterion.
Cue-cue correlations
• One factor
• Several factors
• Also correlated factors
Spreadsheet projects cue-cue correlations
• If two cues are each correlated with the criterion, they’ll be correlated with each other
r(cue 2, cue 3)=r(cue 2,crit)*r(cue 3,crit)
• E.g., if only specify cue-criterion correlations
Spreadsheet also projects cue-cue correlations
• E.g., if specify some cue-cue correlations
Spreadsheet constructs cases• For cue-criterion correlations only
• Each cue value a mix of the criterion value and random.
• Continuous values (scale -0.5 to +0.5) chopped into ordered levels
Spreadsheet computes correlations• Correlation among continuous cues
• Correlations among ordered category cues
• Comparison with specified and projected correlations
Is this sample good enough?
• A random sample’s cue-cue correlations do not necessarily resemble the intended correlations– Among all the possible samples, one could even
find one where all cue-cue correlations are 0. • Extremely rare.
– (but this is what factorial surveys often use)
– So you have to inspect. • Is this close enough to expected/desire cue-cue
correlations?
Extensions, use, availability
• Extension: – Class–conditional dependence. In principle and
probably in practice, the intercorrelations among “cases” may differ from the cue-cue correlations among “non-cases”.
– The worksheet has a sheet that accommodates this.
• Use:– One study has used this, so far. Data not in.
• Availability: – CueCorrels.xlsx– [email protected]
Opportunities
• Opens up possibility of producing stimulus sets with chosen cue-cue correlations
– (approximately)
• Allows researchers to address considerations of representative design of stimulus sets
– (also called “ecological validity” of the stimuli)