Upload
sppippukm
View
147
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Designing, Validating & Pre-Testing A Questionnaire
Citation preview
Designing, Validating & Pre-Testing A Questionnaire
Dr Azmawati Mohammed Nawi
Dept. Of Community Health
UKMMC
What is validity?What is validity?
• The ability of a scale to measure what it is intended to measure
• The extent to which a measure reflects the real meaning of the concept under consideration
• The extent to which a measure reflects the opinions and behaviors of the population under investigation
• Can not be valid unless also reliable
What is validity? (cont…)
• Refers to the degree to which a study accurately reflects or assesses the specific concept that the researcher is attempting to measure.
• While reliability is concerned with the accuracy of the actual measuring instrument or procedure, validity is concerned with the study's success at measuring what
What is validity? (cont…)
Depends on the Purpose of the measure
• E.g. a ruler may be a valid measuring device for length, but isn’t very valid for measuring volume
Measuring what ‘it’ is supposed to be
Types of validity
Types of validity
• Face validity
– professional agreement that variables cover range of meanings included within the concept
Items should be evaluated for their presumed relevance
Items should cover a range of ideas rather than a single topic area
Items should be evaluated in terms of the abilities of the individuals under investigation
Content ValidityContent Validity
• How well elements of the test relate to the content domain?
• How closely content of questions in the test relates to content of the curriculum?
• Directly relates to instructional objectives and the fulfillment of the same!
• Major concern for achievement tests (where content is emphasized)
• Can you test students on things they have not been taught?
Criterion Validity
Criterion Validity
• The degree to which content on a test (predictor) correlates with performance on relevant criterion measures (concrete criterion in the "real" world?)
• If they do correlate highly, it means that the test (predictor) is a valid one!
• E.g. if you taught skills relating to ‘public speaking’ and had students do a test on it, the test can be validated by looking at how it relates to actual performance (public speaking) of students inside or outside of the classroom
Two Types of Criterion Validity
Two Types of Criterion Validity
• Concurrent Criterion Validity = how well performance on a test estimates current
performance on some valued measure (criterion)? (e.g. test of dictionary skills can estimate students’ current
skills in the actual use of dictionary – observation)
• Predictive Criterion Validity = how well performance on a test predicts future performance on some valued measure (criterion)?
(e.g. reading readiness test might be used to predict students’ achievement in reading)
• Both are only possible IF the predictors are VALID
Construct validityConstruct validity
• The degree to which a measure relates to other variables, as expected, within a given system of theoretical relationships `
Construct Validity (cont…)Construct Validity (cont…)
• Issue is: do you have a good operationalization of your construct (the thing you are trying to measure)?
• First it is necessary to define the construct: what is it you want to measure or operationalize? (I want to know how in love people are.)
• Two general approaches to assessing construct validity:– Examine the content of the measure:
does it make sense? – Study its relation to other variables
Construct Validity Types
Construct Validity Types
Construct ValidityTranslation
ValidityCriterion-
related Validity
Face ValidityContent Validity
Predictive Validity
Concurrent Validity
Convergent Validity
Discriminant Validity
Construct Validity TypesConstruct Validity Types
Construct ValidityTranslation
ValidityCriterion-
related Validity
Face ValidityFace
ValidityContent Validity
Predictive Validity
Concurrent Validity
Convergent Validity
Discriminant Validity
You look at the operationalization and see whether on its face it seem like a good translation of the construct.
Construct Validity Types
Construct Validity Types
Construct ValidityTranslation
ValidityCriterion-
related Validity
Face ValidityFace
ValidityContent ValidityContent Validity
Predictive Validity
Concurrent Validity
Convergent Validity
Discriminant Validity
You check the operationalization against the relevant content domain for the construct.
Construct Validity Types
Construct Validity Types
Construct ValidityTranslation
ValidityCriterion-
related Validity
Face ValidityFace
ValidityContent ValidityContent Validity
Predictive Validity
Predictive Validity
Concurrent Validity
Convergent Validity
Discriminant Validity
You check the operationalization’s ability to predict something it should theoretically be able to predict
Construct Validity Types
Construct Validity Types
Construct ValidityTranslation
ValidityCriterion-
related Validity
Face ValidityFace
ValidityContent ValidityContent Validity
Predictive Validity
Predictive Validity
Concurrent Validity
Concurrent Validity
Convergent Validity
Discriminant Validity
You check the operationalization’s ability to distinguish between groups that it should theoretically be able to distinguish between.
Construct Validity TypesConstruct Validity Types
Construct ValidityTranslation
ValidityCriterion-
related Validity
Face ValidityFace
ValidityContent ValidityContent Validity
Predictive Validity
Predictive Validity
Concurrent Validity
Convergent Validity
Convergent Validity
Discriminant Validity
You examine the degree to which the operationalization is similar to other operationalizations to which it theoretically should be similar
Construct Validity Types
Construct Validity Types
Construct ValidityTranslation
ValidityCriterion-
related Validity
Face ValidityFace
ValidityContent ValidityContent Validity
Predictive Validity
Predictive Validity
Concurrent Validity
Convergent Validity
Discriminant Validity
Discriminant Validity
You examine the degree to which the operationalization is notnot similar to other operationalizations to which it theoretically should not be similar to.
CONSTRUCT VALIDITY
• Factor analysis (FA)
(continuous outcome)– Biological
measurement – questionnaire
• Sensitivity and specificity
(categorical outcome)
(screening and diagnostic test evaluation)
Validity analysisValidity analysis
Factors analysis
Sensitivity & specificity
FACTOR ANALYSIS
FA is a method of data reduction
– Take many variables and explain them with a few factors
– Correlated variables are grouped together and separated from other variables with low or no correlation
– Patterns of correlations are identified as indicative of underlying theory (FA)
• GOALS• Data reduction
• Describe relationships
• Test theories about relationships
• ADVANTAGE– Makes sense
– will be easy to interpret
– simple structure
• ASSUMPTIONS• 1. Independence
– Each respondent: participate only once
• 2. Sample Size, Factor analysis– ˜ 5 participants / variable
• 3. Normality– FA is fairly robust against
violation of this assumption• 4. Linearity
– Roughly linear relationship btwn variables
• 5. Assumes reliable correlations
• STEPS– 1 and 2: methodology
– 3 – 5 : during analysis
QUESTIONNAIRE:5 ITEMS“ATTITUDE TOWARDS SMOKING”
PART 1: NORMALITY
PART 1: NORMALITY OUTPUT
PART 2: FA - PROCEDURE
PART 2: FA OUTPUT
-provide factorability of the data
KMO: reports the amount variance in the data that can be explained by the factors
higher value is better (0. 6 and above are acceptable)
Barlett’s test of sphericity: how suitable the data are for FAsignificant is acceptable
Further examine the suitability of data for FAIf less than 0.5: does not have strong relationship with other variables in the matrixShould consider dropping from the analysis
CORRELATION MATRIX
SCREE PLOT
SMOKING BY OTHERS
SMOKING NEAR FOOD
Reliability
Issues in assessing clinical tests
• Validity of the tests – how good is the test to identify the sick and the healthy individuals
• Reliability – how stable is the results of the test .
• Efficiency and cost-effectiveness
How to measure?• Validity/Accuracy – Sensitivity & Specificity• Reliability
– Qualitative • Kappa Analysis• AC-1 Statistic
– Quantitative• Bland-Altman Plot • Cronbach α Coefficient (internal consistency)• Test retest• ICC (intraclass correlation)
Reliability – Quantitative Data
Cronbach α Coefficient
Cronbach α Coefficient
• Estimation based on the correlation among the variables comprising the set
• Used to assess the consistency of results across items within a test.
• Therefore: used to measure internal consistency.
•In scales, the value should be above 0.7.
•In SPSS, select Analyze, Scale, Reliability Analysis.
•Select the two variables that is being compared and move them into the box marked Items.
•In the Model section, make sure Alpha is selected.
•Click on the Statistics button. In the Descriptives section, click on Item, Scale, Scale if item deleted. In the inter-item section, click on correlation
•Click on Continue and then OK.
SPSS COMMAND
USE SAME DATA SET FOR FACTOR ANALYSIS
Pearson’s correlation between each item and the sum of the remaining 2 items
RESULTS VALIDITY AND RELIABILITY TESTING
Loadings
Item mean(sd)Factor
1a
Factor 2a CITC Alpha
1. I think smoking is acceptable 60.35(16.72) 0.983 0.952
0.962. I don't care if people smoke around me 55.18(21.58) 0.939 0.8863.I think people should have the right to smoke 59.21(16.95) 0.985 0.9564. I don't think people should smoke in restaurant 62.93(17.40) 0.933 0.762
0.8635. I don't think people should smoke around food 61.21(15.83) 0.939 0.762aData extraction using Principal Component Analysis and Varimax rotation. The factor loading <.40 is suppressed for presentation. CITC: Corrected Item Total Correlation Aplha: Cronbach's alpha
Conclusion
• To get a good Questionnaire set...it’s not easy
THANK YOU