Nghiên cứu KH-C5

7/27/2019 Nghin cu KH-C5

1/29

CHAPTER 5

MEASUREMENT, RELIABILITY ANDVALIDITY

Nguyen Tien Dzung

Hanoi University of Science and Technology

Email: [email protected]

Website: http://dungnt.tk


2/29

CHAPTER OVERVIEW

1. The Measurement Process2. Levels of Measurement

3. Reliability and Validity: Why They Are Very, Very Important

4. A Conceptual Definition of Reliability

5. Validity6. The Relationship Between Reliability and Validity

7. A Closing (Very Important) Thought

2012 NTD 2Research Methods - Chapter 5


3/29

1. The Measurement Process

Two definitions Stevensassignment of numerals to objects or events

according to rules.

the assignment of values to outcomes.



4/29

2. Levels of Measurement

Variables are measured at one of these four levels

Qualities of one level are characteristic of the next level up

The more precise (higher) the level of measurement, the more accurateis the measurement process

Level ofMeasurement For example Quality of Level

Ratio A is 5 10 and B is 5 5 Absolute zero

Interval A is 5 taller than B An inch is an inch is an inch

Ordinal A is taller than B Greater than

Nominal A is tall and B is short Different from



5/29

Nominal Scale

Qualities Example What YouCan Say

What You Cant

Say

Assignment oflabels

Gender(male orfemale)

Preference(like or dislike)

Voting record(for oragainst)

Eachobservationbelongs in its

own category

An observationrepresents moreor less than

anotherobservation



6/29

Ordinal Scale


What You Cant

Say

Assignment ofvalues alongsome underlying

dimension

Rank in collegeOrder of finishing arace

Oneobservation isranked above

or belowanother.

The amount thatone variable ismore or less than

another



7/29

Interval Scale


What You Cant

Say

Equal distancesbetween points

Number of wordsspelled correctlyIntelligence test

scoresTemperature

One scorediffers fromanother on

somemeasure thathas equallyappearingintervals

The amount ofdifference is anexact

representation ofdifferences onthe variablebeing studied



8/29

Ratio Scale

Qualities Example What You CanSay

What You Cant

Say

Meaningful andnon-arbitraryzero

AgeWeightTime

One value istwice as muchas another orno quantity ofthat variablecan exist

Not much!



9/29

Continuous vs. Discrete Variables

Continuous Variables: can assume any value along some underlying continuum.

Examples: length, time, revenues, income

Discrete Variables:

Categorical variables

Those with values that can be placed into certain

categories that have definite boundaries. Examples: gender, customer satisfaction (scale from 1 to

5)

2012 NTD Research Methods - Chapter 5 9


10/29

What Is All The Fuss?

Measurement should be as precise as possible

In behavioral and social science, most variables are

probably measured at the nominal or ordinal level.



11/29

3. Reliability and Validity

Importance of reliability and validity The methods and tools we use to measure the interested

object influence on the research accuracy.

Reliability: ability to produce the same results for

different measurements.Validity: ability to measure what we want to measure.



12/29

Reliability

ObservedScore

TrueScore

Error


TraitError

MethodError

Error


13/29

A Conceptual Definition Of Reliability

Method error is due to characteristics of the test or testing situation

Trait error is due to individual characteristics

Reliability of the observed score becomes higher if error is reduced!!

Method ErrorObserved Score = True Score + Error Score

Trait Error



14/29

INCREASING RELIABILITY

Decreasing Error

Increase sample size

Eliminate unclear questions

Standardize testing conditions

Use both easy and difficult questions

Minimize the effects of external events

Standardize instructions

Maintain consistent scoring procedures



15/29

How Reliability Is Measured

Reliability is measured using a Correlation coefficient

r test1test2

Reliability coefficients

Indicate how scores on one test change relative to scoreson a second test

Can range from -1.0 to +1.0 (perfect reliability)



16/29

Types of Reliability

Type of

Reliability

What It Is How You Do It What the Reliability

CoefficientLooks Like

Test-Retest A measure of stability Administer the same test/measureat two different times to the samegroup of participants

rtest1test1

Parallel Forms A measure ofequivalence

Administer two different forms ofthe same test to the same groupof participants

rform1form2

Inter-Rater A measure ofagreement

Have two raters rate behaviorsand then determine the amountof agreement between them

Percentage of agreements

InternalConsistency

A measure of howconsistently eachitem measures thesame underlyingconstruct

Correlate performance on eachitem with overall performanceacross participants

Cronbachs alpha



17/29

Validity

A valid test does what it was designed to doA valid test measures what it was designed to

measure



18/29

A Conceptual Definition of Validity

Validity refers to the tests results, not to the test itself

Validity ranges from low to high, it is not either/or

Validity must be interpreted within the testing context

E.g.: Calculation: 2 + 7 = ?

Which ability does the task measure?

Addition abilityor Multiplication ability or Mathematical ability?



19/29

Validity

Validity tool measures what-it-should Example:

Question: 2 + 3 = 5

I check 10 times with ten addition calculations, you answer 100%correctly your answer is very reliable.

I use the test to measure your intelligence. Is the test valid?

NO. BECAUSE THIS TEST IS VALID ONLY FOR MEASURINGADDITION ABILITY, NOT MORE THAN THAT; INTELLIGENCE IS

VERY COMPLICATED ABILITY.



20/29

Example of Validity and Reliability

A wife wants to measure the loyalty of her husband She uses a criterion: never come late

He actually never comes home late.

Is the test valid?



21/29

Types of Validity

Type ofValidity

What Is It? How Do You Establish It?

Content A measure of how well theitems represent the entireuniverse of items

Ask an expert if the items assesswhat you want them to

Criterion

Concurrent A measure of how well atest estimates a criterion

Select a criterion and correlatescores on the test with scores onthe criterion in the present

Predictive A measure of how well atest predicts a criterion

Select a criterion and correlatescores on the test with scores on

the criterion in the futureConstruct A measure of how well a

test assesses someunderlying construct

Assess the underlying construct onwhich the test is based andcorrelate these scores with thetest scores



22/29

Content Validity

A measure of how well the items represent theentire universe of items

Ask experts to check whether the items assesswhat you want them to access

E.g.:

You want to measure education quality

You must ask experts. They show you some relevant

criteria to measure (teachers, libraries, facilities,curricula, research activities and applications, reputation)



23/29

Exercise for homework

Project 1:You want to measure: personal intelligence

How to define the concept? (What are the criteria ofintelligence?)

Project 2:

You want to measure: personal happiness

How to measure?



24/29

Criterion Validity

Concerned with how well a test either estimates performance (concurrent validity) or

how well a test predicts future performance (predictive vadility)

Example:

A research to measure how GPA (Grade Point in Average) of

graduate students relate to their research ability

The research ability is measured by the number of articles publishedon journals

If the correlation between GPA and the number of published articlesis high, it means GPA is a good predictor of the number of publishedarticles (high predictive validity)

Does a high number of published articles mean a good researchability? is the question about concurrent validity of the research



25/29

How To Establish Construct Validity of

A New Test

Correlate new test with an established test

Show that people with and without certain traitsscore differently

Determine whether tasks required on test areconsistent with theory guiding test development



26/29

Multitrait-multimethod Matrix

2 research projects about children: Project 1: To measure impulsivity (Trait 1)

Project 2: To measure activity level or movement (Trait 2)

To test a construct validity, use multitrait-

multimethod matrix Project 1:

Method 1: paper-and-pencil instrument on childs wrists

Method 2: activity meter (e.g.: observation)

Project 2: Method 1: paper-and-pencil instrument on childs wrists Method 2: activity meter (e.g.: observation)



27/29

MULTITRAIT-MULITMETHOD MATRIX

Convergent validitydifferent methods yield similar results

Discriminant validitydifferent methods yield different results

Method 1

Paper andPencil

Method 2

Activity LevelMonitor

Method 1

Paper andPencil

Method 2


Trait 1

Method 1

Paper and Pencil Moderate Low

Impulsivity Method 2Activity Level

Monitor

Moderate

Trait 2

Method 1

Paper and Pencil

Activity

Level

Method 2


Low

Trait 1

Impulsivity

Trait 2

Activity Level



28/29

The Relationship Between Reliability and Validity

(The Books Point Of View)

A valid test must be reliableBut

A reliable test may not be valid

A high reliability is a necessity to have a high validity, but notensure


VALIDITY AND RELIABILITY (Th t h i t f


29/29

VALIDITY AND RELIABILITY (The teachers point of

view)

a) High validity and high reliability b) High validity but low reliability

c) Low validity but high reliability d) Low validity and low reliability 2012 NTD 29Research Methods Chapter 5

Documents

Nghiên cứu KH-C5