Chapter 5

Instrument Selection, Administration, Scoring, and Communicating Results

Selection of an Assessment Instrument

Identify possible alternatives Educational Testing Services may be

useful Mental Measurement Yearbook (MMY)

reviews many tests and is available at Morris or reviews can be ordered on-line for a fee.

Client’s lives can be adversely affected by selecting and using a faulty instrument

Evaluating an Instrument Test Purpose Instrument

Development Appropriate Norm

Group or Criterion Reliability

Validity Bias Interpretation and

Scoring User Qualifications

Test purpose

Does instrument meet counseling needs?

Manual may state purpose Purpose may not relate to name Purpose may not be met because of

construction problems

Instrument development

Construction of items crucial Look for detailed item analysis data Face validity

Selection of norm group

Is norming group appropriate for client? Age Gender Ethnicity Socioeconomic representation Educational level Geographic location

Selection of criterion

How did developers determine criterion? Standards from professional

organizations Reviews by leading researchers Studies of curriculum

Manual must supply enough data to be convincing

Reliability

No clear guidelines for what constitutes “good” reliability

Type of instrument affects how reliability should be viewed

Can be estimated in different ways and manuals will often provide range of coefficients calculated using different methods

Validity

Look for preponderence of evidence Manual must provide enough

information to judge GO TO THE LITERATURE!

Bias

Test bias in context of multicultural issues Differential item functioning (item

difficulty differs for ethnic groups) Reliability coefficients may differ

ethnically Construct validity may be compromised

as the construct may be culturally based

Interpretation and scoring materials

Manual must provide adequate description of what scores mean

Hand vs. machine scoring

User qualifications Level A: manual is sufficient (Holland’s Self-

Directed Search) Level B: master’s degree or equivalent

training (Myers-Briggs, Strong Interest Inventory)

Level C: doctoral level or equivalent training (Wechslers, Binet, MMPI-2)

Code of Fair Testing Practices in Education ACA Code of Ethics, Section E

Practical issues

Cost Time

Evaluating instruments

Form in text, page 95 Instrument evaluation paper

Administering Assessment Instruments

Pre-read administration materials Follow standardized procedures

Administrator Effects

Expectancy (pygmalion effect) Some data support minor effects

Examiner/Examinee Relationship

Some studies support impact of relationship for kids and even more so for lower SES kids

Standardized test manuals usually address relationship boundaries

Examiner race

Research equivocal

Scoring

Hand scoring Clients can score some assessments

(SDS) but may be clinical reason not to Some tests require multiple templates

Computer scoring and interpretation Few to no errors Counselors ethically bound to know

integrity of service and steps taken to develop scoring and interpretation programs

Performance and Authentic Assessment Scoring

Associated with achievement testing (rather than grammar test, write business letter)

Goal is to see if knowledge can be applied

Big issue is objectivity of scoring

Performance and Authentic Assessment Scoring Guidelines

Assessment has specific focus

Scoring plan is based on observable qualities

Scoring is designed to reflect the intended target

Setting for assessment is appropriate

Checklist or rating scales are used

Scoring procedures have been field tested before they are used

Communicating Results

Must know manual information Optimize power of test (It’s useful in

these ways for these reasons) rather than allowing client to maximize (test speaks truth)

Use effective counseling skills Develop multiple methods of

explaining in “plain” English

Communicating Results (cont’d)

Use descriptive terms rather than numerical scores and tie them to the reason for the assessment

Put results in context of other client information

Involve client in interpretation and ask for feedback often

Communicating Results (cont’d)

Limitations are discussed in non-technical terms

Encourage client to ask questions Summarize results to iterate and

stress important points

Communicating Results to Parents

Be empathic Monitor parents’ reactions so child

does not internalize as blame Provide therapeutic environment