1

Development of Valid and Reliable Case Studies for

Teaching, Diagnostic Reasoning, and Other

Purposes

Margaret Lunney, RN, PhDProfessor

College of Staten Island, The City University of New York

2

What Are Case Studies?Simulations of patients’ stories:

Written, Video-taped, Computer-based

Designed for specific purpose(s)

Variety of designs:Length

Content

Style: Linear, branching

3

When Are Case Studies Used?

Practicee.g., orientation of new nurses, discussion of complex cases

Educatione.g., teaching aids, testing

Researche.g., measurement of accuracy, evaluation of knowledge

4

Advantages: Simulations of Patients’ Stories

Standardized

Represent important, usual, familiar, & challenging situations

Restricts the complexity of practice

User “gets involved”

Reasonable cost

5

Why Is It Difficult to Develop Good Case Studies?

Case studies are tools

Quality= Validity & Reliability

Development takes time, energy and commitment

6

Case Studies as Tools Goal

Strengthen the link between knowledge and application

HowOperationalize complex abstract concepts and “real” patients’ stories, e.g.,

Hope

Caregiver Stress

Ineffective Self Health Management

7

Case Studies as Tools to Measure Nursing Concepts

Nursing Concepts nursing diagnoses

Capture key elements of a real patient situation

Use principles of measurement

Instrumentation-Method to measure concepts

8

Case Studies as Tools (cont.)

Measurement: DefinitionSee Waltz, Strickland & Lenz, 2005

Importance of conceptual frameworksChallenges of measurement

9

Case Studies as Tools (cont.)

The dilemma & challenge: Patients’ stories are complexOverlapping variables Biases in interpretationUse of heuristicsMany types of thinking

GoalReduce ambiguous, abstract ideas to concrete behavioral indicators

10

Measurement Frameworks

Norm-referencedEvaluate performance relative to the performance of others

Criterion-referencedEvaluate performance based on

pre-determined

standards

11

Case Studies as Criterion-Referenced Tools: Reliability

Definition:Consistency of scoring

Range of variability is reduced, use nonparametric procedures

Test-Retest

Parallel forms

Interrater & intrarater agreement

12

Case studies as Criterion-Referenced Tools: Validity

Definition:Measures what was intended, systematic error is reduced

Types:Content validity

Criterion-related validity

Construct validity

13

Case Studies as Criterion-Referenced Tools: Tasks

Precisely specify target behaviors Identify standards for target

behavior Discriminate who did and did not

acquire the target behavior Compare subjects’ performance to the standards

14

Case Studies as Tools (cont.)

Types of written simulationsLinear

BranchingFree branching

Modified free branching

Forced branching

15

Linear Technique

Subjects follow same sequenceOne or more sectionsIf more than one section, students receive specific instructions, and

Sections are appropriate for all subjectsOne section does not influence other sectionsEach section samples inquiries or actions

16

Guidelines for Case Study Development: Linear Format1. Identify the overall purposes:

What is the general topic?What kind of problem solving will be represented?How important are the responses, e.g., learning process, grade, research data?

17

Guidelines (cont.)

2. Specify objectives, e.g., toMeasure accuracy of diagnosis

Illustrate relation of cues to inferences

Facilitate planning

Orient new nurses

Teach sequential decision making

18

Guidelines (cont.)

3. Decide the complexity: Length

Number of diagnoses, interventions,

and/or outcomes

Amounts of high, moderate & low

relevance data

Types of data

19

Guidelines (cont.)

4. Obtain literature sources & nurse experts to support validity:

Select authoritative sources: actual cases, literature sources Decide importance of each sourceIdentify expert judges, e.g., may ask colleagues Use well-known experts for research tools

20

Guidelines (cont.)

5. Formulate case studies, directions, & scoring manual

Directions must be explicit,

comprehensive, and clearly written

Scoring manual provides scores for possible answers

21

A) Create Blueprint

Choose a general problem areaWhat should this exercise teach or test?

What kinds of problems need to be solved?

What kinds of knowledge needs to be used?

Define the objectives to be sampled

Select concrete problem situation(s)

Outline and diagram the exercise

22

B) Prepare Specifications

Method of administration

Proportion of content

for each objective

Style of writing case(s)

Restrictions, e.g. time

Describe scoring procedures

23

C) Construct Case Studies

Develop pool of

data/items to

match objectives

Review to determine content

validity and appropriateness

Edit, delete, change as indicated

Assemble the case(s)

24

D) Set Standards or Cut Scores

What types of answers

are “correct”

What answers/scores are

acceptable or unacceptable

Classification of various

answers, # of points toward a

grade

25

Guidelines (cont.)

6. Obtain content validityratings from experts:

Select content validity method Item-objective congruenceInterrater agreementAverage congruency percentage

Send with explicit instructions Be prepared to make changes

26

Guidelines (cont.)

7. Evaluate with a pilot test:Consider demographics, e.g., Experience Knowledge

Evaluate directions, restrictions, scoring methods, interrater reliability

27

Conclusion

Case studies are fun to use

The effort is “worth it”

New case study feature in IJNTC, send submissions to Margaret Lunney

Suggested website:http://ublib.buffalo.edu/libraries/projects/cases/case.htm