NEUROPSYCHOLOGICAL EVALUATION OF SPORTS CONCUSSION/TBI:

HISTORICAL AND PSYCHOMETRIC CONSIDERATIONS

Ruben J. Echemendia

FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive

Function, and TBI/Sports Concussion Diagnostic Devices  

Washington, DC: June, 2011

Disclosure

Dr. Echemendia is a paid consultant to: The National Hockey League Major League Soccer US Soccer Federation

Historical Background

Neuropsychological assessment has been a key component in evaluating/managing TBI for decades.

Neuropsychological evaluations have become a key tool in the evaluation of MTBI since mid 70’s.

Focus on functional vs. structural assessment.

NP assessment of sports concussions began in mid 80’s.

Background

Sports Laboratory Assessment Model Pre-post injury evaluations Paper & Pencil Assessment Batteries Computerized Batteries

Psychometric Considerations Validity vs. Reliability Validity

Does the test measure what it is supposed to measure? Construct Validity –Operationalization of the construct,

measurement of the construct Cognitive tests DO NOT measure concussion! Assess cognitive domains that are believed to be affected by

concussion Convergent/Divergent Validity

Does the test correlate with other known measures of the construct?

Does the test not correlate with domains that it is not supposed to measure Face Validity

Face Validity Does the test appear to be measuring what it is supposed to

measure

Psychometric Considerations

Sensitivity vs. SpecificitySensitivity: Is the measure “sensitive” in

detecting the construct being measured?○ Do concussed individuals score lower than

non-concussed individuals?○ What proportion of concussed individuals are

identified by the test? Positive Predictive Power

○ What proportion of Non-concussed individuals are identified by the test? Negative Predictive Power

HYPOTHETICAL 2X2 VALIDITY MATRIX

Abnormal NP Scores Normal NP Scores

0

0 150150

300

Concussed

Normal

150150

HYPOTHETICAL 2X2 VALIDITY MATRIX

Abnormal NP Scores Normal NP Scores

30/ 20%

150

300

Concussed

Normal

150

120/ 80%

90/ 60%60/ 40%

True Positive False Negative

False Positive True Negative

Psychometric Considerations

Sensitivity vs. SpecificitySpecificity: Is the measure “specifically”

measuring the identified construct and not others?

Concussion vs ADHD, Depression, Anxiety, Fatigue, etc.

Ideal: 100% Sensitivity – 100% SpecificityReal: NOT!Sensitivity vs Specificity trade-offTypically as sensitivity increases, specificity

decreasesThe user must decide on what side to err.

ROC Curve

Psychometric Considerations

Reliability – Estimating ErrorCan we expect to get the same results when

the test is given….○ To the same person an infinite number of

times? (Confidence Intervals)○ To the same person under different

circumstances? (Sideline vs. Office)○ To the same person at different points in time?○ To the same person by different test

administrators?○ In different/alternate forms?

Tests Used in Sports TBI….

Paper & Pencil Tests Selected from tests already in use and validated

for assessing MTBI Test Requirements:

ShortEasily administered by non-NPsEasily scoredSensitive to changes in specified domainAcceptable psychometricsAlternate forms.

Assess several cognitive domains. Typical Tests: HVLT. BVMT. Trails, Digit Span,

SDMT, Verbal Fluency, LNS, Cancellation tests.

Paper & Pencil Tests Cont. Advantages:

Direct observation of player behaviorDirect monitoring of effort/motivation & task

performanceSuperior assessment of memoryBroad normative db with multiple samples.

Disadvantages:Labor intensive (Expensive)Variability in standardized adminLess reliable assessment of info proc speed.Tests must be hand scored

Computerized Batteries Two approaches:

Adapt existing P&P measures into computer platform

Measure multiple cognitive functions (memory, proc speed, reaction time, etc.)

Develop tests de novo for computers Limited assessment of domains (largely

RT, Info Proc Speed) New Instruments are being developed rapidly Existing Instruments: ImPACT, ANAM Sports,

HeadMinders CRI, CNS Vital Signs, CogSport.

Computerized Tests Cont. Advantages:

Can be used in groups (Less costly)Standardized admininistrationImmediate scoring/data captureFiner assessment of RT/Proc Speed

(altho other problems)Multiple languagesTheoretically infinite alternate

forms 

Computerized Tests Cont. Disadvantages:

Less complete assessment of memoryLoss of observational dataLess control of effort/motivationDiminished ability to assess

understanding/track task performanceComputer-specific measurement issuesData loss

Hybrid Approaches

Combine strengths of both P&P and Computerized tests

Once adequate normative data have been established….

Computer for baseline testingComputer plus P&P post-injury

Sports MTBI specific issues

Phase of Evaluation

Baseline

Acute – On Field Management

Sub acute – Office/Training Room

Return to Play/Work/School

Long Term Follow-up

Sports MTBI specific issues Baseline Testing – Is it necessary/helpful? Theoretically designed to minimize error due

to intra-individual comparison Does it do this? Does it increase sensitivity/specificity? Does it introduce additional sources of error? What about multiple post-injury tests? How often should it be repeated? Age dependent?

Sports MTBI specific issues Test – Retest Reliability Significant variability across studies What test-retest intervals should be

used? Reliability estimates for multiple tests

Sports MTBI specific issues Practice Effects Content Practice vs. Procedural Use of Alternate Forms Reliability/Validity of Alternate Forms Are practice effects bad? How to control for / appropriately use

practice effects?RCI, regression based estimates

Sports MTBI specific issues Motivation/Effort Unique aspect of sports was the

assumption that players are motivated to perform well.

Different motivation as different phases of assessment

Baseline vs. Post-injury Individual vs. Group Identification of poor effort

Test specific indicesOutlier identification

User Qualifications

Although not the responsibility of the FDA, clear delineation of user qualifications for administering and interpreting tests is critical.

Failure to do so may lead to scope of practice challenges, medico-legal quandaries and poor practice.

Thank you!