Development of a Rating Scale to Assess Geropsychology Practice Competencies

Michele J. Karel, PhDVA Boston Healthcare System

Harvard Medical School

For the CoPGTP Task Force on Geropsychology Competency Assessment:Michele Karel, Jeannette Berman, Jeremy Doughan, Erin E. Emery, Victor Molinari, Sarah Stoner, Yvette N. Tazeau, Susan K. Whitbourne, Janet Yang, Richard Zweig

APPIC Conference, April 18, 2009

Objectives

Increase awareness of efforts to define and evaluate competencies for geropsychology practice

Consider challenges for designing and implementing a competency rating scale Utility in internship and postdoc settings

Address ongoing challenges and directions for work in this area

Importance of Professional Geropsychology Training and Evaluation Aging population Workforce shortages for geriatric health care, across

disciplines Relatively few psychologists have formal training in

geropsychology Increasing numbers will be working with older adults

How to evaluate competence, and define training needs, for psychological practice with older adults

How to help training programs, and post-licensure psychologists, expand opportunities for geropsychology training

Growth of Professional Geropsychology Growing number of grad, intern, and fellowship programs

offering geropsychology training Growth of research, journals, professional organizations Several recent milestones:

APA Office on Aging, established 1998 APA Guidelines for Psychological Practice with Older Adults (2004) CRSPPP recognition of gero as proficiency area, 1998 and 2005 Pikes Peak Model for Geropsychology Training, 2006 National

Training Conference Pikes Peak Training Model* defined Attitude, Knowledge,

Skill competencies for geropsychology practice How to evaluate professional geropsychology

competence?

*Knight, B.G., Karel, M.J, Hinrichsen, G.A., Qualls, S.H., & Duffy, M. (in press). Pikes Peak model for training in professional geropsychology. American Psychologist.

Project Initiation

Council of Professional Geropsychology Training Programs (CoPGTP) Formed after 2006 Pikes Peak conference For training programs at all levels www.uccs.edu/~cpgtp/

CoPGTP undertook to explore methods to evaluate Pikes Peak competencies

CoPGTP established a Task Force Agreed to start with goal of developing a

competency rating scale

Project Goals

Develop an evaluation tool that would: Address the Pikes Peak geropsychology

attitude, knowledge and skill competencies Be useful for supervisor evaluation of trainees

Formative and summative evaluation Be useful for self-evaluation by learners Be useful across levels of training

Graduate practicum, internship, fellowship, post-licensure

Be useful to help define ongoing training goals

Pikes Peak Geropsychology Practice Competencies Competency definitions informed by:

APA Guidelines for Psychological Practice with Older Adults (2004)

Cube Model for Competency Development (Rodolfa et al., 2005)

Psychology foundational and functional competencies develop over course of training career

2006 Pikes Peak Conference process: Working groups address 6 competency domains

(separate working groups address 4 levels of training) Iterations of competencies lists during conference, and

afterwards via listserv discussion

Pikes Peak Competencies

Are aspirational Aim to facilitate as many psychologists as possible to

work effectively with older adults, not to pose barriers to joining the field

Competencies are aimed at level of practice of a newly licensed psychologist who has finished one year of geropsychology postdoctoral training

Competencies not geared to level of an “expert”, but to someone who practices with wide range of older adults, families, systems

Pikes Peak Competencies: Attitudes

Work within one’s scope of competence Recognize how one’s beliefs about aging and

older adults may affect practice Appreciate diversity among older adults, and

interactions between age/cohort and other aspects of individual diversity

Seek continuing education, supervision, consultation

Pikes Peak Competencies: Knowledge Base General knowledge about adult development,

aging, and the older adult population Foundations of professional geropsychology

practice Assessment Intervention Consultation

Pikes Peak Competencies: Skills Professional Geropsychology Functioning

i.e., foundational skills applied to geropsychology practice

Assessment Intervention Consultation

Note: Skills for functional domains of research, supervision, and management are important for growth of geropsychology, but were not included in Pikes Peak core competencies for practice with older adults

Pikes Peak Competencies:Settings of Care Geropsychologists should be able to deliver services

effectively in range of (at least 2) settings, such as: Outpatient mental health services Outpatient primary care/medical settings Inpatient medical service Inpatient psychiatric service Long-term care settings including nursing homes, assisted

living facilities, home care, day programs Rehabilitation settings Hospice Others

CoPGTP Competency Evaluation Task Force Methodology ~Monthly conference calls, 9/07 to 7/08 Reviews of relevant, informing materials Small working groups define “behavioral anchors” for

each competency domain Iterative reviews of all scale items by larger group Group discussion, selection, editing of:

Rating scale/anchors and vignette Introduction to the evaluation tool

Task Force and other CoPGTP members try out the scale to evaluate: a student, oneself, or both

Informing Materials: Psychology Competencies Assessment Guiding principles and recommendations for the

assessment of competence (Kaslow et al., 2007 and other publications of APA

Task Force on Assessment of Competence) Assessment of Competency Benchmarks Work

Group A developmental model for defining and measuring

competence in professional psychology Joint effort of APA BEA and CCTC

The Practicum Competencies Outline Hatcher & Lassiter (2007), workgroups of ADPTC and

CCTC

Informing Materials:Geropsych Competency Assessment Example of evaluation tools used in various

geropsychology practicum, internship, and postdoctoral programs

Also, examples of competencies and evaluation methods from: Medicine – ACGME Outcome Project, 2000 Geriatrics, Palliative Care, Geriatric Social Work,

Geriatric Psychiatry

Delineation of Geropsychology Knowledge and Skill Competencies Each Pikes Peak competency domain (e.g.,

foundational geropsychology skills) is defined by 3-8 specific competencies (e.g., ethical practice, appreciation of diversity, self-reflection…)

We aimed to further specify each of these competencies with behavioral anchors How would you know it if you saw it (in oneself or

another)? e.g., what does it really mean to “Relate effectively and

empathically with older adults clients, families, and other stakeholders in a range of professional roles and settings?”

This approach leads to a very long evaluation tool!

Example: Delineating one of the foundational skill competenciesB. Professional Geropsychology Functioning – Foundational

SKILLS -- The psychologist/trainee is ABLE TO:

5. Relate Effectively and Empathically N I A P E a. Use rapport and empathy in verbal and nonverbal behaviors to

facilitate interactions with older adults, families, and care teams

b. Form effective working alliance with wide range of older clients, families, colleagues, and other stakeholders

c. Communicate new knowledge to patients and families, adjusting language and complexity of concepts based on the patient and family’s level of sensory and cognitive capabilities, educational background, knowledge, values, and developmental stage

d. Demonstrate awareness, appreciation, and respect for older patient, family, and team experiences, values, and conceptual models

e. Demonstrate appreciation of client and organizational strengths, as well as deficits and challenges, and capitalize on strengths in planning interventions

f. Tolerate and understand interpersonal conflict and differences within or between older patients, families, and team members, and negotiate conflict effectively

Example: Delineating one of the assessment skill competencies

B. SKILLS – The psychologist/trainee is ABLE TO: 2. Utilize Screening Instruments N I A P E

a. Utilize screening tools for mood, cognition, substance use, personality, and other clinical issues to guide and inform comprehensive assessment

b. Evaluate age, educational, and cultural appropriateness of assessment instruments

c. Consider reliability and validity data in using standardized instruments with older adults

d. Assess older adult’s ability to provide informed consent for psychological evaluation

e. Recognize sensory impairments and makes environmental modifications accordingly

f. Consider impact of medical conditions and medications on test performance

g. Make specific and appropriate recommendations, based on testing results, to inform treatment planning

Evaluation of Attitudes

Pikes Peak attitudes for competent geropsychology practice are embedded in the foundational skills E.g., self-reflection re: biases/discomforts,

awareness of limits of one’s competence, seeking consultation, seeking continued education

Attitudes are specified in tool’s introduction For now, attitudes not evaluated separately Perhaps they should be? How?

Rating Scale

Developmental rating scale Novice, Intermediate, Advanced, Proficient, Expert

Scale anchors conceptually complex Adapted definitions from Hatcher and Lassiter (2007) Included in anchor definitions how much

supervision/consultation likely needed Provided vignette and approach to that case to

illustrate each level of competence Instructions to rate each Pikes Peak competency, not

each behavioral specifier (but can if desired)

Rating Scale

Abbreviated scale anchors, printed on each page:

N = Novice: Possesses entry-level skills; needs intensive supervision

I = Intermediate: Has a background of some exposure and experience; ongoing supervision is needed

A = Advanced: Has solid experience, handles typical situations well; requires supervision for unusual or complex situations

P = Proficient: Functions autonomously, knows limits of ability; seeks supervision or consultation as needed

E = Expert: Serves as resource consultant to others, is recognized as having expertise

Rating Scale

Expected ratings vary considerably through levels of training Practicum: Novice through Advanced Internship: Novice through Proficient Fellowship and post-licensure: Intermediate

through Expert Ideally, usable to evaluate a trainee, AND to

evaluate oneself

Introduction to the Evaluation Tool

Reviews: Purpose of tool Pikes Peak competencies as aspirational Application across range of geropsychology practice and

training settings Instructions for use

Including that evaluation should include the learner's perspective (self-assessment), observation of the learner's work, and regular supervision involving case discussion

Psychologists and trainees conducting self-assessments can evaluate their training and supervision needs in each area

Definition of rating scale anchors, and illustration of its use through a case vignette

Piloting the Evaluation Tool

Asked CoPGTP members, and ourselves, to complete the tool to evaluate a student or oneself

13 responses 6 CoPGTP members (other than us) 4 Task Force members 3 from Task Force members’ students/staff

Used tool to: 4 to evaluate a student 5 to evaluate oneself 2 evaluated oneself and a student 2 provided general feedback without specifying

Asked for Feedback:

Asked folks to answer these questions: How did you use the tool (to evaluate a student, or

yourself?) What was your general impression of using the tool? Was the introduction to the tool clear…? Was the rating scale (Novice, Intermediate, Advanced,

Proficient, Expert) understandable…? Was the length of the tool a problem? Did it adequately

address breadth/depth of geropsychology competencies?

What do you think the best use of this tool will be? Do you expect to use this tool? In what way? Any other feedback?

Feedback

Most felt tool is very long, but worth it given attention to breadth and depth

A few felt it was too long for practical purposes (Challenge – how to cut?)

Most felt introduction was clear, but needed clarification not to rate each specifier

Most liked the NIAPE rating scale

Feedback

Concerns raised about the scale included: Does it apply as well to knowledge as skills? Does it work as well for practicum level training where

students may be novices in most areas? Is Expert rating too “loaded” (people not comfortable using

that term?) Can scale capture development over time?

If/how best to evaluate attitudes Suggestion to have summary page

Strengths Areas for growth Ongoing training goals

Use of the Scale to Date

Disseminated to CoPGTP training programs and APA 12-2 and 20, PLTC listservs

Programs (grad, intern, and postdoc) are starting to use the tool for: Initial training needs assessment with students Mid-year/final supervisory evaluations Student self-evaluation (encouraging

“metacompetence”)

Use of the Scale to Date

Tool being used by some for program self-study and development Defining training program objectives Evaluating, in discussion with students and

supervisors, how well program is addressing each competency domain

Helping to evaluate competence of new geropsychology supervisors and define their training goals

Being used in community/CE education programs on geropsychology

Uses for Intern/Postdoc Training

Along with APA standards, can help define rotation/track training objectives

Gero evaluation scale is rotation/track specific, and adjunctive to training program’s more general evaluation tools

Extremely helpful for initial training needs assessment because students come in with widely varying previous gero experience Clinical, academic, research

Can tailor initial training plan and review over course of the training year/rotation

It is additional work at evaluation time for supervisor and student – not clear how strong a barrier this may pose

Study-in-the-Works

Aims: Examine utility and validity of the rating scale, initially for self-evaluation purposes Which competencies rated as most vs least well-

developed? Do self-rated competencies relate to years of training

or practice? To settings and major activities of practice? What feedback is provided after completing the scale?

Planning on-line survey Of psychologists who work with older adults and

geropsychology graduate, intern, posdoc students

Eventually, plan longitudinal study of supervisory evaluations of developing geropsychologists

Concerns/Challenges

Will clinicians see such a tool as helpful or somehow restrictive to practice?

Will supervisors/learners use such a long evaluation tool (even if can pick and choose domains of relevance)?

Can competencies be reduced/streamlined? Can we boil down further “core” competencies? Are

some more essential than others? (??) May vary across settings/populations of older adults

Will tool be useful as self-evaluation, self-study aid? Can competency ratings be linked to training

resources?

Opportunities/Future Directions

Helpful for initial training needs assessment with a student, to guide training plan

May help to develop on-line version, then select only domains that are relevant to training program, rotation

Useful as guide to developers of CE programs? Develop organized, sequential CE offerings?

Ultimately, plan to link competencies to training resources and opportunities Working to keep such list updated at CoPGTP website

Expand toolbox of gero competence evaluation tools Largest challenge - how to expand interest in, and

opportunities for, training in professional geropsychology