1

Mystery Shopping

SHIP Directors’ Conference

June 11, 2007

Julie Leonard & Erika Melman

BearingPoint, Inc.

2

Topics

Mystery Shopping – What is it?– Why Mystery Shop?– CMS and Mystery Shopping– Mystery Shopping Toolkit

Specifics of Mystery Shopping– Goals– Planning and Steps– Benefits of Mystery Shopping

Questions & Answers

3

Mystery shopping is: A beneficiary-focused self-assessment tool that allows you to view, understand, and evaluate your SHIP program from your clients’ perspective.

Surveys

Web site Review

Reports

AuditsMystery Shopping

Mystery Shopping: What is it?Mystery Shopping: What is it?

Mystery shopping is just one piece of the puzzle in quality assurance and process improvement

4

Mystery shopping lets you see your program

through the eyes of your clients

5

What Mystery Shopping is NOT…

Mystery Shopping is a tool used to gauge overall program performance for the purpose of enhancing counseling services.

It should NOT be used as a punitive measure to reprimand a staff member or volunteer.

6

Why Mystery Shop?

Mystery shopping your program can help you:

Identify where training and other process improvement efforts are necessary

Implement a low-cost but highly effective self-assessment

Learn about your program through the data you collect

Enhance the counseling services you provide to your clients.

7

Past CMS Mystery Shopping Projects

1) The SHIP Program Accuracy

– Preventive Benefits– Part D knowledge– Basic Medicare

Access– Counselor availability and responsiveness– Call wait time– Ease of scheduling in-person appointments

Customer Service– Telephone etiquette – The manner in which information was provided

8

Past CMS Mystery Shopping Projects

2) 1-800-MEDICARE

Accuracy– Appropriate use of PDPF tool – Appropriate use of standardized scripts– Basic Medicare knowledge

Access – Average wait times during special events

Customer Service – Greeting– Closing

9

Past CMS Mystery Shopping Projects

3) Medicare Prescription Drug Discount Card Program Sponsors and Medicare Prescription Drug Program Sponsors Compliance

– Marketing practices– Policy

10

How do SHIP Directors

Use Mystery Shopping?

SHIP Directors are interested in mystery shopping

to: Identify training needs, including refreshers Verify message accuracy and consistency Set performance improvement goals Identify best practices and accolades

There is always room for improvement, and this is one of many activities a Director can use to enhance their program.

11

Mystery Shopping Toolkit

Purpose To help you develop a mystery shopping activity

tailored to your program

Contents How to use the mystery shopping toolkit Background of mystery shopping Planning and conducting mystery shopping Templates for mystery shopping

12

Status of Mystery Shopping Toolkit

In progress Development of the toolkit contents

Next steps Test the toolkit with stakeholders in focus groups and

in-depth interviews Pilot test the toolkit with SHIP programs

Your feedback is important!

13

Specifics of Mystery Shopping

14

The GoalDetermine your goal before deciding on

what aspects or behaviors to assess. Sample goals: Access

– Provide timely assistance to callers– Reduce wait time on telephone – Provide timely access to face to face counseling– Provide access to services for people with physical

disabilities and other special needs

15

Sample Goals (Continued)

Accuracy– Provide correct information to clients– Provide all relevant information to clients– Customer Service – Provide a friendly and safe environment to clients

Referrals – Refer the client to the correct organization or

agency, as appropriate and necessary

16

Sample Goals (Continued)

Compliance – Ensure client privacy before, during, and after

counseling service

Publications– Provide to the client all relevant CMS and state

specific publications

17

What do mystery shoppers do and how do they do it?

Mystery shoppers Pose as clients of a SHIP program Follow a specific script (scenario) Record their experience Are trained to behave like typical clients

Mystery shopping is done by Telephone In-person

18

Steps of Mystery Shopping

1. Decide what you want to assess.

2. Decide whether to use in-person or telephone mystery shopping.

3. Decide what kind of information to collect.

4. Decide whether the shopper should pose as a Medicare beneficiary, or a family member, friend, or neighbor.

5. Select shoppers.

19

Steps of Mystery Shopping (Continued)

6. Develop a scenario (and pilot test it).

7. Develop a form to record the experience.

8. Develop a mystery shopping schedule.

9. Train the shoppers.

10. Conduct the mystery shopping.

11. Analyze the information collected.

12. USE THE RESULTS.

20

Step 1:

Decide What You Want to Assess Determine aspects to assess

Example Goal: Provide the client with timely access to a counselor.

Objective that supports the goal Aspect to be assessed Information needed

REMEMBER: The goal helps guide all of the other steps of the mystery shopping process.

21

Step 2: Telephone or In-Person Mystery Shopping?

Choose your Mystery Shopping Method

The appropriate method to use is determined by: What you want to assess (topic) Available resources Time

22

Step 3: Decide What Kind of Information to Collect

Determine what type of information is best

Ask yourself what kind of information will help you assess your program.

In general, qualitative information is documented in words; quantitative information is documented in numbers.

The type of information that you want to collect will affect the kinds of questions that your shopper will ask the counselors.

23

Step 4: Decide on the Shopper Persona

Choose a Mystery Shopper Persona

The shopper can pose as: A Medicare beneficiary A family member A friend or neighbor

24

Step 5:Select the Shoppers

Choose a Mystery Shopper

Mystery shoppers can be: Temporary workers hired by you Your staff or volunteers Employees of a mystery shopping firm

25

Step 6: Develop Mystery Shopping Scenario

Create a Scenario

The elements of the scenario are: Background information of the shopper Shopper questions Knowledge checks

26

Step 7: Develop Form

Create a form to collect shopping data

Purpose: Form for the shopper to record information about the

call or visit

Type of Information: Relevant data elements based on your goal

Each telephone mystery shopping call or in person visit should have its own form

27

Step 8: Develop Mystery Shopping Schedule

Your goals set the shopping schedule Examples of different schedules to test access:

– Normal business hours. – Periods of high call volume – During major events

28

Step 9: Train the Shoppers

Tailor training to meet your goal Possible training methods:

– General training session – Walk-through of the data collection form– Role-playing

29

Step 10: Conduct the Activity

Mystery Shop!

Shoppers should begin conducting in-person visits and telephone mystery calls at this time.

30

Step 11: Analyze the Data

Compile and Assess your Data

To analyze the data: Aggregate information from calls/visits into one file Assess trends

The number of mystery shopping calls or visits you conduct should be in proportion to your normal call or visit volume in order to apply your findings across your entire program.

31

Step 12: Use the Results

Put your results to use

The data that you collect from mystery shopping your program will help you meet your goal for the mystery shopping project.

The data will help you improve your program.

32

Benefits of Mystery Shopping

With mystery shopping, you can:

1. Get real-time, simple, and inexpensive feedback.

2. Evaluate accuracy of information provided, as well as customer service.

3. Identify strengths and weaknesses of your program.

4. Ensure that staff follow agency standards and policies.

5. Improve training.

33

Questions

Thank you for your participation