04/18/2304/18/23 11

Guide to MN WorkForce Center System Guide to MN WorkForce Center System Job Seeker Satisfaction SurveyJob Seeker Satisfaction Survey

Making Sense of the Semi-Annual Making Sense of the Semi-Annual Charts and ReportsCharts and Reports

04/18/2304/18/23 22

About the SurveyAbout the SurveyTelephone interviews are conducted with: Telephone interviews are conducted with:

Exiters from Workforce Investment Act (WIA) Adult, WIA and Exiters from Workforce Investment Act (WIA) Adult, WIA and State Dislocated Worker, WIA and State Youth, and State Dislocated Worker, WIA and State Youth, and Minnesota Family Investment Program (MFIP)Minnesota Family Investment Program (MFIP)

Exiter and in-progress Vocational Rehabilitation Services Exiter and in-progress Vocational Rehabilitation Services (VRS) and State Services for the Blind (SSB) customers(VRS) and State Services for the Blind (SSB) customers

Core Services customers (“universal customer,” formerly Job Core Services customers (“universal customer,” formerly Job Service). Service).

Visit the link below for specific selection criteria information: Visit the link below for specific selection criteria information: http://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/Job_Seeker_Satisfaction/Background_2.aspxhttp://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/Job_Seeker_Satisfaction/Background_2.aspx

04/18/2304/18/23 33

About the SurveyAbout the Survey

Survey Vendor: Strategic Research Group Survey Vendor: Strategic Research Group (SRG)(SRG)

Chosen based on best response to our Request for Chosen based on best response to our Request for ProposalsProposals

Based in Columbus, OhioBased in Columbus, Ohio

Has conducted WIA customer satisfaction surveys for Has conducted WIA customer satisfaction surveys for State of Ohio (our vendor since May 2005)State of Ohio (our vendor since May 2005)

Out-of-state vendor helps limit survey biasOut-of-state vendor helps limit survey bias

04/18/2304/18/23 44

About the SurveyAbout the Survey

Survey questions provide a “score” as well as Survey questions provide a “score” as well as descriptive comments:descriptive comments:

Minnesota Customer Satisfaction Index (MnCSI) is a Minnesota Customer Satisfaction Index (MnCSI) is a numeric score based on 3 standard questions about numeric score based on 3 standard questions about general satisfaction (more on this in the ‘Questions’ general satisfaction (more on this in the ‘Questions’ slide).slide).

Descriptive comments are from open-ended questions Descriptive comments are from open-ended questions asking about what services/activities program or WFC asking about what services/activities program or WFC should START doing, STOP doing, KEEP doing.should START doing, STOP doing, KEEP doing.

04/18/2304/18/23 55

About the SurveyAbout the Survey

The survey presents a broad view of customer The survey presents a broad view of customer satisfaction:satisfaction:

Local area results are based on balanced input from Local area results are based on balanced input from local area customers. local area customers.

Results represent a range of customers: most satisfied, Results represent a range of customers: most satisfied, least satisfied, and everyone in-between.least satisfied, and everyone in-between.

Results are produced semi-annually from on-going Results are produced semi-annually from on-going survey.survey.

04/18/2304/18/23 66

Why is this survey important?Why is this survey important?

Job Seeker Satisfaction Survey (JSSS) is based on Job Seeker Satisfaction Survey (JSSS) is based on large, random samples; these produce a credible picture large, random samples; these produce a credible picture of overall satisfaction for the whole system / program / of overall satisfaction for the whole system / program / local area.local area.

JSSS focuses on the customer’s entire services JSSS focuses on the customer’s entire services experience, particularly in relation to finding a job. experience, particularly in relation to finding a job.

JSSS complements results of Mystery Shoppers that JSSS complements results of Mystery Shoppers that focus on quality of WorkForce Center facilities and short-focus on quality of WorkForce Center facilities and short-

term interaction with staff.term interaction with staff.

04/18/2304/18/23 77

Why is this survey important? (cont.)Why is this survey important? (cont.)

JSSS is on-going, with new results each quarter; this JSSS is on-going, with new results each quarter; this allows tracking over time of customer response to allows tracking over time of customer response to changes in service strategy.changes in service strategy.

JSSS provides detailed understanding of the customer’s JSSS provides detailed understanding of the customer’s experience by capturing verbatim responses to “which experience by capturing verbatim responses to “which services were most help [in getting a job]” and “what services were most help [in getting a job]” and “what about the services could be improved”.about the services could be improved”.

04/18/2304/18/23 88

Reports ProducedReports Produced

WorkForce Center System (minus MFIP & any WorkForce Center System (minus MFIP & any additional extra sample, e.g. VRS, DW)additional extra sample, e.g. VRS, DW)

Each major program at the statewide levelEach major program at the statewide level

Each local area, all programs combined. Each local area, all programs combined.

Each local area, extra sample reports; VRS Each local area, extra sample reports; VRS Team report & DW Provider reportTeam report & DW Provider report

04/18/2304/18/23 99

SamplesSamples For a WorkForce Service Area (WSA), the proportion of For a WorkForce Service Area (WSA), the proportion of

interviews from a program roughly equals that program's interviews from a program roughly equals that program's portion of the total customer population.portion of the total customer population.

One exception: Core Services customers receive less-One exception: Core Services customers receive less-intense services than customers in other programs, so intense services than customers in other programs, so the Core Services population is halved. the Core Services population is halved.

Example:Example:

ProgramProgram Population in WSAPopulation in WSA SampleSampleAdult Adult 80 80 8 8Dislocated WorkerDislocated Worker 600 600 60 60 YouthYouth 240 240 24 24VRSVRS 900 900 90 90Core ServicesCore Services 2800 (use 1400) 2800 (use 1400) 140 140

04/18/2304/18/23 1010

SamplesSamples

SSB and MFIP are not included in local area SSB and MFIP are not included in local area samples or results.samples or results.

SSB program results are based on a statewide SSB program results are based on a statewide sample (~262 interviews over 12 months).sample (~262 interviews over 12 months).

MFIP program results are based on a statewide MFIP program results are based on a statewide sample (~376 interviews over 12 months). sample (~376 interviews over 12 months).

04/18/2304/18/23 1111

QuestionsQuestionsThree questions ask about general satisfaction:Three questions ask about general satisfaction:

Overall satisfaction with the services?Overall satisfaction with the services? To what extent have the services met expectations?To what extent have the services met expectations? How well do the services compare with idealHow well do the services compare with ideal??

Questions are asked on a 1-to-10 scale where “1” is Questions are asked on a 1-to-10 scale where “1” is least satisfied and “10” is most satisfied.least satisfied and “10” is most satisfied.

Responses to the three questions are averaged and Responses to the three questions are averaged and converted to 0-to-100 scale to create MnCSI (Minnesota converted to 0-to-100 scale to create MnCSI (Minnesota Customer Satisfaction Index).Customer Satisfaction Index).

MnCSI can be used to compare results over time. MnCSI can be used to compare results over time.

04/18/2304/18/23 1212

QuestionsQuestionsTwo questions asked about staff:Two questions asked about staff:

How satisfied that staff understand your needs?How satisfied that staff understand your needs? How satisfied that staff responded to request for help in reasonable How satisfied that staff responded to request for help in reasonable

amount of time? amount of time? (response scale - very satisfied to very dissatisfied)(response scale - very satisfied to very dissatisfied)

Two additional questions asked (for all programs except Core Two additional questions asked (for all programs except Core Services) also use “very satisfied” to “very dissatisfied” Services) also use “very satisfied” to “very dissatisfied”

How satisfied that were given enough information to make good choices How satisfied that were given enough information to make good choices for employment plan?for employment plan?

How satisfied that had an active role in decisions about services?How satisfied that had an active role in decisions about services?

Core Services questions: How satisfied that were given Core Services questions: How satisfied that were given enough help in the Resource Area? Have you used enough help in the Resource Area? Have you used Minnesotaworks.net to look for openings?Minnesotaworks.net to look for openings?

If YES to using MNW, then job seeker is asked “Did you find that it was If YES to using MNW, then job seeker is asked “Did you find that it was quite easy, somewhat easy, somewhat difficult, or quite difficult to use?quite easy, somewhat easy, somewhat difficult, or quite difficult to use?

04/18/2304/18/23 1313

QuestionsQuestionsThree open-ended questions ask Three open-ended questions ask (customers asked only 2 of these 3 questions):(customers asked only 2 of these 3 questions):

What would you like the program/WFC to START doing?What would you like the program/WFC to START doing? What would you like the program/WFC to STOP doing?What would you like the program/WFC to STOP doing? What would you like the program/WFC to KEEP doing?What would you like the program/WFC to KEEP doing?

Verbatim responses sorted into categories (Verbatim responses sorted into categories (e.g., service e.g., service quality, staff, level of service)quality, staff, level of service)

Each category reported as a percent of total responsesEach category reported as a percent of total responses

Customers often mention more than one topic in their Customers often mention more than one topic in their response response (e.g., “the computer class, learning what they are looking for in a

resume, and the interview techniques . . .”)

04/18/2304/18/23 1414

Three Things to Keep in Mind When Viewing Three Things to Keep in Mind When Viewing Quarterly MnCSI ResultsQuarterly MnCSI Results

Results fluctuate from quarter to quarter; watch them over Results fluctuate from quarter to quarter; watch them over a longer period. After five or six quarters, ask:a longer period. After five or six quarters, ask:

Is there any consistent pattern to our WSA [program] score? Is there any consistent pattern to our WSA [program] score? Is the score generally rising or falling? Is the score generally rising or falling? If we changed our services or processes, did the score change? If we changed our services or processes, did the score change?

Smaller samples give less exact results. Always check the Smaller samples give less exact results. Always check the width of the confidence intervals.width of the confidence intervals.

It’s best not to compare scores among WSAs. Differences It’s best not to compare scores among WSAs. Differences may be due to variations in sample make-up or:may be due to variations in sample make-up or:

local economic conditionslocal economic conditions customer-to-staff ratioscustomer-to-staff ratios financial resourcesfinancial resources program strategiesprogram strategies customer characteristics, etc. customer characteristics, etc.

04/18/2304/18/23 1515

Overall Satisfaction with Services: Overall Satisfaction with Services: Local Area, Region, StateLocal Area, Region, State

Track the changes over time. Remember, the larger the sample Track the changes over time. Remember, the larger the sample (the bigger the “N”) the more precise the result.(the bigger the “N”) the more precise the result.

Local area MnCSI scores are put in context by comparing to State Local area MnCSI scores are put in context by comparing to State and corresponding region (Metro or Greater MN) MnCSI scores.and corresponding region (Metro or Greater MN) MnCSI scores.

The chart uses an enlarged scale range (60.0 – 80.0) so you can The chart uses an enlarged scale range (60.0 – 80.0) so you can better see trends in MnCSI scores.better see trends in MnCSI scores.

Customer Satisfaction: Lake Woebegone WSA Average MnCSI over past 5 quarters (12-month rolling)

(N=Current N to date)

60.0

65.0

70.0

75.0

80.0

Apr 05-Jun 05 Apr 05-Sept 05 Apr 05-Dec 05 Apr 05-Mar 06 April 05-June 06Year-to-date by Quarter

Min

nes

ota

Cu

sto

me

r S

ati

sfa

cti

on

In

de

x

(Mn

CS

I)

Lake Woebegone(N=455)

Minnesota (N=4781)

Greater MN(N=2834)

04/18/2304/18/23 1616

Overall Satisfaction with Services: Overall Satisfaction with Services:

Local Area/Program ScoresLocal Area/Program Scores

This chart shows average MnCSI score over time. This chart shows average MnCSI score over time. Examine the Examine the confidence intervalsconfidence intervals, the black bars extending above , the black bars extending above

and below the average scores. The bar indicates we are 95 percent and below the average scores. The bar indicates we are 95 percent confident the true score lies within the length of the bar. The confident the true score lies within the length of the bar. The narrower the bar, the more precise the MnCSI score.narrower the bar, the more precise the MnCSI score.

Think of confidence intervals as “margin of error,” much like political Think of confidence intervals as “margin of error,” much like political polls that refer to a “margin of error of plus or minus three points.”polls that refer to a “margin of error of plus or minus three points.”

Customer Satisfaction: Lake Woebegone WSA

Average MnCSI over past 5 quarters (12-month rolling)

(Current to date N=455)

71.6 73.5 74.3 74.4

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Apr 05-Jun 05 Apr 05-Sept05 Apr 05-Dec05 Apr 05-Mar 06 Apr 05-June06

Year-to-date by Quarter

Min

nes

ota

Cu

sto

me

r S

ati

sfa

cti

on

In

de

x

(Mn

CS

I)

Lake Woebegone

04/18/2304/18/23 1717

Summaries of answers to open-ended Summaries of answers to open-ended questionsquestions

Tables provide a quick Tables provide a quick summary of topics mentioned summary of topics mentioned in answers to: in answers to:

What would you like the What would you like the programs/WFC to START, programs/WFC to START, STOP, KEEP doing?STOP, KEEP doing?

The local area summary is put The local area summary is put in context by comparing to in context by comparing to State and region (Greater MN State and region (Greater MN or Metro) summaries .or Metro) summaries .

04/18/2304/18/23 1818

What would you like the program/WFC to What would you like the program/WFC to START, STOP, KEEP doing?START, STOP, KEEP doing?

Responses categorized into 6 Responses categorized into 6 to 7 major groups (issues or to 7 major groups (issues or help finding and searching for help finding and searching for a job, Staff helpfulness or a job, Staff helpfulness or improvements, etc.) and many improvements, etc.) and many more detailed sub-groups.more detailed sub-groups.

Compare to State and Region Compare to State and Region (Metro or Greater MN) by (Metro or Greater MN) by looking across the rows.looking across the rows.

04/18/2304/18/23 1919

Comments on Good Services, Comments on Good Services, September 2009 ResultsSeptember 2009 Results

YouthYouth• ““My counselor went way beyond just finding me a job, he hooked

me up with finding scholarships, that is what I am most thankful for.” (WSA 3)

Rehab ServicesRehab Services• “A program called Brighter Futures. All of these programs boil

down to the individual that has hands on contact. The person that I had contact with was wonderful.” ( WSA 5)

• “My job coach. They helped me adjust problems to my employer properly, and also helped me find affordable transportation.” (WSA 17)

Adult / Dislocated WorkerAdult / Dislocated Worker• “The weekly group session, job club, we got a lot of different

ideas on how to react in interviews.” (WSA 8)

04/18/2304/18/23 2020

Feedback on Good Services (cont.)Feedback on Good Services (cont.)• “Resume writing, interviewing skills, creative job searching. They

seemed to be very current in terms of what employers wanted.” (WSA 12)

• “I went to a three day boot camp; it was beneficial to see how things changed over the years. I also received two computer classes and I really enjoyed those.” (WSA 16)

Core ServicesCore Services• “I had a one on one meeting with a job placement worker and he

spent a lot of time going over possible classes, training and useful internet job searching tools.” (WSA 4)

• “They are actually helping me look for jobs by providing referrals via email and phone calls.” (WSA 12)

• “The combination of everything is important, talking one-on-one with a counselor is important so that they can get a feel for who you are and your needs. The classes and knowing the website is important. There is a wide range of needs so one-on-one talking is important.” (WSA 17)

04/18/2304/18/23 2121

Flexibility of the SurveyFlexibility of the Survey

Local areas that would like more detailed results Local areas that would like more detailed results may purchase a larger sample so their results can may purchase a larger sample so their results can be broken-out by program.be broken-out by program.

(Two programs, Voc Rehab Services and the Dislocated Worker (Two programs, Voc Rehab Services and the Dislocated Worker program have enlarged their samples to break-out results down to program have enlarged their samples to break-out results down to local levels. VRS to 18 RAM or Team level and DW to 22 local area local levels. VRS to 18 RAM or Team level and DW to 22 local area providers)providers)

Local areas or programs can explore additional Local areas or programs can explore additional topics by purchasing additional questions asked just topics by purchasing additional questions asked just of their customers.of their customers.

(State Services for the Blind and LMI Research have purchased (State Services for the Blind and LMI Research have purchased additional questions.)additional questions.)

04/18/2304/18/23 2222

Sample Job Seeker Web PageSample Job Seeker Web PageLook for the link to your local area or program at Look for the link to your local area or program at

these following linksthese following links

System System http://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/Job_Seeker_Satisfaction/Entire_WorkForce_System_Results.aspxhttp://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/Job_Seeker_Satisfaction/Entire_WorkForce_System_Results.aspx

ProgramProgramhttp://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/Job_Seeker_Satisfaction/Results_by_Program.aspxhttp://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/Job_Seeker_Satisfaction/Results_by_Program.aspx

WSA’sWSA’shttp://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/Job_Seeker_Satisfaction/Results_by_Workforce_Service_Area.aspxhttp://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/Job_Seeker_Satisfaction/Results_by_Workforce_Service_Area.aspx

04/18/2304/18/23 2323

Need Further Information?Need Further Information?

Visit Visit http://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/index.aspx http://www.positivelyminnesota.com/About_Us/Customer_Satisfaction/index.aspx to to view the entire Customer Satisfaction Survey results.view the entire Customer Satisfaction Survey results.

Contact Devin Colvin of the Policy, Planning and Measures Unit via Contact Devin Colvin of the Policy, Planning and Measures Unit via email, devin.colvin@state.mn.us or by phone, 651-259-7190, with email, devin.colvin@state.mn.us or by phone, 651-259-7190, with questions, comments or requests about the presentation or results.questions, comments or requests about the presentation or results.