Copyright 2002-2004 Minnesota Council for Quality1 The 2004 MQA Site Visit

Copyright 2002-2004 Minnesota Council for Quality 1

The 2004 MQA Site Visit


Today’s Agenda

• Purpose of the Site Visit• Elements of a Successful Site Visit• Site Visit Roles and

Responsibilities• Site Visit Preparation• Site Visit Process and Products• What Happens Next


Baseline Questions

• Is this applicant use the Narrative or Baldrige-Express?

• Which Criteria are used?• How long is the site visit? (Usually 3

days.)


Purpose of the Site Visit

• Understand key organization factors• Verify and clarify facts and site visit

issues• Prepare the feedback report• Score the organization• Assess commitment, sustainability,

and momentum• Present exit presentation


Elements of a Successful Site Visit

People

Process Products

Roles,Ground rules,

Communication

Assignments,Controlled agenda,

Consensus building

Exit presentation,Feedback report,Judges’ presentation


Evaluator Team Role & Responsibilities

• Gathers and documents facts for feedback report

• Communication with applicant is equally critical

– Take time to explain and educate – You represent the Council and help

sell the Council– Help make it a fun, pleasant

experience


Applicant Representative Role & Responsibilities• Contributes information for findings• Arranges availability of interviewees and

materials• Helps team complete its agenda• Applicant Representative usually provides:

• List of key contacts• Organization chart• Facility layout• Optional requested material


Presentation to Judges

Information Flow


Site Visit Sequence

Advance Preparation

Final Preparation

Site Visit

1b.Exit

Presentation

1a.Executive Preview

2.Feedback

Report

3. Presentation

to Judges


No

Yes

<= DAY 33

<= DAY 34

<= DAY 41

<= DAY 44

Stage 2: Consensus & Site Visit Planning

2.01Team Leads

assign Category / Itemleads & back ups

2.02Standard Message #3:

Team Leads CommunicateConsensus assignments& meeting arrangements

to team memberscc: Lead Judges &

Chief Judges &Council

2.04Team Leads

& Lead Judgesconfer

2.05 Are matrices

OK?

2.06Team Lead

coachesEvaluator(s)

2.07Team Lead

notifiesEvaluator

2.09Evaluators e-mail

completedConsensus Matrix

& Expected Results Tableto team

2.08Evaluators completeConsensus Matrices

& consolidatedExpected Results Tables

for assigned Items

2.03Evaluators completeConsensus Matrix for

first assigned Item& e-mails to Team Leads &

Lead Judges

2.11Team meets from8AM-3PM to reach

consensus on comments& draft KTs

2.10Evaluators review all

materials beforeConsensus meeting &

identify possiblediscussion points <= DAY 47<= DAY 45

<= DAY 44 2.14Council removes

Evaluator(s) to SVsize if needed;Council sendsSV material

2.17 Are comments

& SVIs OK?

<= DAY 48

SV Strategy WorksheetsExit presentation template

Judges presentation templateFeedback Report template

2.15Category/Item Leads

prepare SV Issuesheets & e-mail to team

& Lead Judges

No

Yes GO TOSTAGE 3

2.18Team Leads

revise Commentsand/or SVI Sheets 2.16

Team Leads& Lead Judges

confer on:comments, SVI's,

storyboardissues, KTs,walkarounds

2.12Participant rep.arrives (3PM);

trainers provideJIT SV training

2.13Category Leads

e-mail Consensuscomments to

team &Lead Judges

<= DAY 30

We are at step 2.12


Advance Preparation• Dates set• Expectations set• Site visit issues compiled – SV Issue sheets

– Prepare list of documents needed and individuals to be interviewed on site – use a matrix to help coordinate requests for meetings and documents

• Site visit agenda set• Logistics arranged

– Hotel– Emergency contact number 612-462-3577– Name tags (as needed)– Expense accounting


Site Visit Planning WorksheetSite Visit Date:________________ Site Visit Location:________________ Evaluator:________________

Category:2 and 5Item Ref #: 2.2 and 5.2

Issue: Verify the content of the Human Resource Plan (2.2) and clarify its use in identifying the strategic training needs for the organization (5.2)

Person(s) to Interview: HR Manager

Document(s) or Information to Review:Human Resource Plan

Sample Questions:Tell me about HR plans you create. How do you use these?What is an example of training that was computed in the past year?

Findings:

Verify or Clarify


Focus of Site Visit Issues

• Information that is missing or vague.• Strengths or weaknesses identified from

narrative, Baldrige Express report, results data, or organizational profile.

• Deployment of the required practices.• Cross-cutting issues or themes• Site Visit Issues should be written on the

“vital few” – there should not be more than 3-4 per item.


Site Visit Techniques• Acceptable Techniques

• Reviewing data, reports, documents hard-copy or electronic as provided directly by the applicant

• Interviewing individuals and teams• Receiving presentations by participating

organization

• Unacceptable Techniques• Conducting focus groups or surveys with

customers, suppliers, or dealers• Disrupting work processes• Interviewing sources or collecting data beyond

that provided by the applicant • This includes conducting library or internet This includes conducting library or internet

research and visiting web sites.research and visiting web sites.


“ Typical” Site Visit IssuesVerify/Clarify• Role and activities of senior management (driving customer

focus, empowerment, innovation, learning)• Degree of involvement and empowerment of employees • Reliable and accessible data and information • Use of facts in decision-making• Focus on customers• Systematic approaches to management• Training effectiveness, on-the-job reinforcement• Alignment of recognition and rewards to support strategy,

customer focus, and key business outcomes• Use of strategic objectives, action plans, and related measures

to align work at all levels—links to performance feedback and employee rewards

• Evidence of evaluation and improvement• Integration of all processes—both core business and support• Supplier involvement in performance improvement activities


“Typical” Walk-around Questions

• What are the values, mission, and vision of the organization?

• What are the action plans for your work unit? • What are the goals?• What role do you play in achieving them?

• What kind of training do you receive?• Is it useful to your work? • How is training reinforced on the job?

• How are you empowered to take actions needed to carry out or improve your work? • How are you involved in decision-making about

work?

• What is recognized or rewarded here?


<= DAY 57

Stage 3: Site Visit

3.04Team arrives

at site

3.01Team arrives

at site visit hotel

<= DAY 583.03

Team preparespreliminary

Story Board *

3.02Team conducts

final prep meetingevening before(see agenda)

3.06Team conducts

preliminary caucus

3.05Participant KickOff presentation& introductions(optional tour)

3.07Team conducts site visit:

interviews, data review, walkarounds;frequent team caucuses; daily revision

of storyboard (reviewed with HRO at EOD)and comment worksheets/SVIs

<= DAY 60

3.09Team Leads/Council

meet w/Participant HRO

3.08Team prepares

exit presentations

<= DAY 60<= DAY 60<= DAY 60

3.10Team makes

exit presentation

3.11Team finalizes

Item comments and scores

3.12Team completes

Executive Summary(KTs), PGPs, Judges'

questions

3.14Team provides improvement

input for process, completes 360reviews, conducts final checklist,

& leaves site/hotel

<= DAY 60

GO TOSTAGE 4

* The story board process is coveredin the site visit training.

3.13Team Leads

appoint feedbackwriter (optional)


• Final prep meeting held evening before site visit– Review SV issue sheets– Set up key themes worksheet/storyboard– Establish walk-around questions– Finalize the agenda, assignments, pairs

• Each Evaluator brings draft of site visit issue worksheets, to prepare a composite set

• Applicant representative meets with the team• Aid in planning the time on site• Share general schedule conflicts

Final Preparation


Final Preparation (continued)

• Review and update documentation list as needed

• Review and update list of individuals to interview on site • This will help you identify and manage

hand-off issues.• Finalize the matrix of document and

meeting requests the team created earlier in the processes


Site Visit Interview Matrix

Individual to Interview Site Visit Issue Ref.

Sheila Knight, Marketing 1.1.1, 2.2.1, 3.1.1, 3.1.2, 3.1,3, 3.2.1, 3.2.4, 4.1.1, 4.1.3, 4.2.1, 4.2.2, 7.1.3

Ron McDonald, Administration 1.1.1, 1.1.3, 1.2.1, 1.2.2, 7.2.3

Mary Stewart, Employee Development

1.1.1, 2.1.1, 5.1.1, 5.1.3, 5.2.1, 7.3 all

Alex Hamilton, Operations 1.1.1, 2.1.1, 6.1.3, 6.2.1, 6.2.2, 7.5 all

John Jones, Finance 4.1.4, 4.2.1, 4.3.2, 7.2 all

Leader/manager of Process Improvement Team & members of PI Team

6.1.5

Individual responsible for coordinating leadership communication

1.2.5, 2.2.2

Individual responsible for Support Services Process Improvement Meetings with Leadership

6.2.5


Sample Site Visit Schedule

Monday

Vanessa (1), (2) 7.2Walt (3, 7.1)

Roy & (4, 5, 7.3) Deana (6, 7.4)

7:00 Breakfast

7:30 Travel to Site

8:00 Opening Meeting

9:00 Ron McDonald, CEO & Leadership Team Meeting

Worker Services Team Team Meeting

10:00 Category Lead Meeting, then Quick Caucus

11:15 One on ones with Eos during lunch breaks

1:30 Nate O. Forsight Planning VP

Mary Stew, Dir IT Development

2:35 Caucus

3:00 Customer Team * Sheila Knight (Mktng)

Alex Hamilton, Operations

4:15 Walk around, Shift Change Interviews

5:00 Caucus, revise plan in detail for Day 2


Site Visit Agenda• Opening meeting

– Introductions to the applicant (MCQ/Team Leads)– Participant presentation (up to one hour)

• Facilities tour (optional)• Scheduled meetings with key applicant

resources – Allow time for travel when necessary– Schedule applicant interviewees

• Scheduled times for walk-around interviewing of applicant employees

• Scheduled times for team discussion (caucuses)

• Executive preview and exit presentation


Managing Issues and Handoffs• Create 7 flip charts, one for each

category.

• Ask Evaluators to use yellow sticky notes to record issues they have to hand off to the category owner in the “open” area and red for new issues.

• When the information has been collected, record the answer and move the sticky to “closed” portion.

• This way everyone can keep track of open and closed issues and effectively manage handoffs.

Category 1

Open

Closed

Verify deployment

between shifts

Ask CEO about employee focus (4)


Key Themes Worksheet/Storyboard• A key theme is a Strength or OFI that is

common to more than one Item or Category, is especially significant in terms of the applicants Key Factors, and/or addresses a Core Value of the Criteria.

• Provides key points and an overall summary of the team’s evaluation of the applicant.

• Divided into three sections to address three questions concerning important Strengths, significant OFIs, and key Results

• Will be used to help prepare the exit presentation


Evaluator Team Caucuses

• Compile findings• Review feedback comments• Update SV issue sheets and key themes• Hand-off any outstanding questions• Remember - applicant representative

does not participate except for logistical issues and planned review of key themes

• Be sure to schedule enough time for caucuses plus some personal break time


Structure for Interviewing• Begin the interview with:

– Introduce yourself and why you’re there•We will respect confidentiality of information

•We will be looking for both Strengths and OFIs

– Ask if they have questions– Explain why you are taking notes– Describe the time limits

• End the interview with:– Do you have any questions for me?– Thank them for their time and honesty


Effective Interviewing Tips

• Refocus when they ramble• Ask open-ended questions, probe when

necessary, bring it to a conclusion• Script your lead and follow-up questions• Be frank and respectful

– Use the applicant’s terminology– Actively LISTEN to the applicant

• Be patient with shy interviewees


Asking QuestionsUse open-ended questions•Tell me about …•How do you …•Help me understand …•Please walk me through …

Avoid Leading questions.• Why don’t you – • At my office we do it this way. What do you do?

•Have you always done it this way?•Why did you change?•How did you know what to change?

•Don’t you think it would be a good idea to re-engineer your manufacturing process?

•How do you determine how frequently to check this process?

•Why do you only audit this process every two years?

•How do you know customer requirements are correctly defined?

•Why don’t you involve customers in your design-build teams?


Typical Daily Progress (Day 1)

• Advanced level of understanding of business factors

• Close some Site Visit Issues (typically Cat. 1 and 2)

• Establish remaining agenda• Brief applicant on progress and

outstanding issues• Category Owners begin editing their

comments



• Confirm key themes• Close 100% of remaining Site Visit Issues• Identify remaining information/interviews

needed (if any – there should not be any)• Finalize plan for Day 3• Brief applicant on progress, outstanding

issues, and plans for Day 3• Finalize 100% of the feedback comments



• Close remaining issues (if any)• Prepare and deliver executive preview and exit

presentation• Finalize feedback report

– Category Owners recommend final wording for comments– Entire team reviews and signs off on items as completed

• Identify any Pretty Good Practices• Score• Assess commitment, sustainability, and momentum• Complete logistics (hotel checkout, return of

applicant materials, collection of shredding, 360 feedback)


Preparing the Exit Presentation

• Review the key themes worksheet– Significant Strengths, including Pretty Good

Practices– Significant Opportunities for Improvement

• Prepare the PowerPoint presentation• Identify presenters


Executive Preview

• Applicant’s option• Team Leads only• Council representative also present• Summary review of exit presentation

content• Opportunity for clarifying questions• Expression of thanks for cooperation

and participation


Exit Presentation

• One hour• Moderated by MCQ representative• Audience determined by applicant• Review key themes

– Significant Strengths and Pretty Good Practices

– Significant Opportunities for Improvement

• Clarifying Q&A• Thanks


Scoring• Assign scoring facilitator• Determine order for scoring items

– Allow 5-10 minutes maximum per item

• Review scoring guidelines, terminology(refer to Clarifying Requirements for Basic-, Overall-, and Multiple-

Level Scoring)

• For each item– Category Owner reviews high points of comments– Team members individually determine scores– Team displays scores on facilitator’s signal– Team negotiates variances

• Be mindful of item point values• Don’t bog down

• Record percent scores on scoring matrix Remember--It’s not over till it’s over


Preparing the Feedback Report

• At site– Category owners prepare final wording of comments– Team reviews and signs off on comments– Team reviews judges’ questions

• After site (within one week)– Feedback editor collates comments into feedback

template– Feedback editor writes executive summary based on Key

Themes– Feedback editor distributes report to team for feedback– Team leader forwards finished feedback to lead judges

• Next steps (approximately two weeks)– Judges review and edit final report and forward to Council– MCQ forwards finished report to applicant


Reminder About Comments

• Review writing rules for all comments (refer to your training manual)

• Category 7 comments must:– Address results, not approach, deployment,

integration, or learning– Reflect the measures the applicant says are

important (expected results table)– Be stand-alone and quantified– Address specifics of levels, trends, and

performance relative to comparisons


Stage 4: Post-Site VisitIMPROVEMENT PLANNING/JUDGING

4.02Council sends

post-site scheduleto TLs, Judges

<= DAY 62

4.03Feedback Writer completes

draft FR & e-mails TL(optional, if FW assigned)

4.04Team Leads editreport & e-mail to

Lead Judges

<= DAY 72

<= DAY 79

4.07Council sends final FRto TLs, Lead Judges, & Participant contact

(as appropriate)

<= DAY 81

4.08Council finalizes IPagenda/approach

w/Participant

4.09TLs/Council prepare

for IP Session

4.11Council sends thank youand satisfaction survey toParticipant (OCP, HRO)

4.01Council confirms

time/location for IPSession w/Participant& sends to full team

4.10TLs/Council participate

in IP Session: review FR,prioritize opptys, create

draft action plans(as appropriate)

4.05Lead Judge(s)work with TLsto create final

draft report

<= DAY 82

<= DAY 89

4.06Council makes

final edits

4.13TLs work withLead Judge(s)

to create Judges'Presentation; send to

Council

4.14Council sends agenda,Judges' presentation,

Org Profile, Cat 7,& FR to full POJ

4.15Council preparesscoring profiles &

loads PPT Judges'Presentation(s)

4.12Council confirmstime/location forJudging Session;

sends invite toEval. team(s)

4.18Panel of Judges

deliberates & reachesconsensus on level;if questions, contact

TLs by phone

4.17Lead Judges

present case to POJ& recommend

MQA level

4.16Team Leads

present to POJ(see agenda)

4.20Team Leads

inform team membersof decision (not forpublic consuption)

4.19Council communicatesrecognition level & key

messages to ParticipantHRO & Team Leads

NOTE: Improvement Planning sessions arescheduled individually with each applicant;

Judging Sessions are batched throughout theyear. The process above is typical (IP before

Judging), though on occasion Judgingprecedes the IP Session.

END OFJUDGING PROCESS

END OFASSESSMENT

PROCESS


Judges’ Presentation Agenda

• Key Factors – 2 minutes• Key Themes: Process Strengths – 3

minutes• Key Themes: Process OFIs – 3 minutes• Key Themes: Results – 6 minutes• Significant Differences (if any) – 1 minute• Commitment, Sustainability, and

Momentum – 5 minutes• Questions – 10 minutes


Judges’ Presentation - CommitmentThe extent to which leaders are willing to

invest time, resources and energy in making the journey.

An example of low commitment: leaders spend little time on quality or planning for improvements – few resources are made available for improving quality systems and approaches

An example of high commitment: leaders consistently encourage and lead efforts to improve the organization – leaders plan for high levels of performance – leaders allocate resources to the effort


Judges’ Presentation - SustainabilityThe extent to which the organization is likely

to continue making progress on its journey

An example of low sustainability: The quality journey may be seen as just a management fad – “something the boss wants us to do.” If the leader of the quality effort were to disappear from the scene, the journey would stop.

An example of high sustainability: “Managing quality is everyone’s job. We need to excel as an organization in order to survive and prosper. It’s not just the boss’ thing.” If there were a change in senior leadership, progress would continue.


Judges’ Presentation - Momentum

The direction and rate of progress on the quality journey.

An example of low momentum: There may be slow or no progress overall from year to year, although some pockets of rapid or steady growth may exist. The organization may be little aware of its opportunities for improvement or how to close the gaps. Perhaps, improvements in the past may have been lost and are no longer apparent.

An example of high momentum: Innovation and improvement in business practice is frequent and pervasive. Baldrige scores may have been rising each year through multiple cycle of self-assessment and improvements.


What Happens After the Site VisitFinal judging Applicant receives feedback reportImprovement Planning SessionCelebration!

Documents

Copyright 2002-2004 Minnesota Council for Quality1 The 2004 MQA Site Visit