Upload
chad-winfred-lawson
View
214
Download
0
Embed Size (px)
Citation preview
1
Results from the Performance Management Survey
By Janice Rouiller and Laura Stouffer, SAICand
Amy Culbertson, OQM*
1 April 2004
*Note: All data analyses were conducted by SAIC.
2
Table of Contents• Purpose and Methodology………………………………………………………….3
• Respondent Information…………………………………………………………….6
• Perceptions of Cycle 2 PM Implementation……………………………….…….14
• Perceptions of Needs for Cycle 3…………………………………………………47
• Organizational Climate Perceptions………………………………………………63
• Customer Scorecard Ratings………………………………………………………83
• Summary of Comments..…………………………………………………………..96
• Summary………………………………………………..…………………………..102
• Appendix - Individual Comments from Survey…………………………………..107
3
Purpose
• Administer a customized version of the ORS Customer Scorecard to:• Gather feedback on past Performance
Management (PM) implementation efforts• Collect needs assessment information to tailor the
on-going implementation of PM• Gather data on the required Customer Scorecard
dimensions• Collect organizational culture change information
for the Evaluation Study
4
Methodology• Administer the survey at the completion of the
second cycle of Performance Management implementation on an organization-wide basis
• Which occurred in January 2004
• Designed and tested web survey during February 2004
• Survey went “live” on 2 March 2004• Email requests sent to all PM Team Leaders, Members, and
additional senior ORS/ORF managers• First reminder email sent 8 March• Second reminder email sent 15 March• Closed survey field period 22 March
• Data file sent to contractor for analysis
5
Survey Distribution
Number of email recipients 196
Number of respondents 70
Response Rate 36%
6
Respondent Information
7
Respondents by Role
3
43
19
7
00
5
10
15
20
25
30
35
40
45
50
ORSDirector/Associate
Director
DivisionDirector/Office
Director
PM Team Leader PM Team Member Other
N =69
Note: Multiple responses allowed. One respondent skipped this question.
Nu
mb
er
of
Resp
on
den
ts
8
Respondents by Organization
52
16
10
5
10
15
20
25
30
35
40
45
50
55
60
ORS ORF Other
N = 69
Note: One respondent skipped this question.
Nu
mb
er
of
Resp
on
den
ts
9
Respondents by Program Area
2
10
13
12
5
1515
0
3
6
9
12
15
Program andEmployeeResources
ScientificResources
SecurityServices
ManagementServices
Real Estateand Facilities
Not Part ofthe Services
Hierarchy
Don't Know
N = 68
Note: Multiple responses allowed. Two respondents skipped this question.
Nu
mb
er
of
Resp
on
den
ts
10
Experience with the PM Process
42%
75%
73%
85%
76%
0% 20% 40% 60% 80% 100%
Part of ASA Team inFY01
Part of ASA Team inFY02
Received FY03Measures Workshop
Attended FY03 BSCTraining
Attended FY03Orientation Training
Percent “Yes”
N = 66
N = 65
N = 63
N = 65
N = 65
11
Approximately what percent of time did you spend on PM-related activities during Jan 03 - Jan 04?
3
0
21
0
3
9
20
12
19
0
2
4
6
8
10
12
14
16
18
20
0-10% 11-21% 21-30% 31-40% 41-50% 51-60% 61-70% 71-80% 81-90% 91-100%
N = 69
Percent of time Spent
Nu
mb
er
of
Resp
on
den
ts
12
On average, how often did your PM team meet during the time period Jan 03 - Jan 04?
21
25
28
12
1
0
5
10
15
20
25
30
Several Timesa Week
Once a Week Biweekly Once a Month Once a Quarter Less thanOnce a Quarter
N = 69
How Often PM Team Met
Nu
mb
er
of
Resp
on
den
ts
13
0
4
5
6
13
6
8
12
4
9
2
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent did your team members work between meetings to implement PM?
To No Extent
To a Great Extent
N = 69
Mean = 6.17
Median = 6
14
Perceptions of Cycle 2 PM Implementation(January 2003 - January 2004)
15
How helpful did you find the following in implementing PM?
4.17
5.03
5.26
5.39
5.46
5.54
5.82
5.93
5.97
6
6.28
6.34
6.88
6.91
6.94
7.07
8.13
1 2 3 4 5 6 7 8 9 10
Other PM Teams
PM Conference
Performance Management Website
Services Hierarchy
Design of Data Collection Forms
Process Behavior Chart Analysis
Measures Roadmap Template
PM Related Emails from OQM
Establishing Data Collection
ORS Customer Scorecards
Strategy Mapping
Graphing Data
Your Supervisor
PMPTemplate
PM Presentation Template
OQM Staff
PM Consultants
Not at all Helpful
Extremely Helpful
Mean Ratings
16
How helpful did you find the following tools in implementing PM?
6.94
5.82
6.28
6.91
5.39
1 2 3 4 5 6 7 8 9 10
PM PresentationTemplate forConference
Measures RoadmapTemplate
Strategy Mapping
PMP Template
Listings of theServices Hierarchy N =
68
Not at all Helpful
Extremely Helpful
Mean Ratings
N = 67
N = 69
N = 67
N = 68
17
8
4
5
3
6
5
9 9
4
6
7
2
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following tools in implementing PM?
Listings of the Services Hierarchy
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 5.39
Median = 6
18
1
2 2 2
7 7
8
12
11
13
2
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following tools in implementing PM?
Performance Management Plan (PMP) Template
Not at all Helpful
Extremely
Helpful
N = 67
Mean = 6.91
Median = 8
19
4
3
4
5
8
7
5
13
10
8
2
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following tools in implementing PM?
Strategy Mapping
Not at all Helpful
Extremely
Helpful
N = 69
Mean = 6.28
Median = 7
20
4
3
4 4
9
3
7
12
5
7
9
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following tools in implementing PM?
Measures Roadmap Template
Not at all Helpful
Extremely
Helpful
N = 67
Mean = 5.82
Median = 7
21
2
1
3 3
4
6
7
14
8
13
7
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following tools in implementing PM?
PM Presentation Template for Conference
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 6.94
Median = 8
22
How helpful did you find the following data analysis services in implementing PM?
6
5.54
6.34
5.97
5.46
1 2 3 4 5 6 7 8 9 10
ORS CustomerScorecards
Process BehaviorChart Analysis
Graphing Data
Establishing DataCollection
Methodologies
Design of DataCollection Forms
Not at all Helpful
Extremely Helpful
Mean Ratings
N = 67
N = 68
N = 68
N = 68
N = 67
23
5
7 7
0
4
5
9
12
5 5
3
5
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following data analysis services in implementing PM:
Design of Data Collection Forms
Not at all Helpful
Extremely
Helpful
N = 67
Mean = 5.46
Median = 6
24
3
8
4
0
5 5
11
15
5
6
3 3
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following data analysis services in implementing PM:
Establishing Data Collection Methodologies
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 5.97
Median = 7
25
4 4
2
4 4
1
14
16
4
9
4
2
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following data analysis services in implementing PM:
Graphing Data
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 6.34
Median = 7
26
5 5
3 3
5
4
9
8
3
6
10
7
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following data analysis services in implementing PM:
Process Behavior Chart Analysis
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 5.54
Median = 6
27
5 5 5
1
2
7
5
10
9 9
5
4
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following data analysis services in implementing PM:
ORS Customer Scorecards
Not at all Helpful
Extremely
Helpful
N = 67
Mean = 6.00
Median = 7
28
How helpful did you find the following communication vehicles in implementing PM?
5.03
5.93
5.26
1 2 3 4 5 6 7 8 9 10
PM Conference
PM Related Emailsfrom OQM
PM Website N = 68
Not at all Helpful
Extremely HelpfulMean Ratings
N = 68
N = 68
29
6
4
3
8
9
5
9
8
1
6
9
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following communication vehicles in implementing PM:
Performance Management Website
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 5.26
Median = 5
30
3 3 3
6
9 9 9
13
5 5
3
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following communication vehicles in implementing PM:
PM Related Emails from OQM
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 5.93
Median = 6
31
7
3
4
5
11
8
7 7
1
4
7
4
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find the following communication vehicles in implementing PM:
PM Conference
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 5.03
Median = 5
32
How helpful did you find assistance from the following groups in implementing PM?
6.88
4.17
8.13
7.07
1 2 3 4 5 6 7 8 9 10
Your Supervisor
Other PM Teams
PM Consultants
OQM Staff N = 69
Not at all Helpful
Extremely Helpful
Mean Ratings
N = 69
N = 68
N = 68
33
0
1
3 3
8
7
8
15
13
9
1 1
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find assistance from the following groups in implementing PM?
OQM Staff
Not at all Helpful
Extremely
Helpful
N = 69
Mean = 7.07
Median = 8
34
01 1 1 1
5
1011 11
26
0
2
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find assistance from the following groups in implementing PM?
PM Consultants
Not at all Helpful
Extremely
Helpful
N = 69
Mean = 8.13
Median = 9
35
9
6
3
7
2
45 5
12
6
18
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find assistance from the following groups in implementing PM?
Other PM Teams
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 4.17
Median = 4
36
3
1
4
1
7
3
6
14
8
14
1
6
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
How helpful did you find assistance from the following groups in implementing PM?
Your Supervisor
Not at all Helpful
Extremely
Helpful
N = 68
Mean = 6.88
Median = 8
37
Associate Director Involvement in PM
7.48
5.52
1 2 3 4 5 6 7 8 9 10
What extent wasyour Associate
Director involved inyour PM processduring this PM
Cycle?
To no Extent
To a Great Extent
7.29
5.67
1 2 3 4 5 6 7 8 9 10
How helpful wasthe involvement of
your AssociateDirector in PMrelated matters
during this cycle?
FY02 FY03Not at all Helpful
Extremely Helpful
Mean Ratings
N = 69
N = 69
N = 62
N = 63
38
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
4.04
5.27
5.32
6.24
6.02
6.22
6.16
6.37
1 2 3 4 5 6 7 8 9 10
Using Visio
Using PowerPoint
Using Excel
Working in a Team
Presentation, Preparation and Delivery
Methods to Analyze Data
Methods to Gather Data
BSC approach to PM N = 69
To No Extent
To a Great Extent
Mean Ratings
N = 69
N = 69
N = 69
N = 69
N = 69
N = 69
N = 68
39
3
1
4
2
8
9
11
12
8
5
4
2
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
Balanced Scorecard Approach to PM
To No Extent
To a Great Extent
N = 69
Mean = 6.37
Median = 7
40
3 3
4
1
8
13
12 12
4
6
2
1
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
Methods to Gather Data
To No Extent
To a Great Extent
N = 69
Mean = 6.16
Median = 7
41
2 2
5
4
6
12
13
12
6
5
2
1
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
Methods to Analyze Data
To No Extent
To a Great Extent
N = 69
Mean = 6.22
Median = 7
42
5
2
4
2
5
11
13
15
4
3 3
2
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
Presentation Preparation and Delivery
To No Extent
To a Great Extent
N = 69
Mean = 6.02
Median = 7
43
5
3
1 1
11
7 7
16
10
3 32
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
Working in a Team
To No Extent
To a Great Extent
N = 69
Mean = 6.24
Median = 7
44
10
4
3
5
8
6
5
6
11
3
2
6
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
Using Excel
To No Extent
To a Great Extent
N = 69
Mean = 5.32
Median = 5
45
13
1 1
5
7
9
7
5
9
3
2
7
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
Using PowerPoint
To No Extent
To a Great Extent
N = 69
Mean = 5.27
Median = 6
46
14
45
32
3 3
9
21
5
17
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?
Using Visio
To No Extent
To a Great Extent
N = 68
Mean = 4.04
Median = 3
47
Perceptions of Needs for Cycle 3
48
In planning for the next PM cycle, how helpful would each of the following be to your PM Team?
5.4
5.61
5.63
6.02
6.05
6.08
6.18
6.27
6.75
7.35
7.4
8.31
1 2 3 4 5 6 7 8 9 10
Yearly Performance Mgmt Conf
Rotating Quarterly Presentation Meetings
Training on the BSC
Training on data collection systems and forms
Training on analyzing data through graphs
Training on ORS ABC Model
Training on process behavior charts
Training on customer assessment
Regular Meetings with Mgmt to discuss PM data
IT Support to establish Data Collection systems
OQM Staff
PM Consultants provided by OQM
Not at all Helpful
Extremely Helpful
Mean Ratings
49
In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team?
7.35
5.4
5.61
6.75
7.4
8.31
1 2 3 4 5 6 7 8 9 10
IT Support to Establish Data CollectionSystems
Yearly PM Conference
Rotating Quarterly Presentation Meetings
Regular Meetings with your Mgmt to discussPM Data
OQM Staff
PM Consultants provided by OQM N = 69
Not at all Helpful
Extremely Helpful
Mean Ratings
N = 69
N = 69
N = 68
N = 68
N = 67
50
0 0
21
4
23
9
18
26
2 2
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team:
PM Consultants Provided by OQM
Not at all Helpful
Extremely Helpful
N = 69
Mean = 8.31
Median = 9
51
0
2
1
2
5
8
7
14
13 13
3
1
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team:
OQM Staff
Not at all Helpful
Extremely Helpful
N = 69
Mean = 7.40
Median = 8
52
4
1
4
2
3
5
13 13
7
11
3 3
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team:
Regular Meetings with your Management
Not at all Helpful
Extremely Helpful
N = 69
Mean = 6.75
Median = 7
53
8
1
8
3
1
6
3
9
6
8
13
2
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team:
Rotating Quarterly Presentation Meetings
Not at all Helpful
Extremely Helpful
N = 68
Mean = 5.61
Median = 6
54
10
5
4
2
6
5
4
11
6
7 7
1
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team:
Yearly Performance Management Conference
Not at all Helpful
Extremely Helpful
N = 68
Mean = 5.40
Median = 6
55
1 12
32
4
98
9
20
7
1
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team:
IT Support to Establish Data Collection Systems
Not at all Helpful
Extremely Helpful
N = 67
Mean = 7.35
Median = 8
56
In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?
6.08
6.27
6.18
6.05
6.02
5.63
1 2 3 4 5 6 7 8 9 10
ORS ABS Cost Accounting Model
Customer Assessment (interviews, focusgroups, surveys)
Analyzing Processes with Process BehaviorCharts
Analyzing Data through Graphs
Setting up Data Collection Systems andForms
Balanced Scorecard Methodology N = 67
Not at all Helpful
Extremely HelpfulMean Ratings
N = 68
N = 68
N = 68
N = 67
N = 67
57
7
2
5
4
8
6
5
11
5
8
3 3
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?
Balanced Scorecard Methodology
Not at all Helpful
Extremely Helpful
N = 67
Mean = 5.63
Median = 6
58
5
4
3 3
6
8 8
12
7 7
3
2
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?
Setting up Data Collection Systems and Forms
Not at all Helpful
Extremely Helpful
N = 68
Mean = 6.02
Median = 7
59
7
0
4 4
7
5
8
15
7
6
3
2
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?
Analyzing Data Through Graphs
Not at all Helpful
Extremely Helpful
N = 68
Mean = 6.05
Median = 7
60
6
1
4
3
5
7
8
15
7 7
3
2
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?
Analyzing Processes with Process Behavior Charts
Not at all Helpful
Extremely Helpful
N = 68
Mean = 6.18
Median = 7
61
5
2
3 3
5
6
9
10 10
9
3
2
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?
Customer Assessment through Interviews, Focus Groups and Surveys
Not at all Helpful
Extremely Helpful
N = 67
Mean = 6.27
Median = 7
62
6
1
5
3
6
5
4
8
10 10
5
4
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?
ORS ABC Cost Accounting Model
Not at all Helpful
Extremely Helpful
N = 67
Mean = 6.08
Median = 7
63
Organizational Climate Perceptions
64
4.04
4.01
3.46
4.13
3.83
3.73
4.21
3.72
1 2 3 4 5
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Organizational Climate PerceptionsSummary
Strongly Disagree
Strongly Agree
N = 68
N = 68
N = 68
N = 68
N = 68
N = 68
N = 67
N = 68
Neither Agree nor Disagree
65
4
89
19
27
10
0
5
10
15
20
25
30
1 2 3 4 5 DK NA
Fre
qu
ency
of
Res
po
nse
I see the value of performance management for improving business operations in my area.
Strongly Disagree
Strongly Agree
N = 68
Mean = 3.72
Median = 4
Neither Agree nor Disagree
66
21
8
19
33
5
00
5
10
15
20
25
30
35
1 2 3 4 5 DK NA
Fre
qu
ency
of
Res
po
nse
My organization (ORS or ORF) is committed to the performance management effort.
Strongly Disagree
Strongly Agree
N = 68
Mean = 4.21
Median = 5
Neither Agree nor Disagree
67
8
3
11
16
26
4
00
5
10
15
20
25
30
1 2 3 4 5 DK NA
Fre
qu
ency
of
Res
po
nse
I understand what the Services Hierarchy is and its purpose.
Strongly Disagree
Strongly Agree
N = 68
Mean = 3.73
Median = 4
Neither Agree nor Disagree
68
1
6
14
2221
4
00
5
10
15
20
25
1 2 3 4 5 DK NA
Fre
qu
ency
of
Res
po
nse
The culture of my organization (ORS or ORF) is changing to be more results-oriented.
Strongly Disagree
Strongly Agree
N = 68
Mean = 3.83
Median = 4
Neither Agree nor Disagree
69
12
12
20
26
7
00
5
10
15
20
25
30
1 2 3 4 5 DK NA
Fre
qu
ency
of
Res
po
nse
Managers in my organization (ORS or ORF) believe accountability is an important organizational value.
Strongly Disagree
Strongly Agree
N = 68
Mean = 4.13
Median = 4
Neither Agree nor Disagree
70
6 6
15
23
17
10
0
5
10
15
20
25
1 2 3 4 5 DK NA
Fre
qu
ency
of
Res
po
nse
Performance management has contributed to improvements in my area.
Strongly Disagree
Strongly Agree
N = 68
Mean = 3.46
Median = 4
Neither Agree nor Disagree
71
4 4
10
17
21
7
4
0
5
10
15
20
25
1 2 3 4 5 DK NA
Fre
qu
ency
of
Res
po
nse
Performance management will assist my organization if we have to go through an A-76 competition.
Strongly Disagree
Strongly Agree
N = 67
Mean = 4.01
Median = 4
Neither Agree nor Disagree
72
0
5
10
19
33
10
0
5
10
15
20
25
30
35
1 2 3 4 5 DK NA
Fre
qu
ency
of
Res
po
nse
My PM Team (or the groups I am responsible for) are actively involved in data collection and analysis.
Strongly Disagree
Strongly Agree
N = 68
Mean = 4.04
Median = 4
Neither Agree nor Disagree
73
Organizational Climate Perceptions by Program Area
74
Program and Employee Services Climate Perceptions
69%
81%
69%
69%
63%
69%
56%
69%
0% 20% 40% 60% 80% 100%
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Percent Agree or Strongly Agree
N = 16
N = 16
N = 16
N = 16
N = 16
N = 16
N = 16
N = 16
75
Scientific Resources Climate PerceptionsPositive Responses
40%
67%
40%
53%
67%
53%
36%
73%
0% 20% 40% 60% 80% 100%
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Percent Agree or Strongly Agree
N = 15
N = 14
N = 15
N = 15
N = 15
N = 15
N = 15
N = 15
76
Security Services Climate Perceptions
40%
60%
40%
60%
80%
40%
20%
80%
0% 20% 40% 60% 80% 100%
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Percent Agree or Strongly Agree
N = 5
N = 5
N = 5
N = 5
N = 5
N = 5
N = 5
N = 5
77
Management Services Climate Perceptions
82%
73%
82%
64%
82%
55%
64%
91%
0% 20% 40% 60% 80% 100%
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Percent Agree or Strongly Agree
N = 11
N = 11
N = 11
N = 11
N = 11
N = 11
N = 11
N = 11
78
Real Estate and Facilities Climate Perceptions
92%
83%
67%
83%
67%
67%
67%
83%
0% 20% 40% 60% 80% 100%
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Percent Agree or Strongly Agree
N = 12
N = 12
N = 12
N = 12
N = 12
N = 12
N = 12
N = 12
79
Organizational Climate Perceptions by Role
80
Division Directors’/Office Directors’ Climate Perceptions
100%
100%
100%
100%
86%
100%
57%
86%
0% 20% 40% 60% 80% 100%
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Percent Agree or Strongly Agree
N = 7
N = 7
N = 7
N = 7
N = 7
N = 7
N = 7
N = 7
81
PM Team Leaders’ Climate Perceptions
83%
78%
83%
89%
67%
78%
55%
89%
0% 20% 40% 60% 80% 100%
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Percent Agree or Strongly Agree
N = 18
N = 18
N = 18
N = 18
N = 18
N = 18
N = 18
N = 18
82
PM Team Members’ Perceptions
56%
72%
47%
49%
63%
49%
53%
74%
0% 20% 40% 60% 80% 100%
Actively Involved in Data Collection
PM Assist with A-76
PM Contributed to Improvements
Accountability is an Important Value
Results-Oriented Culture
I Understand Services Hierarchy
Committed to PM
See Value of PM
Percent Agree or Strongly Agree
N = 43
N = 42
N = 43
N = 43
N = 43
N = 43
N = 43
N = 43
83
Customer Scorecard Ratings
84
Customer Scorecard Ratings for PMOverview for Cycle 2
6.33
7.06
6.78
7.14
7.03
6.83
6.58
6.82
6.12
1 2 3 4 5 6 7 8 9 10
Handling of Problems
Competence
Convenience
Responsiveness
Availability
Reliability
Timeliness
Quality
Cost N = 68
Unsatisfactory Outstanding
Mean Ratings
N = 68
N = 67
N = 68
N = 68
N = 68
N = 68
N = 68
N = 65
85
0 1 2 2
7
13
64 4
36
2
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Cost Satisfaction Ratings
Unsatisfactory Outstanding
N = 68
Mean = 6.12
Median = 7
86
0
2 23
8
3
8
17
9
7
9
00
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Quality Satisfaction Ratings
Unsatisfactory
Outstanding
N = 68
Mean = 6.82
Median = 8
87
0
3
2
5
7
2
5
14
11
7
11
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Timeliness Satisfaction Ratings
Unsatisfactory
Outstanding
N = 67
Mean = 6.58
Median = 8
88
1 1 1
4
7
4
6
15
10
7
12
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Reliability Satisfaction Ratings
Unsatisfactory
Outstanding
N = 68
Mean = 6.83
Median = 8
89
0
2
3 3
6
3
11
15
9
12
4
00
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Availability Satisfaction Ratings
Unsatisfactory Outstanding
N = 68
Mean = 7.03
Median = 8
90
01
23
54
11
19
7
11
5
00
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Responsiveness Satisfaction Ratings
Unsatisfactory
Outstanding
N = 68
Mean = 7.14
Median = 8
91
0
2
0
8 8
3
7
16
7
10
7
00
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Convenience Satisfaction Ratings
Unsatisfactory Outstanding
N = 68
Mean = 6.78
Median = 8
92
1 1
5
2
7
2
6
17
8
15
4
00
5
10
15
20
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Competence Satisfaction Ratings
Unsatisfactory
Outstanding
N = 68
Mean = 7.06
Median = 8
93
2
3
1
3
7
0
9
13
9
8
9
1
0
3
6
9
12
15
1 2 3 4 5 6 7 8 9 10 DK NA
Fre
qu
ency
of
Res
po
nse
Handling of Problems Satisfaction Ratings
Unsatisfactory Outstanding
N = 65
Mean = 6.33
Median = 7
94
Customer Scorecard Ratings for PMTrend Analysis
7.03
7.06
7.36
6.57
6.83
6.58
6.82
6.12
1 2 3 4 5 6 7 8 9 10
Reliability
Timeliness
Quality
Cost
Cycle 1 Cycle 2
N = 68
Unsatisfactory Outstanding
Mean Ratings
N = 68
N = 67
N = 68
N = 37
N = 70
N = 67
N = 67
95
Customer Scorecard Ratings for PMTrend Analysis (cont.)
6.33
7.33
7.5
7.5
7.17
6.33
7.06
6.78
7.14
7.03
1 2 3 4 5 6 7 8 9 10
Handling of Problems
Competence
Convenience
Responsiveness
Availability
Cycle 1 Cycle 2
N = 68
Unsatisfactory Outstanding
Mean Ratings
N = 68
N = 68
N = 68
N = 65
N = 69
N = 68
N = 66
N = 69
N = 65
96
Summary of Comments
Note: 34 out of 70 respondents made comments (49%).
97
What did you value most about this cycle of performance management?
• Assistance provided by consultants and OQM staff (9)• Professional guidance and support from OQM staff.• Knowledgeable and helpful consultants.• Increased involvement of OQM staff in team meetings.
• Implementing measurement systems (6)• Developing systems to gather data.• Using data to improve processes.
• Knowledge and Experience Gained (4)• Experience has led to fuller appreciation of the Performance
Management Process.• Continued experience is leading to greater comfort with the
process.
Note: Comments were sorted into categories. Most frequently mentioned categories listed first.
98
What did you value most about this cycle of performance management? (cont.)
Note: Comments were sorted into categories. Most frequently mentioned categories listed first.
• Understanding our processes (4)• Greater understanding of the processes.• Actually seeing improvements in processes.
• Training Offered (2)• Measures Workshop and Balanced Scorecard Orientation.• Just in Time training.
99
What obstacles were most challenging to you during this cycle of performance management?
Note: Comments were sorted into categories. Most frequently mentioned categories listed first.
• Lack of time/resources (10)• Office is short staffed and workloads at all-time high• Difficulty scheduling OQM staff for timely meetings• Office identified late and struggled to “catch up” with process
• Data Collection and Measurement (9)• Manual data collection• Definition of measures difficult• Analysis of measures difficult
• Applying Balanced Scorecard Approach (6)• Lack of flexibility in application• Use of resulting data to make changes is absent
100
What obstacles were most challenging to you during this cycle of performance management? (cont.)
Note: Comments were sorted into categories. Most frequently mentioned categories listed first.
• Lack of management involvement (3)• PMP not reviewed• Close communication missing
• Conference (2)• Lack of management presence• Full presentation too time-consuming – focus should be on presentation
of a few key indicators
101
How can we help integrate PM into your daily business activities?
• Foster use of data. (7)• Show how data is being used by Service Groups and by
ORS/ORF Managers.• Follow up with Service Groups on implementation of objectives.
• Continue support currently provided. (6)• Maintain communication throughout the year.• PMP is very helpful in its present form.
• Increase management involvement in process. (2)
102
SummarySurvey Respondents
• Respondents were primarily from ORS (3/4th)• About half of the PM team leaders answered the survey • Smallest response from the security program area
• Respondents were experienced with PM (3/4th) and had received training (3/4th)
• About 74% of respondents spent 30% or less of their time during the past year on PM
• Team meetings typically occurred biweekly (40%) or once a month (36%)
103
Summary (cont.)
Feedback on Past Cycle
• The five most helpful tools/resources were the Consultants, OQM Staff, PM Presentation Template, PMP template, their supervisors
• Respondents reported just above mid-scale improvements to their KSAs for most topics listed
• Respondents valued assistance provided, implementing measurement systems, knowledge and experience gained, understanding of processes
• Obstacles included lack of time/resources, issues with data collection/measurement, applying the BSC approach
104
Summary (cont.)
Planning for Next PM Cycle
• Most helpful to teams for the future would be PM consultants, OQM staff, IT Support for data collection systems
• To foster the integration of PM into daily business operations, respondents said we need to help foster the use of data, continue support provided, increase management involvement in process
105
Summary (cont.)
Organizational Climate
• Majority of respondents either agree or strongly agree that:
• Their organization is committed to PM (76%)• They see the value of PM for improving business operations
(68%)• Managers in their organization believe accountability is
important (68%)• The culture of their organization is changing to be more
results-oriented (63%)
• Senior managers more strongly endorse these items compared to team leaders and team members
106
Summary (cont.)
Customer Scorecard Trend Analysis
• Ratings on all dimensions are lower in Cycle 2 compared to Cycle 1
• Current ratings:• Highest on responsiveness, competence, availability• Lowest on cost, handling of problems, timeliness
107
Appendix
Individual Comments From Survey
108
What did you value most about this cycle of performance management? Individual Comments
• Really customizing the strategic plan to our particular area . Implementing a system to gather data to understand our service delivery.
• Measuring customer service performance is valuable, however, I am not sure much was gained by this exercise over the past several years other than taking people away from assigned duties.
• Understanding the process.• Nothing.• News I can use!• Results from data collection - mostly trend reporting.• Nothing - I waste of time and tax payers money. • Amy is a wealth of knowledge and always eager to assist when
needed.• The increased involvement of the OQD staff in the team meetings.
109
What did you value most about this cycle of performance management? (cont.) Individual Comments
• Our OQM representative and PM consultant listened to our concerns about this method of measuring our productivity. They convinced us that this type of measurement and representation could represent our points of view and culture better than we imagined.
• Continuity from prior year. Beginning to provide a context, esp. in using strategy mapping.
• Our consultant was great. This is the first year I understood what we were doing.
• It spurred us to initiate and develop a new call center that we know use to keep track of our work and help keep more statistics on the work we do. The concept of coordinated, consistent performance mgt techniques and metrics is a good one.
• I support the formalized training for ABC, Balance Scorecard etc. However, it was extremely beneficial to have just in time training provided by Pat Patterson as we progressed through our PMP process.
110
What did you value most about this cycle of performance management? (cont.) Individual Comments
• Learning about data collection and how to use that data to develop improvement plans.
• OQM supported a program review and a root cause analysis as part of this years PMP that was extremely useful. Really appreciate the expertise and responsiveness of both OQM and contract staff. Our contract staff is a real standout.
• The Measures Workshop and Balanced Scorecard Orientation.• Our consultant was extremely helpful - Pat Patterson. Carmen, Amy and
Joe were also extremely helpful even though they were not our consultant. They answered our questions before we got to Josh (who was not our OQM consultant) but he was helpful.
• I gained experience in and fuller appreciation for the importance of collecting and analyzing data about our Division's efficiency and effectiveness. The PMP has provided insights that are guiding us in improving both of these areas. The data we collected and analyzed had use beyond informing us about our own operation--when SDs external to our program recently requested detailed information about our operations, we were able to respond quickly and completely.
111
What did you value most about this cycle of performance management? (cont.) Individual Comments
• It's become so familiar that I spend more time on the data versus the process.
• OQM and the consultant's support throughout the process.• Experience under my belt in all areas this cycle.• I valued the professional guidance provided by OQM. Ms. Amy
instilled the knowledge to our group on why we perform this task which gave more definition and purpose for doing it. She was outstanding and I would like to commend her for a job well done. From depositing her guidance and understanding into our group, I have learned the reasons for the PMP which will reveal to the manager how to effectively administer the business.
• Having a chance to make suggestions as to what can be improved in the process. Actually seeing some improvements in the different layers of the process - this is invaluable!
112
What obstacles were most challenging to you during this cycle of performance management?Individual Comments
• The OQM staff member for my team, was not committed, only attended less then half of the mtgs.
• Too many man hours are required for the PM Initiative during a time when FTEs are reduced and workloads are at an all-time high Question the cost effectiveness of the PM Initiative. Do we have any data the supports the cost (i.e., man-hours, OQM salary, consultants) of the PM Initiative?
• THE BSC is not being implemented right, nor are the results being used for anything. I think the current system is too prescriptive, it's not flexible enough for organizations to incorporate their own modifications or metrics. It has to be balanced scorecard, which is ok, but the not only way.
• The Associate Director never reviewed our template though given to him in the beginning months. Management in our office is in process of changing and I am apprehensive to new management.
113
What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments
• A culture that doesn't support accountability Resistance to change by people in the organization.
• Trying to identify our division and its measurable values.• I do not think we need the conference at the end. It was a farce to
have the ORSAC list who they wanted to present and then not one of them show up. It makes no sense and just takes up valuable time. I think that the Management Council should be consulted on what to do this year. Let them vote on what should be done, instead of us being told what to do, so there is buy in.
• The lack of time to fully define, measure, and implement results.• Defining and measuring unit cost.• Too much time spend on this project by team members.• Having to manually collect data.• Analyzing the data through graphs and making trend projections.
114
What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments
• Move from Discrete Service to Service Area review continues to prove problematic in applying Internal Business perspective to improve business performance.
• Trying to define meaningful measurements of productivity and progress within this model of activity.
• financial perspective. Hard to relate to effectiveness. Is a measure for the sake of measuring only. insistence on full up presentation preparation each year. Why not some key indicators only except for a select few chosen each year on rotating basis.
• Director of our organization did not find time to meet with our team even once, although we had a number of issues to discuss. Success of our service group is greatly dependent on close communication with upper management in ORF and with other NIH stakeholders. This is missing. We can only correct what we control; our suggestions for changing what we don't control are always ignored.
115
What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments
• I think the ASA/PMP is pushed down to too low a level. My five person office really shouldn't be doing it's own ASA/PMP. It's a time consuming dis-economy of scale. Our parent office should incorporate it into their ASA/PMP, we are just too small in numbers and dollars to get any significant value out of the time we spend on the ASA/PMP. We collected stats and performance metrics on our work even before the ASA/PMP, because that's what good mgt does, which was sufficient enough for us to track and improve our service. But it was less intensive than the current PMP.
• The lingo. If you do this on a daily basis it's easy to remember. The entire process of balanced scorecard, service hierarchy, etc. is hard to remember if you do not do it every day.
• Getting information to resolve issues.• OQM Staff member hardly ever attended any of our meetings.
Miscommunication regarding what was needed.
116
What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments
• Due to multiple layers of management involvement we did some re-tracking over materials that produced some levels of frustration. Some tracking forms proved less than useful.
• Our Team was not initially convened when others in our Division started; so we had to play catch-up. The service for which I was designated Team Leader passed to the MEO, so it was determined late in the cycle that we would not present.
• Time -our office was short staffed (and still is). Being able to get to have any quality focus was difficult -- This is where our consultant was extremely helpful. Because of the systems we used for counting (e.g. ADB) we had to do a lot of manual counting.
• Gathering data for entire discreet service.• After three years of ASA/PMP, I feel that our division has
addressed everything that was/could be of concern. I see PMP as now being in a maintenance mode for us since our process is fairly straightforward.
117
What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments
• Team member participation.• Last minute changes.• Reading the graphs.• Data collection - I waited too late before collecting it. Obtaining the
correct measures and objectives.• Trying to make the experience as interesting as possible since the
process could be boring at times.• Internal disagreements in OQM staff led to confusion on how
proceed with the process. Delay in proceeding forward due to schedule in-availability of OQM staff to meet.
118
How can we help integrate PM into your daily business activities? Individual Comments
• Managers realizing the importance of data for decision making.• Make the reasons of why we are doing this more concrete - more
specific examples of how it has helped others.• Understand what our division does and applying the PMP to the
function.• There must be something else that can be done instead of the
conference. We should be able to learn from each other on how to collect data.
• I have used the provided control charts to forecast and project overtime needs.
• Have upper management do this project.• Follow up on documentation of implementation of objectives, i.e.,
training records and frequencies of meetings/actions.• Insist on development of standard and specialized business tools and
services to support the service activity.
119
How can we help integrate PM into your daily business activities? (cont.) Individual Comments
• As far as I can tell the ASA/PMP is a waste of time, nothing is ever done with any of the information collected. The data is collected and analyzed and we look at it and say "oh that's nice" or "that's interesting", but no changes or new initiatives have ever started because of the data, not our level or at the ORS/ORF level. If there have been changes at any level I would be interested in knowing about them. Particularly at the ORS/ORF level.
• I already do by collecting data and putting it into spreadsheets almost monthly to track trends. I did not ever do this until the PMP program so it has helped me significantly. The data really does help in showing the value of my programs and where there are weaknesses and strengths!
• By being focused on true value added.• Maintain communication throughout the year.
120
How can we help integrate PM into your daily business activities? (cont.) Individual Comments
• The PMP is very helpful in its present form. Like virtually any good thing, however, it has the potential to grow to a point of diminishing returns, with gradual increases in requirements to collect, analyze and present data detracting from a Division's
• Unfortunately, the only way of assuring it gets tied in is to tie it to performance reviews.
• You already have.• Continue support you are currently providing.• Data review and collection.• You have helped.• We are integrated and know the cost of the activities of our service
group.
121
OtherIndividual Comments
• This is too long and I don't have time to fill it all out.