Structuring Performance Data to Effectively Drive Management Decisions Tim Gribben, Josh Bull, & Mat Pascarella O FFICE OF P ERFORMANCE M ANAGEMENT U.S

Structuring Performance Data to Effectively Drive Management Decisions

Tim Gribben, Josh Bull, & Mat Pascarella

OFFICE OF PERFORMANCE MANAGEMENTU.S. SMALL BUSINESS ADMINISTRATIONDecember 5, 2012

1

Embedding Performance Management

Cascading Data-Driven Decision-Making

Framework of SBA’s Training Course

Permeation & Permutation of Training

2





3

Performance Management as Organizational Culture

• Senior Leadership must use data-driven quarterly performance reviews of CAP and Priority Goals

(GPRA Modernization Act of 2010)• Many agencies run this exercise

out of their respective Performance Management Offices

• However, for performance management to effect dramatic change, the more difficult task will be to integrate the same data-driven decision-making throughout the agency, especially among program managers and mid-level analysts.

Senior Leadersh

ip

Program

Managers

Mid-level Analysts

4

Recognizing a Problem & Finding the Solution

IDENTIFIED GAP

SBA not effectively

analyzing the wealth of data available nor using data

effectively to drive

management decisions

LEADERSHIP BUY-IN

Office heads select interested

managers or analysts with basic experience in data analysis and willing to commit 3-4 hours per week to training

PMO RUNS TRAINING

PMO develops six week training

course to teach the skills via

lecture, activities, and homework as

well as “real-world” final presentation

5

‘Lesson Learned’ continually improves course content & format

Final presentations raised by course participants turn into agency-wide projects

Future presentations to mgmt incorporate class teachings

Making The Training “Stick”

Goals

Expectations

MANAGERS INSTITUTIONALIZATION

Getting buy-in from managers for maximum impact on participants, from

executive leadership is key for institutionalization

Invest in personal & professional development of employees

Surface actionable, interesting analytic questions for participants to explore Required to meet with their participating employees to discuss training goals and expectations

Asked to check in on participants to proffer help or feedback during training

Including phrase into SBA’s new strategic plan –

“Investing in SBA’s employees through business, technical, and analytical training”

6

Summary of SBA’s Path

Identified skills gap related to data analysis / analytical presentations Relatively few used data effectively to inform management decisions Reliance on data summarization without insights

WHERE WE WERE IN 2009

Created a training course to build new skills Included it as an objective & strategy in the new strategic plan Senior leaders and managers were also trained in the process

WHAT WE DID

Data is being presented more effectively to inform management decisions

Senior leaders are using data to address management challenges

WHERE WE ARE NOW

7





8

Structure the Presentation

Communicate the analysis

Synthesize & Develop

Recommendations

Session

KeyLessons

Structuring a management presentation

Developing a story-line

Presenting and communicating with data

Creating visual charts and graphs

Synthesizing findings into management recommendations

Developing data-driven ‘so-what’ insights

PURPOSE OF TRAINING: To help participants develop and structure analytic presentations that inform SBA

management decisions

Training Purpose & Format9

WHAT THE TRAINING IS

WHAT THE TRAINING ISN’T Training to help develop

presentations that inform management decisions

Structuring

Communicating

Data Synthesis

Way to EMPOWER employee to drive SBA business decisions

Χ Excel 101

Χ Powerpoint 101

Χ How to Report Data

Χ Oral Presentation Coaching

Χ Understanding SBA’s Programs

Risk-free environment Χ Formal Assessment of . Employee’s Abilities

Interactive with assignments

Group oriented

Χ US lecturing THEM

Χ Rigid Curriculum

Data Analysis Training Ground Rules

10



Synthesize and Develop

Recommendations

Two step process – “Structure the Presentation”:• Structure the management question (using an

Issue True) Organize the issues into buckets Anticipate the data and analyses that will

answer the questions

• Build a story line (using a “Ghost Deck”) Plan the work to be done Align with managers on direction before

investing the time

1

2

SESSION #1: Structure the Presentation

11

While developing the ‘Issue Tree’ to structure issues, we might consider two

principles to answer ALL and the most RELEVANT questions

1

…and keep digging deeper (ask “So-

What?” or “Why?”) to ensure the issues are

relevant

Example: Why is my credit card bill

increasing?

Be “Mutually Exclusive,

Collectively Exhaustive” (MECE) to identify all issues…

Example: How can I save money for a

vacation?

Structure the Presentation12

How can I save money

for a vacation?

Spend LESS money

Earn MORE money

Questions and analyses to consider

Can I earn more interest on existing income?Can I invest in stocks/funds instead of checking account

Can I earn new income?Can I sell garage items by having a yard sale

Can I buy cheaper items?Coffee machine, not Starbucks; Shop at discount stores

Can I buy fewer items? Spend less on gas by driving at 55 mph on the highway

Be “Mutually Exclusive, Collectively Exhaustive” (MECE) to identify all issues

1Structure the Presentation: Organize the Issues

13

Situation ComplicationsOptions to Consider

Depending on the purpose, a senior management presentation might include

the following components

Inform a

Decision

Problem(s) to address

Recommendations & Rationale

Implementation Plan

Build a Proposa

l

Purpose Potential Considerations

Reason for Action (or context)

Actions TakenResults Achieved

Showcase a

Success

2 Build the Story-line14

• How to Build an Issue Tree

• Mutually exclusive, collectively exhaustive (MECE)

Start as MECE as possible. It’ll help as you build.

• How to Turn an Issue Tree into Relevant Questions

Keep asking “so-what” and “why”

• Approaches for building a story line

Situation, complication, resolution

• Constructing a “ghost deck”

Key Lessons: Structure the Presentation

15


Recommendations



• Determine your message

1

• Identify the comparison

2

• Select the best chart type

3

MATERIAL ADAPTED FROM GENE ZELAZNY’S “SAY IT WITH CHARTS”

SESSION #2: Communicate the Analysis

16

The chart type you choose depends on the

specific point or message you want to

conveyTherefore, the first step is to identify

what message you want to share

1 Determine Your Message17

5 TYPES OF COMPARISONCOMPARISON

TYPEPURPOSE EXAMPLE

Component Displaying the size each component as a percentage of the total

X% of government contracts go to small businesses

Item Compare how things rank

Center X ranks fourth among all regions for program activity

Time Series Trends or changes over time

Student loan debt increased significantly

Frequency Distribution

Number of items that fall into numeric ranges

Most grants fall into the $15 million -$25 million range

Correlation Identify the strength of a relationship between two variables

There is a strong relationship between employee satisfaction and employees’ awareness of responsibilities

Source: Gene Zelazny, “Say it with Charts”

2 Identify the Comparison18

KEY TYPES OF CHARTS

Pie chart

Bar chart

Column chart

Line chart

Dot chart

Component

Item Time series

Frequency

Correlation

Pie

Bar

Column

Line

Dot

3 Select the Best Chart Types19

SCENARIO COMPARISON TYPE

CHART TYPE

Percentage of government contracts by business size classifications (i.e., small business)

Relationship marketing outreach & attendance at national parks

Change in student debt levels from 2007-2010

Breakdown of the value of goods exported by region, as a percentage of the total

Breakdown of grant awards by size buckets (i.e. 0-$500k; $500k-$1M, etc.)

Frequency of small business government contracts by NAICS code

Employee satisfaction vs. AWS participation rate

Speed of disaster assistance for a given disaster by time buckets (i.e., 1 week; 2 weeks; etc.)

2009-2010 weekly count of White House press releases

3Exercise!!!

20

• Determine the message

• Identify the comparison

Component, Item, Time Series, Frequency Distribution, Correlation

• Select the chart type

Pie, Column, Bar, Line, Scatter, or Other

Key Lessons: Communicate the Analysis

21



• Examples of Synthesis

• How to Synthesize & Develop Management Recommendations


Recommendations

1

2

Session #3: Synthesize & Develop Recommendations

22

Don’t just summarize. Summary re-states all the facts briefly; synthesis includes insights that help managers make decisions

Ask “so-what”, “why”, or what does this mean for my manager, office, team, constituent, etc.?

Lead with the answer (but it depends on the context)

Foreshadow: Your initial synthesized statement should map to the remainder of the actions/recommendations/analysis

Develop actionable recommendations that are supported by the facts

1 Tips for Synthesis23

1 • Structure/highlight the major issues

2 • Spell out the recommendations

3 • Present the action plan (or path forward)

4• Ask for agreement/commitment (if

appropriate)

5 • Close off with “next steps”

1Potential Components of a Synthesis

24

Dear Elizabeth W.,

How are you? I hope this note finds you well. I must say, I have not been doing too well—everything seems to be going wrong. My life is in shambles. For example, I had to pull two all-nighters last week for a very big report that was due last Friday. My boss said he told me about in two months earlier, but I totally forgot about it until he asked for a semi-final draft last Wednesday. I scrambled and just barely got it done—though I am so exhausted now. OMG.

And then this weekend was really sad because it was my 30th birthday on Saturday, and nobody came to my party!!! On Wednesday, I sent people a text message Friday night saying “come to my house tomorrow for dinner at 8 pm”. Half of my friends already had plans, and the other half went to my old house—because I forgot to tell them I moved!! Then they got mad at me for all the hassle. So my dog and I alone ate the 5 trays of meatloaf I cooked. Ugh.

And to top it off, I got a call from my bank, and they are going to repossess my car! Apparently I missed three months of payments (I totally forgot to send them my new address for the bills!). And when I tried to use my credit card at the grocery store, it didn’t even work, as it had been shut off after reaching my credit limit. Now I’m hungry and can’t drive to work.

What is wrong with me? Why do I have the worst luck?

Sincerely,

Josh

275 Words

2EMAIL EXAMPLE: Before synthesis…

25

2EMAIL EXAMPLE: After Synthesis…

Dear Josh, It seems you have a lot going on in your life. You may need some personal training in the areas of:

• Time Management• Communication• Organizational Skills

Have you tried using a Life Coach? Let me know if I can help. Good Luck!

Sincerely, Elizabeth W.

49 Words

26

2EMAIL EXAMPLE: Model Structure for an Answer

96 Words

Dear Josh,You’re disorganized—you need to better plan/schedule your work activities, social gatherings, and finances.

• At work, you haven’t anticipated deadlines or managed workload, causing work to build. Keeping a better team calendar may help.

• In social settings, you don’t plan events with advance notice and clear instructions, leading to damaged relationships. You should communicate better with friends, and send formal party invitations.

• Financially, you have missed several due dates, damaging your credit standing. Consider on-line ‘bill pay’ and other tools to easily update your records and manage your expenses.


27

Dear Josh,You’re disorganized—you need to better plan/schedule your work activities, social gatherings, and finances.

• At work, you haven’t anticipated deadlines or managed workload, causing work to build. Keeping a better team calendar may help.

• In social settings, you don’t plan events with advance notice and clear instructions, leading to damaged relationships. You should communicate better with friends, and send formal party invitations.

• Financially, you have missed several due dates, damaging your credit standing. Consider on-line ‘bill pay’ and other tools to easily update your records and manage your expenses.


Actionable recommendati

ons

Track the

options

What is the

action?

Lead with the

answer

Maintain a

structure

Highlight implicatio

ns

2EMAIL EXAMPLE: Model Structure for an Answer

96 Words

28

Amount =< $25K

$25K < Amount =<

$50K

$50K < Amount =<

$100K

$100K < Amount =<

$150K

$150K < Amount =<

$350K

$350K < Amount =<

$500K

$500K < Amount =<

$1M

$1M < Amount =<

$1.5M

$1.5M < Amount =<

$2M

Regular 7a 10.4% 8.7% 8.7% 10.3% 10.4% 9.0% 8.1% 9.8% 7.2%

SBA Express 15.1% 17.7% 14.3% 6.7% 4.2%

0%

5%

10%

15%

20%

25%

Nor

mal

ized

Pur

chas

e Ra

te

Normalized Purchase Rates, 2005 Cohort

Because larger SBA Express loans are

better performing, raising the caps on SBA Express above $350K may have a modest impact on

program cost

Amount =< $25K

$25K < Amount =<

$50K

$50K < Amount =<

$100K

$100K < Amount =<

$150K

$150K < Amount =<

$350K

$350K < Amount =<

$500K

$500K < Amount =<

$1M

$1M < Amount =<

$1.5M

$1.5M < Amount =<

$2M

Regular 7a 5.8% 9.7% 10.9% 13.4% 13.3% 11.9% 10.2% 10.8% 10.7%

SBA Express 16.6% 18.0% 17.5% 5.0% 4.6%

0%2%4%6%8%

10%12%14%16%18%20%

Nor

mal

ized

Pur

chas

e Ra

te


2Although SBA Express loans have relatively higher purchase rates, larger SBA Express loans are higher performing

Note: Normalized purchase rate is defined as SBA purchase amount as a percent of SBA-guaranteed portion of the total disbursement amount.

PowerPoint Example

29

Amount =< $25K

$25K < Amount =<

$50K

$50K < Amount =<

$100K

$100K < Amount =<

$150K

$150K < Amount =<

$350K

$350K < Amount =<

$500K

$500K < Amount =<

$1M

$1M < Amount =<

$1.5M

$1.5M < Amount =<

$2M

Regular 7a 10.4% 8.7% 8.7% 10.3% 10.4% 9.0% 8.1% 9.8% 7.2%

SBA Express 15.1% 17.7% 14.3% 6.7% 4.2%

0%

5%

10%

15%

20%

25%

Nor

mal

ized

Pur

chas

e Ra

te


Because larger SBA Express loans are

better performing, raising the caps on SBA Express above $350K may have a modest impact on

program cost

Amount =< $25K

$25K < Amount =<

$50K

$50K < Amount =<

$100K

$100K < Amount =<

$150K

$150K < Amount =<

$350K

$350K < Amount =<

$500K

$500K < Amount =<

$1M

$1M < Amount =<

$1.5M

$1.5M < Amount =<

$2M

Regular 7a 5.8% 9.7% 10.9% 13.4% 13.3% 11.9% 10.2% 10.8% 10.7%

SBA Express 16.6% 18.0% 17.5% 5.0% 4.6%

0%2%4%6%8%

10%12%14%16%18%20%

Nor

mal

ized

Pur

chas

e Ra

te


2Although SBA Express loans have relatively higher purchase rates, larger SBA Express loans are higher performing

Note: Normalized purchase rate is defined as SBA purchase amount as a percent of SBA-guaranteed portion of the total disbursement amount.

HIGHLIGHT THE

INSIGHTTHE SO-

WHAT

A DESCRIPTIVE LEAD

HIGHLIGHT THE

INSIGHT

PowerPoint Example

30

• Synthesize EVERYWHERE

• Ask “so-what”, “why”, or what does this mean

• Lead with the answer (depending on the context)

• Foreshadow

• Develop actionable recommendations

• Don’t just summarize. Highlight the insights

KEY LESSONS: Synthesize & Develop Recommendations

31

• Teams of 2-4 participants• Judges will be “VIP” SBA panel

Who?

• Develop and present a 5-10 page management PowerPoint presentation on any SBA-specific topic. Work should:

o Exhibit command of structuring and story-lining

o Use data-driven charts and graphs effectively

o Contain clear, actionable management recommendations and synthesis

What?

• To show everyone what they have learned• To have fun

Why?

SESSION #4: Applying the lessons…THE CONTEST

• Prizes for best weekly assignments• GRAND PRIZE for best final

presentation• Lunch at White House with Administrator

What can I win?

32

COMPETITION CHECKLIST: Criteria to Consider for Data-Driven Presentations

Mastery Area FeatureStructuring

the PresentationDoes the presentation build a compelling, coherent story?

Does the presentation identify (and prioritize) the most appropriate issues for analyzing a specific management question?

Communicating the Data

Does the presentation incorporate relevant data/evidence to inform a management decision?

Is the data presented in a visually-compelling format that sends a clear message?

Synthesizing Findings & Developing

Recommendations

Does the presentation surface interesting, high-level insights supported by the data?

Does the presentation provide actionable recommendations (or options) for management to consider for improving SBA’s operations/programs?

33





34

PERMEATION: Cultural shift created in the SBA

RESULTS OF HQ CLASSESo 1st Class Winner: Senior level task force formed that developed a

comprehensive plan to address retirements and led to a VERA/VSIP action.

o 2nd Class Winner: Analysis being used to inform Agency efforts for boosting volume of loans going to underserved markets.

35

Performance data highlighted a management question that needed exploration

SBA’s Average Time-to-Hire was Worse than Agency Target

36

An Issue Tree was created to ask relevant questions and inform the data analysis

How can SBA

reduce time to hire?

Cut # of steps

Cut time

per step

SBA Steps

Candidate Steps

Program Office

OHCM

Utilize hiring

flexibilities

Pre-Intervie

w

Interview

Offer

…

Direct Hire Authority

Excepted Service

Veterans

Part-Time Reemploymen

t (2010 NDAA)

STEP

PMF

Questions to Assess

• How many employees are eligible to retire?

• Ease of execution?

• Cost implications?

• Continuity in critical missions?

• Mentorship opportunities?

• Infusion of new talent?

37

Further Data Analysis Led To:

RISK In 5 years,

Retirement & VERA eligible employees will

outpace ineligible employees by

66% (2 out of 3 GS will qualify for an immediate annuity through voluntary or early retirement)

CURRENTLY

+3 Years

+5 Years

How can SBA mitigate effects of impending retirement tsunami?

38

How can SBA loan dollars more effectively go to underserved communities? Questions and analyses to

explore

Increase Demand for

Loan Programs by Underserved Community

Increase outreach to underserved

business communities

Increase the number of

businesses that qualify for loans

• What loan programs do banks use most in underserved communities?

• What are the characteristics/criteria for these loan programs?

Increase Supply of Loan to

Underserved Community

Increase the # of SBA

programs targeting

underserved businesses

Increase the number of

banks lending money in

underserved communities

• How many banks are serving small businesses in underserved communities? • What are characteristics of banks in underserved communities? (Size, national branch, local bank, etc)?

• What percentage of SBs are approved for loans through targeted underserved programs?

• Are SBA loan programs evenly distributed to underserved communities, by Region?

• Are SBA loans programs reaching the high concentration of underserved small businesses that they are created for?

39

21.4%

14.8%

12.8%

10.1% 10.1% 10.0%

6.7% 6.5%

5.2%

2.5%

0%

5%

10%

15%

20%

25%

8 6 1 3 7 4 10 5 2 9

SBA Region

% Distribution of Lenders, Underserved Businesses and SBA Dollars

% of Lenders

% of 8a/Hubzone

% of SBA USM Lending

21.4%

14.8%

12.8%

10.1% 10.1% 10.0%

6.7% 6.5%

5.2%

2.5%

0%

5%

10%

15%

20%

25%

8 6 1 3 7 4 10 5 2 9

SBA Region

% Distribution of Lenders, Underserved Businesses and SBA Dollars

% of Lenders

% of 8a/Hubzone

% of SBA USM Lending

Lender Outreach can be tailored to be region specific

.

Region 9’s Best Practices should

be shared

Highest concentrati

on of Lenders

USM Need not aligned

with results

A chart was used that highlighted insights…

40

PERMUTATIONS: The Training Course at SBA

BASICPrinciples

4 hours Covers basic

topics much like this presentation

Example Audience: Senior Leadership

CONDENSED Seminar

1.5 days More in-depth look at

principles Built-in homework Pre-defined options

and data sets for the final presentations

Example Audience: regional/national conference attendees

PRIMARYFull Course

4 to 6 weeks, each 2-3 hrs

Individual & Group homework

Deep dive into agency’s data sets & team-defined ACTIONABLE projects

Incentives for final presentation

Example Audience: Headquarters staff or dedicated field office

GOOD BETTER BEST

41

Interested in Finding Out More?

CONTACTTim Gribben

Director, Office of Performance Management

[email protected] (202) 205-7565

U.S. Small Business Administration409 3rd Street, SW, Ste 6000

Washington DC 20416

Josh BullProgram Analyst & Budget Analyst,

Office of Performance Management and Office of CFO

[email protected] (202) 205-7603Mat Pascarella

Program Analyst, [email protected] (202) 205-

6102

42

mailto:[email protected]



Documents

Structuring Performance Data to Effectively Drive Management Decisions Tim Gribben, Josh Bull, & Mat Pascarella O FFICE OF P ERFORMANCE M ANAGEMENT U.S