Performance Based Studies Research Group

www.pbsrg.com

State of Oklahoma Risk State of Oklahoma Risk Management TrainingManagement Training

State of Oklahoma Strategic Plan

• Compete based on value (proven performance and expertise)

• Transfer risk and control to vendor

• Minimize client decision making, directives, and control

• Vendor writes majority of the contract

• Vendor manages contract

• Measurement, measurement, measurement

State Vendor Expectations

• Proactive and Accountable– Vendor has control– Focus on risk in the seams– Risk – Plans to minimize risk – impact – customer satisfaction – Performance measurement – uses dominate information– Performance information/risk identification disengages the

bureaucracy

• Win-Win– Vendor’s success is just as important as State’s success– Goal alignment

• “No contract” – “Vendor writes their own” – Vendor has control– Contract is a guide– If we have to go to the contract then everyone has already lost– Should never have to go the contract

Filter 1Registration:

Past Perf. Info.

Filter 2Scope

RAVA plan

Filter 4Best Value

Prioritization

Filter 5Pre-planning

Phase:WR and Scope

Filter 6Weekly

Report &Post-

Rating

Time

Filter 3Interviews

High

Low

2009 State of Oklahoma 2009 State of Oklahoma Consultant Selection and Risk Consultant Selection and Risk Management ProcessManagement Process

Aw

ard

Sh

ort

listCriteria:

RAVATech.PPIQual.

Criteria:InterviewRAVAPPIQual. Id

en

tify

BV

5

Final RatingFinal Rating

• After the project is complete, the client will evaluate the project.

• The Final Rating will be heavily impacted by: – Change orders– Project delays– Poor quality– Owner surprises– Complaints– Ability to submit accurate and timely weekly reports

• In Best-Value Procurement, the vendors goal is to get a high rating in order to stay competitive.

6

50% Modification50% Modification

NO CRITERIA Ven 1 Ven 2 Ven 3 Ven 4

1 Ability to manage the project cost 9.4 9.8 9.4 9.52 Ability to maintain project schedule 9.6 9.8 9.7 9.83 Quality of workmanship 10.0 9.8 9.7 9.94 Professionalism 9.6 9.8 9.7 9.75 Close out process 9.3 9.7 9.5 9.56 Ability to communicate 9.4 9.8 9.4 9.57 Ability to follow the users rules 9.7 9.8 9.4 9.88 Overall customer satisfaction 9.9 9.8 9.7 9.89 Total number of different jobs 19 18 18 20

10 Total number of different customers 19 17 18 17

NO CRITERIA Ven 1 Ven 2 Ven 3 Ven 4

1 Ability to manage the project cost 9.4 7.4 9.4 9.52 Ability to maintain project schedule 9.6 7.4 9.7 9.83 Quality of workmanship 10.0 7.4 9.7 9.94 Professionalism 9.6 7.4 9.7 9.75 Close out process 9.3 7.4 9.5 9.56 Ability to communicate 9.4 7.4 9.4 9.57 Ability to follow the users rules 9.7 7.4 9.4 9.88 Overall customer satisfaction 9.9 7.4 9.7 9.89 Total number of different jobs 19 18 18 20

10 Total number of different customers 19 17 18 17

NO CRITERIA Ven 2

1 Ability to manage the project cost 9.82 Ability to maintain project schedule 9.83 Quality of workmanship 9.84 Professionalism 9.85 Close out process 9.76 Ability to communicate 9.87 Ability to follow the users rules 9.88 Overall customer satisfaction 9.89 Total number of different jobs 1810 Total number of different customers 17

NO CRITERIA Ven 2Current Project

1 Ability to manage the project cost 9.8 5.02 Ability to maintain project schedule 9.8 5.03 Quality of workmanship 9.8 5.04 Professionalism 9.8 5.05 Close out process 9.7 5.06 Ability to communicate 9.8 5.07 Ability to follow the users rules 9.8 5.08 Overall customer satisfaction 9.8 5.09 Total number of different jobs 18

10 Total number of different customers 17

NO CRITERIA Ven 2Current Project

Ven 2

1 Ability to manage the project cost 9.8 5.0 7.42 Ability to maintain project schedule 9.8 5.0 7.43 Quality of workmanship 9.8 5.0 7.44 Professionalism 9.8 5.0 7.45 Close out process 9.7 5.0 7.46 Ability to communicate 9.8 5.0 7.47 Ability to follow the users rules 9.8 5.0 7.48 Overall customer satisfaction 9.8 5.0 7.49 Total number of different jobs 18 18

10 Total number of different customers 17 17

State of Oklahoma Risk Management Process

• Vendor takes into account all client requirements and constraints

• Vendor creates a baseline (cost, schedule, scope, quality)

• Vendor identifies all risks that they do not control that could impact baseline plan and shows a plan to minimize them.

• Vendor takes into account all client concerns and shows how they will be minimized.

• If any deviation occurs during the project it is documented and tracked on the weekly report.

• Vendor will be rated by the client on their deviation from the baseline.

The Weekly Report

• An excel document that tracks pre-identified risks/concerns and actual risks/concerns on a project.

– Pre-Identified Risks/Concerns: • Any concerns of the client at any point in the project• Any risks identified before the project begins

– Actual Risks/Concerns• Risks that are unforeseen• Risks that the contractor feels has a good probability of

occurring

• All information on the weekly report must be in “Dominant” fashion.

– “Dominant” means:• Simple, clear, concise, accurate…..• Easy to understand without a lot of information.

Weekly Report Schedule

• Inserted in the “Schedule&Budget” tab of the weekly

• The schedule should go from task order award till construction close-out (even if contractor was only awarded till the design/workplan phase)

• The schedule items should include the following types of activities:– Major milestones– High risk activities

• The schedule should be updated every week. – It should have incorporated impact of risks– It should accurately reflect the current status of the project

Reporting Pre-Identified Risks

• Before a project begins the vendor should have done the following:

– Talked with all client parties and documented their concerns– Identified any potential risks that the vendor does not control– Created a plan to minimize all concerns and risks– Placed all concerns/risks and minimization plans in the weekly

report in the RMP tab

• Throughout the course of a project if anyone has concerns or identifies more potential risks the vendor should add them to the RMP tab with plans to minimize them.

• If a pre-identified risk becomes a potential problem on the project it is then documented on the Risks tab as well.

Reporting Actual Risks

• Actual risks should be identified on the “RISKS” tab of the weekly report

• When an actual risk is identified the following should documented:

– How it became a risk– If it was not pre-identified, an explanation of why it was not– A dated history of actions performed to minimize the risk– Impact to cost and time– Owner rating of risk

• Until a risk is resolved the following must be updated:

– Estimated impact to time and cost– Actions performed to minimize the risk– Owner rating of the risk

• Every resolved risk with an impact, must clearly be linked to a modification on the “Schedule&Budget” tab.

Industry Structure

High

I. Price Based

II. Value Based

IV. Unstable Market

III. Negotiated-Bid

Specifications, standards and qualification based

Management & Inspection

Best Value (Performance and price measurements)

Quality control

Competition

Pe

rfo

rman

ce

Low

High

Owner selects vendor

Negotiates with vendor

Vendor performs

Best Value is best predictable

performance

Award is based on subjective minimum

which has to be enforced

High

Low

Perf

orm

an

ce

Owners

“The lowest possible quality

that I want”

Contractors

“The highest possible value that you will get”

Minimum

Problem with Priced Based Systems: proliferates decision making

High

Low

Perf

orm

an

ce

Maximum

Perf

orm

an

ce

High

Low

Ris

k

High

Low

Impact of Minimum StandardsBlind vs Visionary

Contractor 1Contractor 2Contractor 3Contractor 4

Contractor 1

Contractor 2

Contractor 3

Contractor 4

Perf

orm

an

ce

High

Low

Ris

k

High

Low

Decision making: what is the minimum standard, and do all contractors meet the minimum standards

Traditional PM vs. New PM Model

RiskRiskss

Control Don’t Control

Me & Them Us

RisksRisks

Control Don’t Control

Initial conditions

Final conditions

Accurate perception leads to less decision making and expectations

Time

Laws Laws

Best Value Selection

Contractor 1

Contractor 2

Contractor 3

Contractor 4

Perf

orm

an

ce

High

Low

Ris

k

High

Low

Contractor 1

Contractor 2

Contractor 3

Contractor 4

Perf

orm

an

ce

High

Low

Ris

k

High

Low

Industry performance and capability

Highly Trained

MediumTrained

Vendor XCustomers

OutsourcingOwner

PartneringOwner

PriceBased

MinimalExperience

Impact on Professionalism

Best Value Environment• More accountability • More expertise• More professionalism

Price Based Environment• Less accountability• Less expertise• Less professionalism• Less perceived “value added”• Movements to make designer

work for contractors due to lack of value added

High

I. Price Based

II. Value Based

IV. Unstable Market

III. Negotiated-Bid

Specifications, standards and qualification based

Management & Inspection

Best Value (Performance and price measurements)

Quality control

Competition

Pe

rfo

rma

nce

Low

High

Owner selects vendor

Negotiates with vendor

Vendor performs

Minimized decision making

Maximized decision making

Dominant information

• What does the most qualified, experienced people in a firm do?

• Get work, or do work?

• Why is this?

• It is more important in a destabilized industry to get work then to do work….

• Downside….what happens to our professionalism????

ASTM Standard Setting

ASTM Standard Setting

What Will Manufacturer ‘C’ Do?

CInitial Elongation % (break) D412 200% 100%Min.Initial Tensile Strength (mass Stress) D412 450psi 150psi Min.Final Elongation % Weathering D412 200% 100%Min.

Permeance E96. B 3.52.5 U.S. perms

MinWater Absorption (mass) D471 <1%

Delete requirement

Weathering (5000 hours) G53

No cracking No cracking

Adhesion C794 3 pli 2pli min.

Fungi Resistance G21 No growthDelete

requirement

Tear Resistance D624 30 lb/in 20 lb/inLow Temperature Flexibility D522 Passes Passes

Viscosity D2196115-130

KU35,000-

50,000cps

Volume Solids D2697 57 - 66% 57-80%

TEST PROPOSEDPHYSICAL PROPERTY

Standards motivate

manufacturers to lower quality

There is something wrong with the delivery of services…..

No one knows how bad the problem really is…..

Entire system is broken….

Requires more management….

Performance/value is decreasing….

Relationships are more important than results….

Management

….it becomes less important to be skilled, accountable, and able to minimize risk

As management, control, and direction become more important…..

Skill 1 Skill 2 Skill 3 Skill 4

“Manager’s Code” The movement of risk.....when making

decisions

Don’t Mess With It!

YES NO

YES

YOU IDIOT!

NO

Will it Blow UpIn Your Hands?

NO

Look The Other Way

Anyone ElseKnows? You’re SCREWED!

YESYES

NO

Hide It

Can You Blame Someone Else?

NO

NO PROBLEM!

Yes

Is It Working?

Did You Mess With It?

Impact on Designer’s Value and Professionalism

Best Value Environment• More accountability • More expertise• More professionalism• Dominantly clear that the

designer can add the most value

Price Based Environment• Less accountability• Less expertise• Less professionalism• Less perceived “value

added”• Movements to make

designer work for contractors due to lack of value added

High

I. Price Based

II. Value Based

IV. Unstable Market

III. Negotiated-Bid

Specifications, standards and qualification based

Management & Inspection

Best Value (Performance and price measurements)

Quality control

Competition

Pe

rfo

rma

nce

Low

High

Owner selects vendor

Negotiates with vendor

Vendor performs

Minimized decision making

Maximized decision making

What is the difference?Do designers spend more time in reacting to bureaucracy or doing a good technical job?

2

3

1

4

1

2

3

4

Traditional PM vs. New PM Model

RiskRiskss

Control Don’t Control

Me & Them Us

RisksRisks

Control Don’t Control

Evolution of Delivery Processes

Design-Bid-Design-Bid-Build Build (DBB)(DBB)

Design-Design-Build Build (DB)(DB)

ConstructioConstruction Manager n Manager

At Risk At Risk (CMAR)(CMAR)

Design BuildBid

Design

BuildBid

Design-BuildBid

Where is value added?????

Design-Bid-Design-Bid-Build Build (DBB)(DBB)

Design-Design-Build Build (DB)(DB)

ConstructioConstruction Manager n Manager

At Risk At Risk (CMAR)(CMAR)

Design BuildBid

Design

BuildBid

Design-BuildBid

Who should be the center of the Universe in construction? In design?

High PerformingVendor

Low PerformingVendor

ClientProfessional

Client

Outsourci

ng

Direction,

control, review

Design Intent

ScopeCost Est.

Baseline $

Re-Scopew/Modifications

Final Design

CMAR Best Practices

Contractor

Designer

0% 30% 60% 100%

1

2

3

4

5

Bonding capacity

Self Regulating Loop(Six Sigma DMAIC Generated)

Actions• Minimize data

flow• Minimize analysis• Minimize control Scope

Risk Assessment/Value Added Plan

Preplanning, Quality Control Plan

Measure again

50%

Identify value (PPI, RA, Interview)V

50%

Interview Key PersonnelPast PerformanceInformation

M

Requirements(DBB, DB, CMAR, DBO)

Efficient Service

M R

MM

R

R

R

= Minimize Risk

= Self Measurement

= Identify Value

M

R

V

M

M

M

M

Filter 1Registration:

Past Perf. Info.

Filter 2Scope

RAVA plan

Filter 4Best Value

Prioritization

Filter 5Pre-planning

Phase:WR and Scope

Filter 6Weekly

Report &Post-

Rating

Time

Filter 3Interviews

High

Low

2009 State of Oklahoma 2009 State of Oklahoma Consultant Selection ProcessConsultant Selection Process

Aw

ard

Sh

ort

listCriteria:

RAVATech.PPIQual.

Criteria:InterviewRAVAPPIQual. Id

en

tify

BV

Past Performance Surveys/ Past Past Performance Surveys/ Past Performance SheetPerformance Sheet

Design Firm

Past Client

State of Oklahoma

Mail survey back to Design Firm

Compiles Information

Email/Fax Survey

Call To Verify

Design Firm

compiles and sends surveys and past

performance sheet back to State

Past Performance InformationForm 254

• Design firms

• Critical sub elements

• Lead project managers

Initial conditions

Final conditions

Traditional Management

Time

Laws Laws

Risk is deviation fromexpected measurements

D

D M&C

Initial conditions

Final conditions

When do we determine the expectation so we can measure performance?

Time

Laws Laws

Risk is deviation fromexpected measurements

M

M M

M

Scope

• Will be rated in relative fashion

• Conservative plan on how the work will be done

• If other options are available, they will be considered along with the risk

• 30,000 foot level description

• Final design description

• Milestone schedule

Risk Assessment and Value Added Risk Assessment and Value Added Plan (RAVA)Plan (RAVA)

• The Risk Assessment and Value Added (RAVA) Plan consists of two piecesThe Risk Assessment and Value Added (RAVA) Plan consists of two pieces– The Risk AssessmentThe Risk Assessment– The Value AddedThe Value Added

• The Risk Assessment section is used to identify high performing vendors The Risk Assessment section is used to identify high performing vendors that can:that can:– Identify and minimize risk Identify and minimize risk beforebefore the service has started the service has started – Deliver plan to minimize risk during the life of the service.Deliver plan to minimize risk during the life of the service.

• The RA Plan should clearly address the following items:The RA Plan should clearly address the following items:

– List and prioritize major risk itemsList and prioritize major risk items (areas that may cause the service (areas that may cause the service not to meet the expectations of the Government).not to meet the expectations of the Government).

– Each vendor should Each vendor should focus on risks it does not controlfocus on risks it does not control and and

– Explain how the vendor will Explain how the vendor will minimize the riskminimize the risk..

Risk Assessment Plan Steps

• What events cause risk

• How will the design firm minimize the risk from happening

• If the risk happens, what will the designer do

• Transfers control and the opportunity to minimize risk to the design firm

RAVA Plan Rated BlindRAVA Plan Rated Blind

• The RAVA Plan must NOT exceed The RAVA Plan must NOT exceed 22 pages. pages.

• The RAVA Plan must not have The RAVA Plan must not have ANY NAMESANY NAMES (such as (such as vendor or manufacturer names, personnel names, vendor or manufacturer names, personnel names, project names, product names, or company letterhead) project names, product names, or company letterhead)

• Do NOT include brochures, marketing information, or Do NOT include brochures, marketing information, or product names!product names!

• The RAVA Plans will be evaluated “blind” in order to The RAVA Plans will be evaluated “blind” in order to minimize any bias. minimize any bias.

Value Added DifferentiationValue Added Differentiation

• Answer the questions: “What value do I bring that differentiates Answer the questions: “What value do I bring that differentiates me from my competitors?”me from my competitors?”

• The Value Added Differentiation section of the RAVA Plan is to The Value Added Differentiation section of the RAVA Plan is to allow a vendor to:allow a vendor to:– Show how it will add value (that its competitors cannot)Show how it will add value (that its competitors cannot)– The impact of that value in simple, provable termsThe impact of that value in simple, provable terms– How the value added will be measured – must have an How the value added will be measured – must have an

impact on dollars, time, and/or satisfaction of the impact on dollars, time, and/or satisfaction of the GovernmentGovernment

• Marketing material is worthless and will negatively impact Marketing material is worthless and will negatively impact scoresscores

• Value Added is where vendors provide “great ideas” Value Added is where vendors provide “great ideas”

How The Submittal Process Works

Submittal

Evaluation Members

Proposal Form(1 page)

Proposal Form & Other Documentation

Proposal Form(1 page)

RAVA and Tech. Prop.(2 Pages Each)

Average Score

ContractingOfficer

ContractingOfficer

InterviewsInterviews

• The following individual will be interviewedThe following individual will be interviewed– Project Manager/Lead DesignerProject Manager/Lead Designer

• All interviews will be individual All interviews will be individual

• The same individuals used for the past performance survey will The same individuals used for the past performance survey will be the same individuals interviewed and will be the same be the same individuals interviewed and will be the same individuals that will be assigned to the projectindividuals that will be assigned to the project

• No bait and switchNo bait and switch

Sample Interview QuestionsSample Interview Questions

1.1. Why were you selected for this project?Why were you selected for this project?

2.2. How many similar projects have you worked on?How many similar projects have you worked on?

3.3. What is different about this projects from other clients that you have worked for?What is different about this projects from other clients that you have worked for?

4.4. What are you doing differently on this project than you have done before?What are you doing differently on this project than you have done before?

5.5. Identify, prioritize, and how you will minimize the risks of this project.Identify, prioritize, and how you will minimize the risks of this project.

6.6. What risks don’t you control? How will you minimize those risks?What risks don’t you control? How will you minimize those risks?

7.7. What are your personal goals on this project?What are your personal goals on this project?

8.8. How are you going to measure your performance?How are you going to measure your performance?

9.9. What value do you bring to the project in terms of differences based on dollars, What value do you bring to the project in terms of differences based on dollars, quality, or time?quality, or time?

10.10. How will you measure performance for this project? How will you measure performance for this project?

Best Value Selection

• Past Performance

• Risk Assessment (don’t control) / Value Added

• Scope/Technical Proposal (2 Page proposal/milestone schedule)

• Qualifications

• Interview

Filter 1Registration:

Past Perf. Info.

Filter 2Selection:RAVA plan

Filter 4Best Value

Prioritization

Filter 5Pre-planning

Phase:WR and Scope

Filter 6Weekly

Report &Post-

Rating

Time

Filter 3Interviews

High

Low

Rating Scheme (1-10, 1(not qualified, Rating Scheme (1-10, 1(not qualified, 5(can’t differentiate/don’t know), 10 5(can’t differentiate/don’t know), 10

(dominantly better(dominantly better))

Aw

ard

Sh

ort

listCriteria:

RAVATech.PPIQual.

Criteria:InterviewRAVAPPIQual. Id

en

tify

BV

Unforeseen Risks

PERFORMANCE SUMMARY• Vendor Performance• Client Performance• Individual Performance• Project Performance

QUALITY ASSURANCE• Checklist of Risks• Sign and Date

Risk Management Plan Sheet (RMP)

• Pre-Identified Risk• Risk Minimization

Risk Sheet (Actual)• Risk that actually occurs• Unforeseen Risks

Weekly Report

What are the measurements of a Designer

• Scope and cost

• Plan with minimal risk

• Plan to minimize risk

• Knowing this with minimal effort

• Has the designer done this before

• Do they know what they don’t know on a future project

• Can they clearly articulate their vision

Traditional PM Model: Influence and Control (micromanagement)

• Reactive

• Focus on changing people

• Followers are the constraint

• Requires maximum amount of resources

• Relieves management from accountability of nonperformance

• Inefficient and nonperforming

New PM model: Leadership Model(measurements)

• Proactive

• Alignment (measurement)

• Requires Understanding (acceptance of measurements)

• Leader/manager and system is the constraint (blindness)

• Focus is on setting up a quality controlled system based on performance measurements, best value requirements, and dominant information

• Efficient (measurement)

Filter 1Past

PerformanceInformation

Filter 2Proposal & Risk / Value

Plan

Filter 4Prioritize (Identify

Best Value)

Filter 5Pre-Award

Phase (Pre-Plan)

Filter 6Weekly

Report &Post-

Rating

Time

Qualit

y o

f V

endors

Filter 3Interview

Aw

ard

High

Low

State of OklahomaState of OklahomaBest Value Process for Designers and ContractorsBest Value Process for Designers and Contractors

Differences in new system

• Provide past performance information (PPI) from references

• PPI on organization and critical components

• Submittals (scope and RAVA plan) will be two pages each– One additional page for milestone schedule

• Interview will be with the project manager– How do you know you can do it– How are you doing it– How are you taking control of the project and minimizing risk

and decision making of client– How did you pick your engineering subvendors

Requirements on Competing Firms

• More or less time invested on a project?

• More or less detail?

• Risk they control or do not control?

• Dominant differential or technical qualifications?

• Risk minimization done when?

• More risky approach or conservative control?

Designer should separate the requirement from the risk

• Propose the vision that they can do with minimial risk

• Propose to immediately analyze any “potentially risky options” in the beginning of the design contract

• Only pursue “risky options” if there is dominant information that justifies the “perceived risk”

• Cost should be based on “vision” and “requirements”

• Makes all other parties accountable and gives control of the design back to the design professional

Delivery of Design Services

• Control, manage, direct, and inspect

• Increase the flow of information

• Inefficient, ineffective

• Maximizes technical issues

Initial conditions

Final condition

s

Time

Laws Laws

Initial conditions

Final condition

s

Time

Laws Laws

RisRisksks

ControlDon’t

Control

Me & Them

Us

RiRiskskss

Control

Don’t Contro

l

• Align• Transfer control to the contractor• Preplanning• Quality Control• Measure• Minimize flow of information•Minimizes technical issues

Status Quo

New Design Delivery Model

M

M

M

M

MX