Proposals & EvaluationsProposals & Evaluations– A Government/Industry Perspective – A Government/Industry Perspective
Bill StockmanDayton Aerospace, Inc.
2 © 2006 Dayton Aerospace, Inc.
Cost Issues Today
FAR 12 to FAR 15 Conversions Berry Amendment AFMC Sponsored RFP Improvement Study Source Selection Approaches
3 © 2006 Dayton Aerospace, Inc.
FAR 12 to FAR 15 Changes
12.214 -- Cost Accounting StandardsCost Accounting Standards (CAS) do not apply to contracts and subcontracts for the acquisition of commercial items when these contracts and subcontracts are firm-fixed-price or fixed-price with economic price adjustment (provided that the price adjustment is not based on actual costs incurred).
15.406(b) When cost or pricing data are required, the contracting officer shall require the contractor or prospective contractor to submit to the contracting officer (and to have any subcontractor or prospective subcontractor submit to the prime contractor or appropriate subcontractor tier) the following in support of any proposal:(1) The cost or pricing data.(2) A certificate of current cost or pricing data, in the format specified in 15.406-2, certifying that to the best of its knowledge and belief, the cost or pricing data were accurate, complete, and current as of the date of agreement on price or, if applicable, an earlier date agreed upon between the parties that is as close as practicable to the date of agreement on price.
• What is the impact on the cost community when existing programsSwitch mid-stream from FAR 12 to FAR 15 cost requirements?
• “Commercial-like” programs from the mid-1990s that were competed but no competition left today
4 © 2006 Dayton Aerospace, Inc.
Buy America & Berry Amendment Provisions FY 2004 Defense Department Authorization Bill
The Berry Amendment requires the U.S. Defense Department (DOD) to buy certain products -- judged essential to our military readiness - with 100% U.S. content and labor. These products include clothing and other textile items, specialty steel, and food. In addition to the Berry Amendment, certain other "Buy American" provisions apply to DOD.
ISSUE:Rep. Duncan Hunter, Chairman of the House Armed Services Committee, has strengthened and expanded both the "Berry" and "Buy American" provisions in House version of the FY 2004 Department of Defense (DOD) Authorization bill by:
•Tightening the waiver process so that the Berry Amendment is not waived for inappropriate and arbitrary reasons. (Berry)•Clarifying that the Berry Amendment requirements must be met throughout all levels of the supply chain. (Berry)•Directing DOD and prime defense contractors to purchase U.S. made machine tools & dies and specialty metals. (Buy American)•Increasing the minimum U.S. content for defense procurement from 50% to 65%. (Buy American)•Requiring the Secretary of Defense to create a break-out list of critical components and technologies fundamental to our national defense effort, with a phased-in requirement that these items be 100 % U.S. made. (Buy American)
5 © 2006 Dayton Aerospace, Inc.
Berry Amendment & the Cost Community
Cost of compliance on new programs Cost of compliance on existing programs
Determining current content for primes and subs Determining cost to change design to comply Impact on current production programs Impacts on reliability Cost benefit analysis Waivers
Improving Proposals through better RFPs Improving Proposals through better RFPs – A Government/Industry Perspective – A Government/Industry Perspective
Bill Buzzell & Bob KayuhaDayton Aerospace, Inc.
7 © 2006 Dayton Aerospace, Inc.
Purpose of StudyPurpose of Study
Improving Proposals through better RFPs – A Government/Industry Perspective
Basic premise is that if we can improve RFPs- It will result in better proposals,
- Which will result in better executed programs
This study was accomplished under contract with the US Air Force HQ AFMC.This study was accomplished under contract with the US Air Force HQ AFMC.
8 © 2006 Dayton Aerospace, Inc.
DAI Study ApproachDAI Study Approach
DAI Group SessionDAI Group Session
Industry On-lineSurvey
Industry On-lineSurvey
One-on-OneInterviews
One-on-OneInterviews
Industry GroupSession
Industry GroupSession
Analysis &Documentation
Analysis &Documentation
Final ReportFinal Report PresentationsPresentations
• Generated initial subject areas• Guided Survey & Interview questions
• Blind Survey• Non-Attribution (company or individual)• Results guided Industry Session
• Brainstormed problems• Categorized problems into groups• Prioritized problems in each group• ID’d owners & provided recommendations
• Consolidated data• Summarized trends and major findings
• HQ AFMC• APMP Conference Presentation
9 © 2006 Dayton Aerospace, Inc.
Statement of Objectives – Focus of PresentationStatement of Objectives – Focus of Presentation
1. Identify specific industry capture strategies
2. Organization of industry capture teams
3. Methods used to develop a proposal
4. Identifying known problems with Air Force RFPs
5. List industry best practices and lessons learned
6. Examples from actual programs
7. Providing specific recommendations for process improvement
10 © 2006 Dayton Aerospace, Inc.
RFP Study ResultsRFP Study Results
Survey Results
Interview Results
Group Session Results
11 © 2006 Dayton Aerospace, Inc.
Survey ResultsSurvey Results
What are the major government drivers inhibiting a good RFP?
0% 10% 20% 30% 40% 50% 60% 70% 80%
Inexperienced evaluators
RFPs not reviewed by decision makers with experience
Functional “pet rock” requirements disrupt the RFP
DRFP released before acquisition strategy fully thought out
Selection criteria used not specified in Section M
RFP developed by inexperienced people – too much “cut and paste”
Other
12 © 2006 Dayton Aerospace, Inc.
Survey ResultsSurvey Results
Which of the following RFP timetable issues normally cause you the most problems?
0% 10% 20% 30% 40% 50% 60% 70% 80%
Locked in dates for submission
Not enough time to respond
Delay in release of final RFP
Major changes between DRFP and final RFP (with no submittal extension)
Government doesn’t realize cost impact to proposal schedule slips
Other
13 © 2006 Dayton Aerospace, Inc.
Survey ResultsSurvey Results
Which of the following RFP program schedule issues (as dictated in an RFP) normally cause you the most problems?
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Government mandated program schedule or milestones
Last minute re-phasing of funding in final RFP release
Funding profile unrealistic (can’t match a realistic program schedule)
Government mandating a one-for-one match between the WBS and the Integrated Master
Schedule tasks
Other
14 © 2006 Dayton Aerospace, Inc.
Survey ResultsSurvey Results
Which of the following parts of the RFP usually cause the most problems in terms of inconsistent program guidance?
0% 10% 20% 30% 40% 50% 60% 70% 80%
Overall Instructions
Section L Instructions
Section M Instructions
Cost Instructions
Past Performance Instructions
Other
15 © 2006 Dayton Aerospace, Inc.
Survey ResultsSurvey Results Which of the following parts of the RFP usually cause the most
problems in terms of inconsistent format guidance?
0% 10% 20% 30% 40% 50% 60% 70% 80%
Overall Instructions
Section L Instructions
Section M Instructions
Cost Instructions
Past Performance Instructions
Other
16 © 2006 Dayton Aerospace, Inc.
Survey ResultsSurvey Results
Which of the following government prioritization issues normally cause you the most problems?
0% 10% 20% 30% 40% 50% 60% 70% 80%
Poor program technical definition
Too prescriptive/restrictive program requirements
Bad or non-existent pre-RFP requirements review process
Funding Profile – incompatible funding profile to program requirements
Conflicting government priorities (e.g., conflicting functional requirements, conflicting
priorities within different sections of the RFP)
Other
17 © 2006 Dayton Aerospace, Inc.
Survey ResultsSurvey Results
Which of the following government RFP review process issues do you feel result in a poorly prepared RFP?
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
No industry input to cost ( & schedule) of program (funding required) prior to RFP release
Draft RFP process not handled well (e.g., badly coordinated, too short a timeframe, etc.)
RFP poorly coordinated within government
Poorly run Q&A process
Other
18 © 2006 Dayton Aerospace, Inc.
RFP Study ResultsRFP Study Results
Survey Results
Interview Results
Group Session Results
19 © 2006 Dayton Aerospace, Inc.
Interview ResultsInterview ResultsAcquisition Strategy
• Make sure the RFP reflects the acquisition• Past RFPs are applied incorrectly to new acquisitions• Use a gate process – start with past performance and screen bidders, then move to SOW, IMP/IMS etc• Need execution team on both sides - not just evaluation team and proposal team – “avoid bait & switch”
Acquisition Strategy• Make sure the RFP reflects the acquisition• Past RFPs are applied incorrectly to new acquisitions• Use a gate process – start with past performance and screen bidders, then move to SOW, IMP/IMS etc• Need execution team on both sides - not just evaluation team and proposal team – “avoid bait & switch”
Pre-Proposal Discussion• Don’t use only one industry day – Have a series of industry days & provide early feedback to contractor• Negotiate proposal page limitations during Industry Days• Be absolutely explicit in answering contractor Q&As Risk Reduction review – look at entire program – facilities, people etc, rather than just engineering status More discussion is better – need to develop relationship between all Govt/Industry functional players
Pre-Proposal Discussion• Don’t use only one industry day – Have a series of industry days & provide early feedback to contractor• Negotiate proposal page limitations during Industry Days• Be absolutely explicit in answering contractor Q&As Risk Reduction review – look at entire program – facilities, people etc, rather than just engineering status More discussion is better – need to develop relationship between all Govt/Industry functional players
Requirements• Make sure Section L and Section M agree• Past Performance evaluation criteria that clearly reflects the most important requirements • Past Performance criteria should reflect risk items.• Section L & M conflicts generate the biggest issues in services type proposals – RFP Traceability matrix • Govt shouldn’t hide assumptions or concerns (risks) – if it is considered a risk, then tell the contractors
Requirements• Make sure Section L and Section M agree• Past Performance evaluation criteria that clearly reflects the most important requirements • Past Performance criteria should reflect risk items.• Section L & M conflicts generate the biggest issues in services type proposals – RFP Traceability matrix • Govt shouldn’t hide assumptions or concerns (risks) – if it is considered a risk, then tell the contractors
20 © 2006 Dayton Aerospace, Inc.
Interview Results - continuedInterview Results - continuedExperience/OJT
• Main problem – experience level in program office• Senior leadership needs to be more proactive• Use more senior review – but they need to be experienced and have the right information• Need to get back to basics – overall process is sound, mainly a lack of discipline• Lack of experience, especially in the contracts area – training is a big issue• Smaller contracting shops at centers not disciplined – award size “driven under the radar”• Variability extensive between centers – standardization needed - simplify task order procedure• Need independent risk assessment of proposals
Experience/OJT• Main problem – experience level in program office• Senior leadership needs to be more proactive• Use more senior review – but they need to be experienced and have the right information• Need to get back to basics – overall process is sound, mainly a lack of discipline• Lack of experience, especially in the contracts area – training is a big issue• Smaller contracting shops at centers not disciplined – award size “driven under the radar”• Variability extensive between centers – standardization needed - simplify task order procedure• Need independent risk assessment of proposals
Formal Training• Risk Day – good concept – no one understands risk!! Force training for new people
Formal Training• Risk Day – good concept – no one understands risk!! Force training for new people
Proposal Response Govt needs to ask for a risk confidence level in proposal assumptions Oral proposals over written
Proposal Response Govt needs to ask for a risk confidence level in proposal assumptions Oral proposals over written
Scheduling IMP/IMS – Govt doesn’t understand what they are asking for – can’t evaluate it when they get it IMP/IMS growth – IMS too large – ask for in rolling wave
Scheduling IMP/IMS – Govt doesn’t understand what they are asking for – can’t evaluate it when they get it IMP/IMS growth – IMS too large – ask for in rolling wave
21 © 2006 Dayton Aerospace, Inc.
RFP Study ResultsRFP Study Results
Survey Results
Interview Results
Group Session Results
22 © 2006 Dayton Aerospace, Inc.
Industry Group SessionIndustry Group Session
Acquisition Strategy Experience/OJT Formal Training Funding Pre-proposal Discussion Proposal Response Requirements Scheduling
Major Categories of Findings
23 © 2006 Dayton Aerospace, Inc.
Acquisition Strategy FindingsAcquisition Strategy FindingsFinding Score Description OPR Recommendation
Lack of a well thought out total Program Acquisition Strategy
62
No clear roadmap
Lack of disciplined planning
Lack of experience and proper Senior level review
PM PEO
ASP training to the SS team
Keep it simple – consider incremental baseline reviews
Coordinate with Contractor and Warfighter early
Acquisition Strategy Changes
49
Requirements changes after RFP release
Programmatic changes drive schedule slippages which drive increased B&P costs
SAF/ ACE
PM
User
Involve Industry upfront in reqmts generation/refinement
Changes/slips approved at higher management level
Evaluation Criteria 43
Unclear or mismatched criteria to the requirements
CMMI not effectively used as evaluation criteria
PM
ACE
Use ASP to review evaluation criteria
Govt Red Team the criteria
Lack of Understanding the Program Cost Risk
29
On down selects competitions there are multiple technical TIMS, but cost TIMS are rarely conducted
SAF/ ACE
PM
SAF/ACE needs to establish minimum cost risk criteria
Actively address contractor’s cost estimating assumptions
Government desire to be integrator
20Misunderstanding about the Govt’s integration role
PMFormally define categories of Govt support – use ASP process for vetting
24 © 2006 Dayton Aerospace, Inc.
Experience/OJT FindingsExperience/OJT FindingsFinding Score Description OPR Recommendation
Inexperienced Program Managers
45
Lack of management overlap due to re-assignment
Formal training but little actual acquisition experience
HQ AFMC
SAF/ AQ
Need active mentoring process
Need effective career management plans for future senior PMs
Inexperienced technical evaluators
38
Reductions in workforce have stretched resources
Failure of civil service to hire primarily at entry level
SAF/ ACE
HQ AFMC
Develop pool of technical evaluators at each center
Formal SS training coupled with OJT
Contractor Questions 34Questions not clearly answered prior to formal RFP release
PM
PCO
Need effective review process for release of Govt answers
Build time into acq schedule
Use additional Industry Days to address questions
Government and Industry Interface
27Not working as partners when the DRFP is written
PMPM should set the tone and ensure a level playing field
Award Fee Contracts 21
Failure to identify Award Fee Evaluation criteria prior to contract award
Contractors need to know criteria prior to submission
SAF/ ACE
Need formal policy addressing Award Fee Criteria in RFPs
Award Fee Plans should be negotiated prior to contract award
25 © 2006 Dayton Aerospace, Inc.
Formal Training FindingsFormal Training FindingsFinding Score Description OPR Recommendation
Lack of a clear, integrated RFP
51
Leads to inconsistencies, omissions & contradictions
A symptom of inadequate training and experience
PM
PCO
Process issue under the control of the PM and PCO
Build reqmts matrix to align Section L, M, SOO, etc
T&Cs not always appropriate for the type of contract being solicited.
31
Contractor wastes valuable time sorting incorrect clauses
A symptom of inadequate contracts experience
PCOTraining and templates will help
Early preparation and review of the model contract during DRFP
Inconsistency between what Contracts and Engineering expects from the awarded contract
29
Reflects and inadequate Acquisition Plan and ASP
Lack of experience in creating an executable contract
PM
Ensure contract execution is foremost in the development & structure of the contract
Feedback from Industry/User
Little or no risk identification by government
28
Govt doesn’t understand - Not presented in the written draft RFP or at the industry day
Government does not solicit inputs from Industry
PM
Need formal proposal and program risk training
Don’t hide risks – Need open discussion of risk during Industry Days
Government training on RFP writing is deficient or non existent.
26
Systemic issue - tendency in Program Office to press forward without proper preparation/training
SAF/
ACE
Need ACE guidance and well-defined training program
Make mandatory before DRFP process is initiated
26 © 2006 Dayton Aerospace, Inc.
Funding FindingsFunding FindingsFinding Score Description OPR Recommendation
Unrealistic funding profiles and total program estimate from the beginning
67
Funding profile does not reflect work scheduling
Inefficient profile – shortage of funding early
Failure of leadership to recognize/acknowledge program uncertainties
PM
SSA
ASP must address funding profile vs. system requirements and expected delivery times
PM must facilitate trade-off and cost impacts discussions between contractors and user
SSA must be final decision maker if shortfall identified after the fact
Too many changes to RFP requirements which are cost prohibitive
40Lack of control of the RFP by the PM, allowing user changes late in the process
PM
Requirements need to be well defined prior to RFP release
Use industry feedback on DRFP/SOW to finalize requirements
Spiral development is requested, but Section L asks for waterfall
23
Lack of effective review prior to RFP release
Can result in improperly proposed program costs
PM
ASP
Govt Red Team review of the RFP
Should be addressed as part of the ASP
27 © 2006 Dayton Aerospace, Inc.
Pre-Proposal Discussion FindingsPre-Proposal Discussion FindingsFinding Score Description OPR Recommendation
Discussions with industry
57
Inadequate opportunity for technical discussions prior to formal RFP release
RFP authors work in isolation
PM
PM sets the tone and “requirement” for industry participation
Mandatory Industry Days
Draft RFP 48
Inadequate use of Draft RFPs
No follow-up face-to-face meeting with industry after Draft RFP comments
PM
SAF/
ACE
Should be a firm reqmt and part of the ASP and acquisition schedule
Need standard DRFP procedure
Internal Government Discussions
37
Lack of coordination between contracts, engineering, user
Too much “Govt only” discussion on key issues
PMPM responsibility to assure balance and flow of information within Govt and out to industry
Proposal Schedule Slippages
28
Formal RFP released too soon resulting in both proposal and program schedule adjustments
Post RFP release schedule slips can drive significant additional B & P costs
PM
PCO
Responsibility of ASP process to hold program to a realistic and executable RFP schedule
PM and SSA must agree on the RFP schedule and make necessary allowances for holding to it through award
28 © 2006 Dayton Aerospace, Inc.
Proposal Response FindingsProposal Response FindingsFinding Score Description OPR Recommendation
Not enough lead time to respond to RFP
46
Especially when no DRFP
Lack of effective planning
No understanding of industry process
PMEarly planning with industry involvement
Industry Day & DRFP
RFP Q&A response time 38
Government takes too long to answer questions from industry on program (Industry Day, DRFP, etc.)
PCO
Establish reasonable disciplined schedule to receive industry comment and government response
Schedule slippages ... drive B & P dollars...after company budget has been approved for the year
31Too often government changes solicitation schedule in the middle of the process
PM
Establish schedule & stick to it
Change only as last resort
Inform Industry early if slip required
In many competitions there are multiple technical TIMS, but cost TIMS rarely conducted
25Government spends a lot of time on technical issues and not enough on cost issues
FM
Conduct Cost TIMS parallel with technical TIMSs to help develop a common understanding of the cost risk
Lack of appreciation of contractor effort to submit a fully compliant proposal
25Government personnel have very little understanding of industry proposal process
DAU
AQ
Provide material on generic industry proposal process in acquisition and source selection training
29 © 2006 Dayton Aerospace, Inc.
Requirements FindingsRequirements FindingsFinding Score Description OPR Recommendation
Not enough discussion on requirements - interface with user is inadequate
34
Inadequate opportunity to discuss requirements with tech team and users
Results in unreasonable expectations
PM & User
More focus on Industry Days - include Users
Initiate an early requirements risk management process
Section L and M issues 33Still – inconsistencies between Sections L & M
MDA / PEO
Need an independent last scrub (red team/murder board) prior to RFP release
Unrealistic funding profiles and total program estimate from the beginning
33Inconsistency between requirements and available funding
PM / FM / User
Early industry involvement to establish reasonable level of requirements for available funding
RFP Changes 27
After release of the RFP
Not enough upfront time taken to fully define requirements
PM / User
Allow sufficient time before RFP to nail down requirements.
Do not allow any but imperative Senior Group directed changes
SOW requirements/ schedule conflicts
23SOW requirements inconsistent with other parts of RFP
PM/ PEO
Industry/Government communication open and early.
Disciplined independent review process of RFP
Take time to do it right
30 © 2006 Dayton Aerospace, Inc.
Scheduling FindingsScheduling Findings
Finding Score Description OPR Recommendation
Inadequate funding for the requirements
43The scheduling aspect of this finding is a mismatch of expecting too much too soon
PMEnsure a balance between cost, schedule and technical requirements
Large changes to the RFP requiring major proposal re-submittals
41Changes in RFP timeline causes significant impact to industry participants
PM
Government needs to understand the actual industry proposal process.
Establish a disciplined RFP schedule and don’t change it
SOW requirements
/schedule conflicts40
Over restriction of schedule for the funding profile and requirements
PM
Early interface with industry to allow them to assist in definition of a realistic schedule for the funding and requirements
RFPs released with no or inaccurate information
33Cut and paste results in RFPs that do not match program or missing data
PMIndependent review of RFP.
Interface with industry
IMP/IMS 31
Ineffective use of this tool.
Limitations that make it impossible to demonstrate the entire program
PMTraining on purpose and use of IMP/IMS
Do not put limitations on IMS
Three Analysis Examples
32 © 2006 Dayton Aerospace, Inc.
Resource Analysis PlanAA BB CC DD EE FF
Narrative Review Yes Min Yes Yes Yes Yes
Detailed Labor Dist Yes Yes Yes Yes Yes YesBy FTEFTE Summary/Rate Yes Yes Yes Yes Yes Yes
Detailed Labor Rate Yes Yes Yes Yes Yes Yes
Subcontr Labor Anl Yes Yes Yes Min Diff Imbedded
O/A Rates Yes Yes Yes Yes Yes Yes
HW Refresh Details Yes Yes Yes No Yes No
HW Costs Yes Yes Yes Min Yes No
CLIN Summary Yes Yes Yes Yes Yes
Summary Cost B/O Yes Yes Yes Yes Yes Yes
WBS summary Yes Yes Yes Min Diff Yes
Transition WBS Yes Yes Yes No Diff No
Transition B/O Yes Yes Yes Yes No Yes
Trans Labor rates Yes Yes Yes Yes Yes Yes
Subs WBS B/O Yes Yes Yes
Trans Sched Yes yes Yes Yes yes yes
FTE Trans Ramp
SS FTEs
O/A Rate Impacts
Mgmt Metrics
Small Business
Dollarization TBD TBD TBD TBD TBD TBDS/W/RContract ClauseImpacts
33 © 2006 Dayton Aerospace, Inc.
Combined PRAG/Resource Allocation
VRRSRNRNO PP
WBS 1.2 WBS 1.3 WBS 1.4 WBS 1.5 Total TotalFTEs $$ FTEs $$ FTEs $$ FTEs $$ FTEs $$
AA 24.9% 18.9% 62.6% 50.6% 83.7% 76.9% 27.6% 21.6% 36.5% 30.4%A 6.8% 9.8% 20.7% 27.8% 11.2% 15.0% 10.4% 14.2%B 19.3% 15.5% 9.9% 7.5%CTI 18.1% 16.1% 9.3% 7.9%D 10.7% 15.9% 3.0% 5.6% 16.3% 23.1% 7.0% 10.6%E 4.4% 4.5% 4.9% 3.9% 10.1% 9.3% 5.6% 5.2%F 23.0% 23.6% 5.2% 5.4%G 9.8% 10.7% 5.0% 5.2%H 20.3% 22.8% 4.5% 5.2%I 2.4% 3.0% 7.8% 10.4% 2.9% 3.7%J 6.0% 5.8% 1.3% 1.3%K 2.4% 3.0% 1.2% 1.5%L 1.2% 2.5% 0.6% 1.2%M 1.8% 1.9% 0.4% 0.4%N 1.0% 1.8% 0.2% 0.4%
34 © 2006 Dayton Aerospace, Inc.
Total Evaluated Costs
• In theory, if one dollarizes all risks and color ratings, then all bids are “normalized” to green/low• This would allow for a simple selection of lowest total evaluated cost• Reality normally is that some items can’t be dollarized, so best value is the rule
AA BB CC DD EE FF
Total Proposal Costs 2,744,441,539$ 2,887,974,133$ 2,658,372,238$ 2,959,151,881$ 2,316,283,803$ 2,371,052,616$
Direct Costs/Savings (105,274,578)$ (86,537,850)$ (81,927,900)$ (87,735,050)$ (98,232,915)$ (90,768,200)$
Direct Benefits/Risks 72,837,396$ 160,554,692$ 341,698,715$ 85,188,829$ 334,654,804$ 393,705,250$
Eval Factors--SWR (30,163,824)$ (37,783,380)$ (11,436,000)$ (5,704,000)$ (8,457,480)$ (13,855,732)$
Total Evaluated Cost 2,681,840,534$ 2,924,207,595$ 2,906,707,053$ 2,950,901,660$ 2,544,248,211$ 2,660,133,934$