Challenges and Solutions to Conducting High Quality

Contract Evaluations for the U.S. Government

David J. Bernstein, Ph.D., Westat: Rockville, MD

Discussant: Kathryn E. Newcomer, Ph.D., The George Washington University, Washington, DC

Note: Opinions/perspectives in this presentation are those of the presenter or discussant, and do not reflect the position of Westat, its clients, or The George Washington University.

Presented July 7, 2015: Washington Evaluators Brown Bag Session

2

IntroductionProblem: The vast majority of United States Federal government evaluations are conducted by contractors, but effective contracting is rarely examined.

Solution: A panel of government evaluators, contractors, and academics addressed 5 questions related to evaluation contracting and how it can be done more effectively.

3

BackgroundPresentation based on the 2nd of 3 panels at the American Evaluation Association Conference.

•First panel: 2012 AEA Conference: U.S. Federally-funded Contract Evaluation: Communicating to Enhance Relationships, Responsibilities, Relevance

•Second panel: 2014 AEA Conference: Effective and Sustainable Practices for Government Evaluation Contracting (Results being presented here)

•Third panel: Proposed for 2015 AEA Conference: Exemplary Practices in Contracting for Public Program Evaluations (Stay tuned!)

4

2014 AEA Panel Participants• David J. Bernstein, Westat, Panel Chair• Kathryn Newcomer, George Washington University, Discussant

Consultants• William H Scarbrough, PIRE• Jennifer Hamilton, Westat• Herbert Mark Baum, HMB Consulting Services

Government Evaluators• Rakesh Mohan, Idaho State Legislature• Jason Compy, U.S. Department of Agriculture

NOTE: Opinions in this presentation are those of the presenter and maybe these panelists, and are not official positions of Westat or the agencies for which panelists work.

5

Contracting for Evaluation

http://bj-o23.deviantart.com/art/NOT-Rocket-Science-324795055

However, it is complicated because:

• Procurement regulations are detailed, maybe outdated.

• Agency practices differ across the Federal government.

• Appears to be a lack of research (if any) focused on contracting for Federal evaluation work (although there are GAO and other studies on Federal contracting).

6

Format for Discussion1. David will present one of the five issues, and the

discussion summary from the 2014 AEA Panel.

2. The audience will then address the question from their own perspective. We want a mix of government evaluator and contractor perspectives, but especially want to hear from government evaluators.

3. Kathy will provide her own perspective on the question, and raise related questions/issues.

4. Total of 15 minutes per question.

7

Question 1

1. Name one legal and/or regulatory obstacle that can affect the quality of contracted evaluations.

What are the potential solutions?

8

Q1: Legal and Regulatory Obstacles

Obstacle 1: Agency leadership sometimes do not take evaluation seriously.

•Solution 1: Obama Administration’s emphasis on strategic priorities, evidenced-based approaches, set-aside authority, funding flexibility.

•Solution 2: Chief Evaluation Officers and/or Chief Performance Officers’ influence.

Can provide guidance on improving evaluation scopes.

More productive evaluation portfolios and RFPs.

9

Q1: Legal and Regulatory Obstacles

Obstacle 2: Seemingly arbitrary/inflexible timelines:

Legislative timelines matter. Agency funding cycles matter. BUT, agency timelines are not always realistic.

•Solution 3: Stage evaluation study timelines.

Evaluation Planning: Evaluability assessment, data collection form development, and OMB review in Year 1.

Data Collection: Data collection, analysis in the outyears. Context: Build in time for process, implementation reviews. Outcomes: Allow interventions time needed to produce and

demonstrate intended outcomes.

10

Q1: Legal and Regulatory Obstacles

Obstacle 3: OMB Information and Regulatory Affairs (OIRA) reviews for Paperwork Reduction Act (PRA).

Adds 6-9 months to an evaluation. Some program offices do not understand PRA, are

unrealistic about timing, or think that 9 is a magic number.

•Solution 4:

Realistic discussions during kick-off meetings re: what is doable. Don’t reinvent the wheel. Use blanket clearances only when appropriate. Use administrative data (no PRA) if relevant. HEY OMB: Raise the arbitrary # of “10 or more” to something

statistically defensible (30?) and fast track PRA or waive it when collecting data on the effectiveness of Federal grants.

11

Q1: Legal and Regulatory Obstacles•Obstacle 4: Fixed price contract requirements.

OMB memorandum, March 4, 2009 on government contracts. Shrinking discretionary budgets: Agencies may be unrealistic

about how much should be spent on an evaluation. Agencies may not want to demonstrate the need for cost-

reimbursement contracts to OMB or management. There may be real or imagined pressure from OMB or agency

management to drive down costs using fixed-price contracts. Fixed-price may result in unrealistic bids just to win, so… the evaluation methodology may be inadequate to properly

address the evaluation questions.

•Solution 5: See Solution #4 re: Realistic discussions about what is doable. See Question #2 re: REOIs and Q&A. See Nick Hart’s presentation about flexibility.

12

Q1: Legal and Regulatory Obstacles•Obstacle 5: Perspective of procurement and

contract offices was not represented on our panel.

•Solution 6:

For a truly effective dialogue on evaluation contracting, representatives from procurement offices and additional individuals trained as contract officer technical representatives (COTRs) should be included.

More Federal agency evaluator perspectives would be useful.

The 2017 AEA conference in Washington might allow additional participation of these perspectives.

13

Audience and Discussant

Name one legal and/or regulatory obstacle that has not been discussed that can affect the quality of contracted evaluations.

What are the potential solutions?

14

Question 2

2. Do Requests for Expressions of Interest and question and answer processes improve the quality of evaluation Requests for Proposals (RFPs)?

15

Q2: REOIs and Q&AObstacles:

•REOIs and RFP Q&As have the potential to improve the scopes of work, and ensure that evaluation RFPs are realistic, but use may be infrequent in some agencies.

•Contractors may be wary of revealing that they are bidding or informing the competition regarding possible technical approaches.

•Separating the design phase from the conduct of an evaluation may be useful BUT:

• It may prevent the best qualified party from bidding because of perceived conflict of interest.

16

Q2: REOIs and Q&ASolutions:

•Checklists (3ie, Western Michigan University) can help focus on evaluation quality and what should be included.

•Both Federal evaluators and contractors may benefit from incorporating these into their work.

•All parties want to improve RFPs, BUT the habits/ requirements of agency contracting offices may lead to less effective RFPs, even if the practices are meant to help ensure fairness and legal standards.

17

Audience and Discussant

Do Requests for Expressions of Interest and question and answer processes improve the quality of evaluation Requests for Proposals (RFPs)?

Question 3

3. How do government estimates of level of effort (or lack thereof) and time frames influence evaluation budgets and the conduct of evaluations?

18

Q3: Levels of Effort and Timeframes

19

Obstacles:

•Contractors need more guidance from agencies on realistic levels of effort.

•Some agencies refuse to estimate level of effort.

•Perhaps contracting offices may believe this information drives up the cost of contracted work.

•Contractors feel just the opposite, since realistic levels of effort can help determine if it is realistic to bid.

Q3: Level of Effort and Timeframes

20

Solutions:

•Training: Agencies may lack trained staff with experience or lack in-house evaluation expertise. Such expertise will lead to stronger RFPs and result in more useful, less wasteful evaluation results.

•Best Value: The “best value” approach may be useful, with agencies selecting the lowest bidder of those achieving threshold scores on technical proposals.

•This is fine as long as the threshold is realistic, and does not circumvent requirements for genuine competition.

Q3: Level of Effort and Timeframes

21

Solutions:

•Saving $: Is the government actually saving money by going with lowest bids? What about add-ons or agencies accepting lower quality methods because that is all they can afford.

•Gaming: Could be reduced if agencies provided

approximate amounts available for work. Proposals would be more realistic about what can be done for the $. Overall, gaming would be reduced, and proposal quality could improve/reflect reality.

Q3: Level of Effort and Timeframes

22

Solutions:

• Contractors can use “optional” tasks to fill the gap between responding to the letter of the Scope of Work and what would produce higher quality evaluations. However, it is not clear if this approach works or not.

• More transparency in the process is needed.

• Perhaps GAO could do a study on the amount of waste or contract modifications related to different contracting vehicles and approaches. More data could inform deliberations BUT:

• Need member(s) of Congress to express interest in such a study for GAO to take it up.

23

Audience and Discussant

How do government estimates of level of effort (or lack thereof) and time frames influence evaluation budgets and the conduct of evaluations?

24

Question 4

4. How do contractors decide to bid or not?

Do certain practices discourage bidding?

25

Q4: To Bid or Not to BidObstacles:

• Incumbents often appear to have an advantage.

•While illegal wiring of contracts may be limited, proposal criteria are often stacked towards rehiring of an incumbent by:

Reducing the number of years that recent experience and references can be used.

Creating levels of specificity for staff qualifications that can only be met by an incumbent.

Shortening the proposal time to such a degree that only a contractor currently doing the work could quickly and easily pull together a relevant proposal.

26

Q4: To Bid or Not to BidSolution:

• Increased but realistic debriefings when a bid is not accepted would be useful, BUT…

•only if people included in the debriefing can provide a big picture view of where the problems are, and not just nitpick AND…

• if the focus is genuinely on improving future bids and not just going through due process motions.

• Increased use of competitive task order contracts that allow for continued competition between pre-qualified vendors.

27

Question 5

5. What are the pros and cons of performance-based contracting?

Is it possible or desirable for contracting evaluation services?

28

Q5: Performance-based Contracts

Obstacle:

•Performance-based contracts might be subjective and may focus on less relevant issues such as strict adherence to timeliness and typos.

Solution:

•While criteria are included in RFPs, agency staff should review the criteria with contractors to ensure common understanding and benchmarks to be used to rate performance.

29

Audience and Discussant

What are the pros and cons of performance-based contracting?

Is it possible or desirable for contracting evaluation services?

30

Audience and Discussant

One more question:

What are other obstacles and solutions to producing high quality contracted federal government evaluations?

31

Contact InformationDavid J. Bernstein, Ph.D.Senior Study Director, Westatdavidbernstein@westat.com(301) 738-3520@DJBernstein on Twitter

Kathryn E. Newcomer, Ph.D.Director, Trachtenberg School of Public Policy and

Administration, The George Washington Universitynewcomer@gwu.edu(202) 994-6295

32

Want a Copy of the Presentation?

Visit the Washington Evaluators website at:http://washingtonevaluators.org/event-materials

If you are not a Washington Evaluators member, please consider joining to stay well informed about evaluation issues in the DC area:

http://washingtonevaluators.org/membership