Internal Audit Report Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope February 62015 A defined set of control objectives was utilized to focus on operational

Internal Audit Report

Professional Engineering Procurement Services (PEPS) Contract and Work Authorizations TxDOT Office of Internal Audit

PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope

February 2015 2

Objective To determine the effectiveness and efficiency of the Professional Engineering Procurement

Services (PEPS) consultant selection and work authorization process.

Opinion Based on the audit scope areas reviewed, control mechanisms are effective and

substantially address risk factors and exposures considered significant relative to impacting

operational execution and compliance. The organization's system of internal controls

provides reasonable assurance that most key goals and objectives will be achieved despite

significant control gap corrections and improvement opportunities identified. Control gap

corrections and improvement opportunities identified are likely to impact the achievement

of the organization's business/control objectives, but management has agreed to corrective

action plans to address the relevant risks within 6 months.

Overall Engagement Assessment Satisfactory

Findings

Title Control Design

Operating

Effectiveness Rating

Finding 1 Provider Evaluation and Selection

Process x x Needs Improvement

Finding 2 Statement of Qualification

Screening Process x x Needs Improvement

Finding 3 Work Authorization Assignment x x Satisfactory

Management concurs with the above findings and prepared management action plans to

address deficiencies.

Control Environment Both PEPS management and staff recognize the division’s importance and the overall

impact it has on the department’s core business functions. They are also aware of the risks

inherent in procuring TxDOT’s professional engineering services contracts and work

authorizations. Management strives to reduce risks and implement controls necessary to

ensure the success of the division and TxDOT. However, concerns over limited rationale and

documentation to support decisions in the screening and scoring process lead to an

increased need for additional management monitoring and oversight controls. While

exceptions were noted in the screening and scoring process, PEPS management has

designed a robust process that provides assurance that program objectives are being met

across the state. Exceptions identified in this report did not have an impact in PEPS

division’s selection of the most qualified providers, but likelihood of a negative impact could

increase if not corrected.


February 2015 3

Summary Results

Finding Scope Area Evidence

1 Selection

Process

Scoring

Testing included reviewing 18 score sheets from 10 Request

for Qualifications (RFQs) to evaluate if the evaluation

component of the score sheet was properly calculated. The

score sheets include an evaluation component, which generally

makes up 5% of the total score.

3 of 18 (17%) provider score sheets tested did not include

both the firm and project manager scores within the

Evaluation Score Average (ESA) calculation, as required.

These exceptions did not result in erroneous selection of

provider and were isolated to one service center.

Annual Prime Provider Evaluations

Prime provider evaluations are required to be performed at

least annually in the Consultant Certification Information

System (CCIS). These evaluations are then used to calculate a

portion (generally 5%) of the prime provider’s score. If an

evaluation is not completed in CCIS, a provider is given

maximum points in this category.

19 of 30 (63%) completed contracts tested from Fiscal

Year (FY) 2011 through FY 2013 did not have an annual

firm evaluation or evaluation performed at the completion

of the contract in CCIS, as required. Without an evaluation

in CCIS, PEPS is unable to determine the adequacy of the

provider’s past performance and by default, would be given

maximum points in this category if the provider applied for

a new contract.

2 Selection

Process

Statement of Qualifications (SOQ) Screening

Testing included reviewing 10 Request for Qualifications

(RFQs) and 367 screening checklists used to determine pre-

qualification of each provider to be a consultant for a PEPS

contract.

25 of 367 (7%) SOQ screening checklists did not include

final determinations (i.e. “meets” or “does not meet”

decision).

o 13 of 342 (4%) SOQ screening checklists with a final

determination had no supporting evidence for that

decision and the checklist was left blank.

Other exceptions identified, which related to

inconsistencies in the completion of the SOQ screening

checklist, primarily stemmed from two of seven PEPS

service centers.


February 2015 4

3 Work

Authorizations

Multiple Work Authorization Assignments

Individual work authorizations under $1 million (M) had

characteristics that in aggregate would require approval higher

than a Division Director. Work authorizations under $1M

completed by the districts/divisions (without additional

signature authorization) made up 40% of the entire amount of

work authorizations.

13 of 826 (2%) work authorizations reviewed were issued

by four divisions to their same vendors, under the same

contracts, either on the same day or within one week of

each other, that totaled over $1M. Work authorizations

over $1M require approval from the Chief Procurement

Officer or TxDOT Administration. The work authorizations, in

aggregate, issued by each of these divisions totaled over

$1M which would exceed the delegation of authority limit.

This was referred to the Office of Compliance, Ethics, and

Investigations (CEI) for further review, and no inappropriate

activities were identified.

Audit Scope The scope of the audit work focused on the organizational environment, screening and

selection process for contractors, and work authorization controls. These activities were

reviewed to assess the effectiveness and efficiency of the contracting and work

authorization processes.

The scope for audit fieldwork conducted included Request for Qualifications (RFQs) and

resulting contracts executed from October 1, 2013 through July 7, 2014; tests were

performed on sample periods and dollar amounts of work authorizations initiated.

Evaluations were tested from Fiscal Year (FY) 2011 to FY 2013.

The audit was performed by Alma Alvarez, Jessica Esqueda, David Kossa, and Karen Henry

(Engagement Lead). The audit was conducted during the period from June 11, 2014 to

October 8, 2014.

Methodology The methodologies used to complete the objectives of this audit included:

Reviewing TxDOT internal documents, including policy and procedure manuals,

organization charts, process maps, management reports, and training records

Reviewing state codes and manuals

Inquiring and interviewing of key personnel

Reviewing prior audit reports from TxDOT’s Office of Internal Audit and the State

Auditor’s Office

Evaluating PEPS selection process and work authorizations process

Conducting data analysis, obtaining supporting documentation, and sample testing

Reviewing communication and management philosophy of overall organizational tone

Performing an overall risk assessment of the professional engineering procurement

services and work authorization process functions


February 2015 5

Reviewing screening determination documents and if available, selection rationale to

ensure appropriateness

These procedures were applied as necessary to perform the audit fieldwork.

Background This report is prepared for the Texas Transportation Commission and for the administration

and management of TxDOT. The report presents the results of the Professional Engineering

Procurement Services (PEPS) Contract and Work Authorizations audit which was conducted

as part of the Fiscal Year 2014 Audit Plan.

PEPS was formed in October 2013 as a redesigned program for procuring engineering,

architectural, and surveying services under consultant contracts. PEPS was redesigned to

reduce the time to process contracts and provide quality selection outcomes. Prior to the

PEPS formation, the procurement for professional engineering services was under the

consultant contracting program.

PEPS has six divisional service centers in the TxDOT metro and urban districts (Austin,

Dallas, El Paso, Houston, Fort Worth, and San Antonio) and a central office that supports the

additional 19 TxDOT districts and the divisions. PEPS oversees the selection process for

engineering, surveying, and architectural contracts to ensure providers are selected based

solely on their qualifications. PEPS executed 137 contracts totaling over $343 million from

inception (October 2013) through August 2014.

There are several processes that encompass the selection process. Once it is determined

that there is a need for a PEPS contract, a Consultant Selection Team (CST) is assembled.

The team initiates the provider selection process by assembling and submitting “Intent to

Contract”. After developing and posting a Request for Qualifications (RFQ) to advertise the

contract, responses from interested providers called Statements of Qualifications (SOQs) are

collected and evaluated by the CST. Depending on the selection process, providers are

either selected or advanced to a short list for an interview, final evaluation, and selection.

PEPS is also responsible for providing general oversight, guidance, and support to districts

and divisions for work authorizations.

We conducted this performance audit in accordance with Generally Accepted Government

Auditing Standards and in conformance with the International Standards for the

Professional Practice of Internal Auditing. Those standards require that we plan and perform

the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our

findings and conclusions based on our audit objectives. Recommendations to mitigate risks

identified were provided to management during the engagement to assist in the formulation

of the management action plans included in this report. We believe that the evidence

obtained provides a reasonable basis for our findings and conclusions based on our audit

objectives. The Office of Internal Audit transitioned to Committee of Sponsoring

Organizations of the Treadway Commission (COSO) Internal Control – Integrated Framework

version 2013 in December 2013.


February 2015 6

A defined set of control objectives was utilized to focus on operational and compliance goals

for the identified scope areas. Our audit opinion is an assessment of the health of the

overall control environment based on (1) the effectiveness of the enterprise risk

management activities throughout the audit period, and (2) the degree to which the defined

control objectives were being met. Our audit opinion is not a guarantee against operational

sub-optimization or non-compliance, particularly in areas not included in the scope of this

audit.

Best Practices PEPS has established contracting performance measures that evaluate the time it takes for

a contract to be signed starting with the time the solicitation occurs (e. g., this includes the

review of the SOQs, RFQs, and scoring of the provider). PEPS personnel review the

performance measures on a frequent basis and monitor their performance. When internal

performance measures are not being met, PEPS management, including the service center

managers, Administration and Finance Manager, Center of Excellence Manager, and the

PEPS Director review and discuss what has caused the performance variance. The

frequency and type of reviews performed are detailed and include planning, coordination,

strategic considerations, and problem solving. The reviews are performed to help ensure

efficiency in the processes that have been established.


February 2015 7

Detailed Findings and Management Action Plans (MAP)

Finding No. 1: Provider Evaluation and Selection Process

Condition

The processes in place for provider evaluations and selections are not consistently applied.

The Evaluation Score Average (ESA) used to calculate a portion (generally 5%) of the prime

providers’ score was not being calculated correctly by one of the service centers. In addition,

annual evaluations of providers were not always completed.

Rationale to support scoring and score refinement on deliberations were not included on

score sheets.

Effect/Potential Impact

Inconsistent application of evaluations and limited or missing rationale to support decisions

in the scoring process can raise concerns over the objectivity, consistency, and accuracy of

TxDOT’s selection process.

Criteria

Title 43, Texas Administrative Code, Part 1, Section 9.41(d)(1), states that evaluations are to

be conducted during the ongoing activity and at the completion of the contract. In addition,

the TxDOT Provider Evaluation Guide and CTR615 (formerly DES615)–Consultant

Management and Administration Module 5 of the Consultant Selection Team (CST), whose

member’s required training, state that the minimum time frame for performing provider

evaluations is annually.

The PEPS Contracting: Selection Process Manual also requires Consultant Selection Team

(CST) members to take TxDOT’s CTR615 (formerly DES615)–Consultant Management and

Administration training course to understand their roles and responsibilities.

The Professional Engineering Procurement Services (PEPS) Contracting: Selection Process

Manual includes policies and procedures for application of ESA in the scoring process which

requires the use of project manager score and firm score in the calculation.

The State of Texas Contract Management Guide Appendix 5 – Evaluation Team Guidelines

states that evaluators should write comments during the de-briefing process, especially if

the score is unusually low or high.

Cause

There is no automated process in the Consultant Certification Information System (CCIS) to

track evaluation status or to ensure evaluation completion. There is no review process or

monitoring controls to ensure rationale supports the provider selection or that selection files

are complete and include all key components to ensure compliance requirements are met

and that ESA scores are calculated correctly. In addition, not everyone involved in the

selection process is attending required training that provides expectations and guidance on

how to conduct the selection process.


February 2015 8

Evidence

The evidence obtained in the review included:

Scoring

Testing included reviewing the score sheets for 10 Request for Qualifications (RFQs). These

score sheets provide the support for why a provider is selected for a PEPS contract:

10 of 10 (100%) RFQs tested did not have rationale included with score sheets to

support scoring and score refinement on deliberations

o Auditors could not determine if the objectivity and accuracy of selections were

impaired

18 score sheets were selected from the 10 RFQs to determine if the evaluation

component of the scoring sheet was properly calculated:

o 3 of 18 (17%) provider score sheets tested did not include both the firm and

project manager scores within the ESA calculation as required;

These exceptions did not result in erroneous selection of providers

Annual Prime Provider Evaluations

Prime provider evaluations are required to be performed at least annually in CCIS. These

evaluations are used to calculate a portion (generally 5%) of the prime provider’s score. If an

evaluation is not completed in CCIS, a provider is given maximum points (generally 5%) in

this category.

A sample of 30 completed contracts was selected and tested to review if these contracts

had the required annual prime provider evaluations.

19 of 30 (63%) completed contracts tested from Fiscal Year (FY) 2011 through FY

2013 did not have an annual firm evaluation or evaluation performed at the

completion of the contract in CCIS, as required. Without an evaluation in CCIS, PEPS

is unable to determine adequacy of a provider’s past performance and the provider is

given maximum points in this category when applying for a new contract

Training

Training was not taken by all those that participated in the selection process:

10 of 36 (28%) Consultant Selection Team (CST) members reviewed did not attend

required CTR615 (formerly DES615)–Consultant Management and Administration

training in a 5 year period

Management Action Plan (MAP):

MAP Owners:

Lorena Echeverria de Misi, PEPS Division Director

Lucio Vasquez, PEPS Deputy Division Director

Camille Thomason, PEPS Center of Excellence Manager

MAP 1.1:

Management agrees with the findings and is committed to ensuring the provider evaluation

process used in the selection process is performed in accordance with the Professional

Engineering Procurement Services (PEPS) procedures. The following tasks will be performed

during this period:


February 2015 9

A central server is currently under development to provide the division with one main

location to store and retrieve documents. At this time, PEPS Service Centers are

working on individual servers as per the previous Region model.

PEPS will revisit the procedures for the scoring process in the PEPS Contracting:

Selection Process Manual to ensure it fits within best-practices in consultant

selection, making revisions, if needed.

PEPS will work with NTT Data to create an automated mechanism to alert appropriate

individuals on actions required for performing provider evaluations.

On verifying whether the Consultant Selection Team (CST) members have taken

required courses: a new procedure will be included in the PEPS Contracting:

Selection Process Manual to have contract specialists perform verification of CST

member’s training through the Professional Services Contract Administration

Management System (PS-CAMS).

MAP 1.2:

PEPS will finalize the PEPS Contracting: Contract Management and Administration Manual.

Language concerning the provider evaluation procedure will be included.

Completion Date:

May 15, 2015


February 2015 10

Finding No. 2: Statement of Qualification (SOQ) Screening Process

Condition

Statement of Qualification (SOQ) screening process checklists, which are used in

determining if consultants are qualified, were incomplete and completed by more than one

person. In addition, signatures indicating review and approval of the SOQ screening

checklists were not consistently retained, documented, or supported.


Inconsistencies in the completion of SOQ screening checklists and limited or missing

rationale to support decisions in the screening process can raise concerns over the

objectivity of TxDOT’s selection process. This can also affect the overall outcome and the

most qualified consultant may not be selected.

Criteria

The PEPS Contracting: Selection Process Manual Chapter 8 Section 4 states that one person

should conduct all screenings to help ensure consistency, SOQs are to be screened using

the SOQ screening checklist, and no checklist items may be deleted.

In addition, the TxDOT Solicitation Package (SP) Screening Guidance and Instructions

Checklist for final determination requires:

Each item in the checklist to be marked whether or not the SP meets the

requirement

If an item is marked as “Does Not Meet,” the situation is to be described in the

comments section

If an item is not applicable to the solicitation, mark “N/A”

The PEPS representative must review, sign, and date each completed checklist

The PEPS Contracting: Selection Process Manual also requires Consultant Selection Team

(CST) members to take TxDOT’s CTR615 (formerly DES615)–Consultant Management and

Administration training course to understand their roles and responsibilities.

Cause

There is no review process or monitoring controls to ensure supporting evidence supports

decisions and finalization of screening determinations is complete and includes all key

components to ensure compliance requirements are met. In addition, not everyone involved

in the screening selection process is attending required training.

Evidence


Screening

Testing included reviewing 10 Request for Qualifications (RFQs) and 367 screening

checklists from those RFQs to determine pre-qualification of each provider to be a

consultant for a PEPS contract:

7 of 10 (70%) RFQs had more than one person conducting the SOQ screenings


February 2015 11

132 of 367 (36%) SOQ screening checklists did not have signature approval

documented

106 of 367 (29%) SOQ screening checklists did not indicate if the provider met the

indicated requirement even though comments were provided

25 of 367 (7%) SOQ screening checklists had missing final determinations (“meets”

or “does not meet” decision)

o 13 of 342 (4%) SOQ screening checklists, with a final determination, had no

supporting evidence for that decision and the checklist was left blank. All 13 of

these checklists were performed by the same individual

From the 276 total SOQ screening checklist exceptions noted in the bullet points above, 111

(40%) exceptions were from one service center, and 98 (36%) exceptions were from another

service center.

Auditors did not identify any providers being selected that were not qualified.

Training

Training was not taken by all those that participated in the screening process:

10 of 36 (28%) Consultant Selection Team (CST) members reviewed did not attend

required CTR615 (formerly DES615)–Consultant Management and Administration

training in a 5 year period


MAP Owners:




MAP 2.1:

Management agrees with the findings and is committed to ensuring the SOQ screening

process is performed in accordance with the PEPS Contracting: Selection Process Manual.

The following tasks will be performed during this period:

PEPS will revisit the procedures for the SOQ screening in the PEPS Contracting:

Selection Process Manual to ensure it fits within best-practices in consultant

selection, making revisions, if needed.

PEPS Management will provide guidance to division to promote adherence to the

division’s manuals.

On verifying whether CST members have taken required courses: a new procedure

will be included in the PEPS Contracting: Selection Process Manual to have contract

specialists perform verification of CST member’s training through the Professional

Services Contract Administration Management System (PS-CAMS).

Completion Date:

May 15, 2015


February 2015 12

Finding No. 3: Work Authorization Assignment

Condition

Individual work authorizations under $1 million (M) have characteristics that, in aggregate,

would require approval higher than a Division Director. Four divisions individually issued

work authorizations to the same vendors, under the same contract, on the same day or

within one week of each other that totaled over $1M which would exceed the delegation of

authority limit and require additional administration approval.


Work authorizations may not have the necessary review and oversight to ensure work scope

and amounts are appropriate and in accordance with signature authority roles.

Criteria

Based on the Delegations of Signature Authority, Division Directors and District Engineers

can execute work authorizations of $1M or less. Work authorizations over $1M require

additional approval by the Chief Procurement Officer or TxDOT Administration.

Cause

Monitoring controls do not exist to detect whether Districts/Divisions/Offices (D/D/O)

are issuing multiple work authorizations that are under the $1M threshold to

circumvent current rules.

Training on how to properly utilize work authorizations is not provided to D/D/O staff

that issue work authorizations under $1M.

The Professional Engineering Procurement Service (PEPS) Division Contracting:

Contract Management and Administration Manual, which includes work

authorizations, is still in development.

Evidence


Multiple Work Authorization Assignments

13 of 826 (2%) work authorizations analyzed were issued by four divisions to their same

vendors, under the same contracts, either on the same day or within one week of each other

that totaled over $1M. This was referred to the Office of Compliance, Ethics, and

Investigations (CEI) for further review, and no inappropriate activities were identified:

On 3/25/2014, 6 work authorizations were executed by one division to one provider

under the same contract totaling over $4.2M

On 3/28/2014 and 3/31/2014, 3 work authorizations were executed by one

division to one provider under the same contract totaling over $1.7M






February 2015 13


MAP Owners:




MAP 3.1:

PEPS will communicate the procedure for work authorization prioritization and signature

authority compliance rules to the Districts/Divisions/Offices (D/D/O).

MAP 3.2:

PEPS will include signature authority reviews by service center managers as part of

the current review process that is performed over work authorizations.

PEPS will finalize the PEPS Contracting: Contract Management and Administration

Manual to provide Districts/Divisions/Offices (D/D/O) guidance on best practices for

work authorization assignment and development.

Completion Date:

March 15, 2015


February 2015 14

Observations and Recommendations

Audit Observation (a): Records Management

Missing records and documentation were noted throughout the audit testing (highlighting

areas for improvement over records management) including the following:

10 of 10 (100%) Request for Qualifications (RFQs) reviewed did not have deliberation

meeting minutes/notes available for review

6 of 9 (67%) Consultant Selection Team (CST) members interviewed indicated that

the records retention requirements were not communicated to them

4 of 27 (15%) CST members indicated having a second job (outside TxDOT); however,

there were no Outside Employment/Professional Activity Request Forms (Form

2490s) on file for any of these employees for Professional Engineering Procurement

Services (PEPS) Division to assess if a potential conflict of interest is present with

their TxDOT evaluation responsibilities.

1 of 10 (10%) work authorizations tested was entered into the wrong contract

number in the Professional Services Contract Administration Management System

(PS-CAMS). In addition, the system does not retain an audit trail to track movement

or deletion of work authorizations


Not communicating record retention requirements to all staff involved in the screening and

scoring process, inability to provide documents subject to open records request, and not

obtaining CST member conflict of interest information could leave TxDOT susceptible to

compliance issues and can raise concerns over the objectivity of TxDOT’s selection process.

Work authorizations entered under the wrong contract can cause encumbrances against

other Divisions/Districts/Office (D/D/O) contracts preventing them from issuing work

authorizations needed on the same contract. In addition, allowing movement of work

authorizations between contracts or deletion from PS-CAMS without an audit trail to track

movement creates a risk for potential fraud, waste, or abuse.

Audit Recommendation

Professional Engineering Procurement Services (PEPS) Division should:

Implement a time-frame requirement to have selection files completed and closed

with all files uploaded to a central controlled location

Establish a file close-out checklist to be completed by the PEPS representative (lead)

Periodic management review should be implemented and completed (with a required

completed review checklist) to ensure all required documents are being retained and

available upon request

As part of the CST team kickoff meeting, Form 2490 should be obtained/retained for

all employees having a second job (outside TxDOT), and records retention

requirements should be communicated to all staff serving on the CST team. These

should be included as part of the planning checklist the PEPS representative (lead)

should complete in the planning phase prior to the screening process

Implement system programming edit checks or additional monitoring and review to

prevent work authorizations being entered under the wrong contract or

moved/deleted without a tracking mechanism


February 2015 15

Summary Results Based on Enterprise Risk Management Framework

Closing Comments The results of this audit were discussed with the Professional Engineering Procurement

Services (PEPS) Division Director and Deputy Director on October 27, 2014. We appreciate

the assistance and cooperation received from the PEPS, Human Resources, and Contract

Services Divisions; Office of Compliance, Ethics, and Investigations; and personnel in the

districts contacted during this audit.

Documents

Internal Audit Report Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope February 62015 A defined set of control objectives was utilized to focus on operational