Bond University Research Repository Evaluation of design

Bond UniversityResearch Repository

Evaluation of design-builder qualifications through the analysis of requests for qualifications

Xia, Bo; Skitmore, Martin; Zuo, Jian

Published in:Journal of Management in Engineering

DOI:10.1061/(ASCE)ME.1943-5479.0000095

Licence:Other

Link to output in Bond University research repository.

Recommended citation(APA):Xia, B., Skitmore, M., & Zuo, J. (2012). Evaluation of design-builder qualifications through the analysis ofrequests for qualifications. Journal of Management in Engineering, 28(3), 348-351.https://doi.org/10.1061/(ASCE)ME.1943-5479.0000095

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

For more information, or if you believe that this document breaches copyright, please contact the Bond University research repositorycoordinator.

Download date: 16 Mar 2022

https://doi.org/10.1061/(ASCE)ME.1943-5479.0000095

https://research.bond.edu.au/en/publications/d8eebc0f-8f21-4c2b-8879-0a3c3a83fcc6


This may be the author’s version of a work that was submitted/acceptedfor publication in the following source:

Xia, Paul, Skitmore, Martin, & Zuo, Jian(2012)Evaluation of design-builder qualifications through the analysis of requestsfor qualifications.Journal of Management in Engineering - ASCE, 28(3), pp. 348-351.

This file was downloaded from: https://eprints.qut.edu.au/48624/

c© Consult author(s) regarding copyright matters

This work is covered by copyright. Unless the document is being made available under aCreative Commons Licence, you must assume that re-use is limited to personal use andthat permission from the copyright owner must be obtained for all other uses. If the docu-ment is available under a Creative Commons License (or other specified license) then referto the Licence for details of permitted re-use. It is a condition of access that users recog-nise and abide by the legal requirements associated with these rights. If you believe thatthis work infringes copyright please provide details by email to [email protected]

Notice: Please note that this document may not be the Version of Record(i.e. published version) of the work. Author manuscript versions (as Sub-mitted for peer review or as Accepted for publication after peer review) canbe identified by an absence of publisher branding and/or typeset appear-ance. If there is any doubt, please refer to the published source.


https://eprints.qut.edu.au/view/person/Xia,_Bo.html

https://eprints.qut.edu.au/view/person/Skitmore,_Martin.html

https://eprints.qut.edu.au/48624/


Accep

ted M

anus

cript

Not Cop

yedit

ed

1

Evaluation of design-builder qualifications through the

analysis of Requests for Qualifications (RFQs)

Bo Xia1, Martin Skitmore

2, Jian Zuo

3

1 Lecturer, School of Urban Development, Queensland University of Technology, Garden

Point Campus, 2 George Street, Brisbane QLD 4000, Australia.

Email: [email protected]

2 Professor, School of Urban Development, Queensland University of Technology, Garden

Point Campus, 2 George Street, Brisbane QLD 4000, Australia.


3 Lecturer, School of Natural and Built Environments, University of South Australia, City

East Campus, GPO Box 2471 Adelaide SA 5001, Australia


ABSTRACT

The two-phase best-value process has been widely used by public agencies for Design

and Build (DB) procurement, with a key issue in the first phase of pre-qualification

being the determination of evaluation criteria. This study identified a set of general

qualification criteria for design-builders and compares their relative importance by a

thorough content analysis of 97 Requests for Qualification (RFQ) for public DB

projects advertised between 2000 and 2011 in various regions of the USA. The thirty-

Journal of Management in Engineering. Submitted May 23, 2011; accepted August 23, 2011; posted ahead of print August 25, 2011. doi:10.1061/(ASCE)ME.1943-5479.0000095

Copyright 2011 by the American Society of Civil Engineers

Accep

ted M

anus

cript

Not Cop

yedit

ed

2

nine qualification criteria found are summarized and classified into eight categories

comprising: experience; project understanding and approach; organizational structure

and capacity; past performance record; professional qualifications; responsiveness to

RFQs, office location and familiarity with local environment; and legal status in

descending order of their relative importance. A comparative analysis of different

types of projects shows that the relative weightings of the qualification criteria vary

according to different characteristics of the DB projects involved.

Key words:

Design/Build, content analysis, request for qualification

INTRODUCTION

Design-build is an alternative delivery method where one entity or consortium is

contractually responsible for both design and construction, and has been gaining in

popularity in public sectors in recent years (Hale et al. 2009; Park et al. 2009; Rosner

et al. 2009). In order to find the right person for the right job, the best-value

procurement process has been widely used by public agencies (Jackson 2010).

Specifically, the preferred approach is a two-phase process where the first phase

consist of an evaluation of bidders’ qualifications and the second phase involves the

technical and price submission of a short-listed group of bidders. According to

Molenaar et al. (1999), the two-phase best-value method outlined in the 1996 Federal

Acquisition Reform Act delivers the best overall budget and schedule performance.



Accep

ted M

anus

cript

Not Cop

yedit

ed

3

With the two-phase best-value procurement method, considerable attention is placed

on the issue of DB RFQs. In particular, in order to solicit design-builders with

necessary and desirable qualifications, owners need to establish clear, objective

selection criteria to measure and determine the potential design-builders to be invited

to prepare and submit a design-build proposal. According to Beard et al. (2001), the

determination of pre-qualification selection criteria is a key issue that an owner must

consider when evaluating prospective DB contractors. When the selection criteria and

process is sufficiently transparent to all the interested DB contractors, it is more likely

that the selected qualified design-builders will adequately address the owner’s

requirements and expectations. However, for many owners the formulation of

selection criteria is not an easy task. Owners should carefully define and summarize

the evaluation criteria for specific types and sizes of projects as every design-builder

has unique qualifications. Otherwise, design-builders that are better qualified may not

be selected as the selection criteria do not address their advantages.

In order to facilitate the determination of selection criteria in RFQ, this paper presents

an examination of key qualification criteria in 97 design-build RFQs collected across

the U.S.. Unlike other opinion-based research studies, the analysis of different sets of

selection criteria and their weightings in RFQs reflect the perceptions and real

practices of owners in the selection of design-builders. Although the determination of

qualification criteria is done separately for each project, it is clear that the analysis of



Accep

ted M

anus

cript

Not Cop

yedit

ed

4

a general set of criteria will not only demonstrate owners’ understanding of qualified

design-builders but also reflect their philosophies of DB practices in general.

RESEARCH METHODOLOGY

Content analysis was adopted to identify the qualification factors and their importance

weighting adopted by public owners in DB RFQs. The content analysis is defined as

the systematic, objective, quantitative analysis of message characteristics (Neuendorf

2002). It is an observational and power data reduction technique for dealing with large

volumes of data (Stemler 2001) to summarize any form of content by counting the

number of times an activity happens or a topic is depicted (Kolbe and Burnett 1991).

The first step in conducting a content analysis is to identify the materials to be

analyzed (Guthrie et al. 2004). A sizable sample of actual DB RFQs was collected

from those posted online by a variety of public sectors in the U.S.. Table 1 illustrates

that the sample consists of 97 RFQs from 33 States and with a total contract value of

over $12.1 billion, the majority (85%) being advertised in the past 5 years. The RFQs

cover a fairly wide range of project types and allow an examination of the most

current approaches to DB project delivery.

Please insert Table <1> here

The second step is to determine whether the form of content analysis to be used is to

be qualitative or quantitative (Fellows and Liu 2008). In a qualitative content analysis,



Accep

ted M

anus

cript

Not Cop

yedit

ed

5

emphasis is on determining the meaning of the data (i.e. grouping into categories).

Quantitative content analysis extends this to generate numerical values of the

categorized data (frequencies, ratings, ranking, etc) which may be subject to statistical

analyses. Both qualitative and quantitative methods were adopted in the current study.

The qualitative method was first used to record, examine, and consolidate different

expressions of qualification criteria and then group them into themes or categories

according to their meanings. Then, the overall frequency of each criterion and

category throughout the entire collection of DB RFQs was examined for its popularity.

The importance weightings of the qualification categories and their average points in

RFQs were also calculated to indicate the different preferences of the DB owners

involved.

RESULTS

Thirty-nine (39) qualification factors in different expressions were identified and

recorded through the content analysis. Those conveying the same meanings were

combined and rephrased, and 18 qualification criteria were finally consolidated. The

cluster of themes of these factors were then summarized and organized based on their

meanings. Finally, eight (8) categories of the qualification criteria were coded,

comprising:

1. experience

2. project understanding and approach

3. organizational structure and capacity



Accep

ted M

anus

cript

Not Cop

yedit

ed

6

4. past performance record

5. professional qualifications

6. office location and familiarity with local environment

7. legal status

8. responsiveness to RFQs.

In order to observe the popularity of these categories, the frequencies of the

qualification criteria and coded categories were calculated by dividing the number of

their appearance by the total number of RFQs. Table 2 indicates that, with frequencies

of more than 50%, experience, project understanding and approach, organizational

structure and capacity, and past performance record are the most frequently used

factors in evaluating design-builder qualification.


Of the 97 RFQs, 57 allowed the relative weightings of qualification categories to be

obtained and these were further examined to enable their prioritization. Of the 57

RFQs, 51 provide detailed weightings or score points to qualification categories, and

the relative weightings of these were simply recorded or calculated. The remaining 6

RFQs provide vague expressions such as “elements are listed in descending order of

importance, and “the factors listed are of equal significance”. For such expressions,

an algorithm introduced by Gransberg and Barton (2007) was adopted to convert the

expressions of keywords to importance weightings.



Accep

ted M

anus

cript

Not Cop

yedit

ed

7

Once all the importance weightings of qualification categories in each RFQ were

recorded, the relative importance for each category in the sample population was

obtained. Figure 1 indicates that experience is the most important criteria category

(accounting for 26% of the total weightings), followed by project understanding and

approach (24%) and organizational structure and capacity (20%).

Please insert Figure <1> here

For the next stage of the analysis, each RFQ received a total of 100 points to be

distributed among the qualification categories according to their stipulated importance

weightings. The average number of points for each category, which delivers a direct

message on the relative weighting of the category, was calculated by dividing the total

number of points by the number of RFQs in which it appears. Figure 2 shows the

results, with project understanding and approach having the highest points (31 points)

and the legal status the lowest (7 points).

Please insert Figure <2> here

The final stage of the analysis involved examining the relative importance of the

qualification categories within different project types. Although the number of RFQs

in each group is comparative small, the results in Table 3 convey an interesting

message concerning the owners’ view of DB qualifications. First, it is found that

categories of experience and office location and familiarity with local environment

have higher weights for commercial buildings than for other types. This may be due

to the fact that, for commercial buildings, owners consider a quick start-up and short



Accep

ted M

anus

cript

Not Cop

yedit

ed

8

duration to be the first priority, and it is advantageous to reach out immediately to a

sufficiently experienced design-builder. Additionally, the design-builders’ familiarity

with the local environment (e.g. regulations, practice) helps to speed up project

progress. Second, project understanding and approach is the most important

qualification category in heavy civil and highway projects, which is understandable as

most heavy civil and highway projects are large and technically challenging - a

thorough project understanding and suitable technical and management approaches

being needed to help for a successful project. Finally, renovation projects have the

greatest weight given to past performance record of all project types. As most

renovation projects require improvements being made to current status, it is likely that

owners pay extra attention to the past performance of potential design-builders in such

cases.


DISCUSSION

The results of the analysis demonstrate, experience, project understanding and

approach, organizational structure and capacity, past performance record and

professional qualifications are the major categories for pre-qualification, accounting

for 95% of the total importance weights. Responsiveness to RFQs, office location and

familiarity with local environment, and legal status comprise the remaining 5% of the

importance weighting.



Accep

ted M

anus

cript

Not Cop

yedit

ed

9

Experience of design-builders is the most important category for prequalification. A

design-builder with sufficient experience will invariably be considered responsible

enough to deliver DB projects. One limitation on the use of experience in judging DB

qualification, however, is that the experience criterion only focuses on the nature of

the contractor’s past work without considering the quality of the past experience. In

order to address this issue, owners also adopt the criterion of past performance for the

evaluation of DB qualification. The previous performance of design-builders may

include documented records of budget, schedule, quality and safety, statements of

customer satisfaction, and compliance with applicable laws and regulations. What is

needed is a design-builder with a satisfactory performance record in addition to

extensive DB experience.

The category of project understanding and approach examines how design-builders

understand owner’s goals and how they achieve these goals. When design-builders

have a clear understanding of the mandates, goals and design needs of the owner, it is

more likely that they will propose suitable techniques and management approaches. A

project approach contains practical solutions to the problems involved, which mainly

concern conceptual design, proposed construction techniques, and overall

management plan for quality, budget, schedule, safety and risk. This category is more

important for prequalification for larger and more technically challenging projects.



Accep

ted M

anus

cript

Not Cop

yedit

ed

10

Finally, design-builders should possess sufficient organizational capabilities and

relevant professional qualifications. Organizational capability is the major basis for

developing a sustainable competitive advantage, and enables organizations to fully

coordinate and utilize their resources. In the RFQ, owners use organizational structure,

sufficiency and stability of staff resources, skilled labor, equipment, and financial

resources as the major criteria to measure the organizational capability of design-

builders. The category of professional qualification requires DB contractors and the

team members to possess all the necessary licenses and certificates to carry out the

functions of their respective professions. In the selection of design-builders, only

those with both sufficient organizational capability and appropriate qualification

certificates are regarded as sufficiently able to execute DB projects.

The remaining categories, including the response to the RFQs, office location and

familiarity with local environment, and legal status, are comparatively less important

with small relative weightings (less than 3%). Some owners may use the “Pass/Fail”

method (with the proposer either meeting the criteria or not) with these criteria for the

prequalification/short-listing of design-builders. However, it should be noted that the

use of Pass/Fail criteria may create a situation where the most highly qualified

proposers could be eliminated by a clerical error rendering it nonresponsive or a

legitimate outstanding claim against the owner. Therefore, when drafting RFQs,

owners and consultants should seriously consider whether or not the emphasis on

responsiveness or the DB mentality to find the design-builder is worth the risk of



Accep

ted M

anus

cript

Not Cop

yedit

ed

11

eliminating the best qualified proposers. Additionally, owners should avoid including

an evaluation criterion that simply reflects their own preference, such as the office

location rather than a qualification requirement of prospective design-builders.

CONCLUSIONS

This paper identifies a general empirical set of qualification categories and their

relative importance for design-builders through a content analysis of a sample of 97

USA RFQs. Experience, project approach and organizational capability are the most

highly valued categories, with each of claiming more than 20% of the total

importance weighting. A comparative analysis of projects of different types shows

that owners place different emphasis on qualification categories according to the

different characteristics of the DB projects involved. Furthermore, the prioritized

qualification categories identified in this paper will facilitate the decision making

process for DB owners – a particularly difficult and important task in practice,

The research findings of this study also offer a number of practical implications. For

DB owners, especially inexperienced ones, it is desirable to clearly define the

evaluation factors of DB qualification in RFQs. This helps owners reduce the cost of

reviewing the proposals. Additionally, it is necessary to adjust the relative weightings

of the qualification criteria according to the different characteristics of the projects. It

encourages the more qualified design-builders to submit proposals for the projects.

Future work is needed to validate the research findings in view of the known



Accep

ted M

anus

cript

Not Cop

yedit

ed

12

subjectivity and possible bias involved in the content analysis. Additionally, a

guideline for pre-qualifications in two-steps best value procurement method could be

further developed based on the research findings in the current study,

REFERENCES

Beard, J.L., Loulakis SR. M. C., and Wundram E.C. (2001). Design-Build—Planning

through Development, McGraw-Hill, NY.

Fellows, R. and Liu, A. (2008). Research methods for construction (3rd Edition).

Blackwell Science, Oxford, United Kingdom.

Gransberg, D.D., Barton, R.F. (2007). “Analysis of federal design-build request for

proposal evaluation criteria.” Journal of Management in Engineering, 23 (2), 105-

111.

Guthrie, J., Petty, R., Yongvanich, K. and Ricceri, F. (2004). “Using content analysis

as a research method to inquire into intellectual capital reporting.” Journal of

Intellectual Capital, 5 (2), 282-93.

Hale, D.R., Shrestha, P.P, Gibson, G.E., and Migliaccio, G.C. (2009). “Empirical

comparison of design/build and design/bid/build project delivery methods.”

Journal of Construction Engineering and Management, 135(7), 579-587.

Jackson, B.J. (2010) Design-build Essentials, Delmar, NY.



Accep

ted M

anus

cript

Not Cop

yedit

ed

13

Kolbe, R.H., and Burnett, M.S. (1991). “Content analysis research: an examination of

applications with directives for improving research reliability and objectivity.”

Journal of Consumer Research, 18, 243-250.

Molenaar, K.R., Songer, A.D., and Barash, M. (1999). “Public-sector design-build

evaluation and performance.” Journal of Management in Engineering, 15 (2), 54-

62.

Neuendorf , K.A. (2002). The content analysis guidebook, Sage Publications, CA.

Park, M., Ji, S.H., Lee, H.S., and Kim, W. (2009). “Strategies for Design-Build in

Korea Using System Dynamics Modeling.” Journal of Construction Engineering

and Management, ASCE, 135 (11), 1125-1137.

Rosner, J. W., Thal, A. E., and West, C. J. (2009). “Analysis of the design-build

delivery method in Air Force construction projects.” Journal of Construction

Engineering and Management, 135, 710-717.

Stemler, S. (2001). “A review of content analysis.” Practical Assessment, Research

& Evaluation, 7 (17), Available on website:

http://pareonline.net/getvn.asp?v=7&n=17.



14

Table 1 Summary of the data sample

Project type RFQs

No. Owner agencies Locations

Commercial building 15 Local (County, Town, City, State)

government, U.S Army Corps of

Engineers, Naval Facilities

Engineering Command, U.S. Air

Force, Department of Defense,

Department of Veteran affairs,

National Aeronautics and Space

Administration, Federal Highway

Administration, Public schools,

Colleges and Universities

AK, AL, AZ, CA,

CO, CT, FL, GA,

IL, KS, MD,ME,

MI, MN, MO,

MS, MT, NJ, NM,

NV, NY, OH,

OK, RI, SC, SD,

TN, TX, UT, VA,

WA, WV, WI

Institutional building 40

Residential building 3

Heavy civil and

highway 15

Industrial and

processing 17

Renovation projects 7

Total 97 90 different owners 33 states

Accepted Manuscript Not Copyedited



15

Table 2 Coded categories of qualification factors

No. Criteria category Frequency (%)

1 Experience 83.51

Experience of design-builders with similar DB projects 58.76

Experience of personal assigned to the project 39.17

Specialized experience (value engineering, partnering, LEED, etc.) 24.74

2 Project understanding and approach 63.92

Understanding of the mandates, owner’s goals and needs. 17.53

Technical approach in design and construction techniques. 34.02

Management approach/plan for cost, quality, safety, risk, Small business

participation ,etc.

43.30

3 Organization structure and capacity 61.86

Organizational/team structures, organizational chart 32.99

Sufficiency of available staff resources 23.71

sufficiency of equipment to accomplish the work 8.25

Sufficiency and stability of financial resources 46.39

4 Past performance record 53.61

Past performance record on budget, schedule, quality, safety, customer

satisfaction and compliance with applicable laws and regulations.

53.61

5 Professional qualifications 41.24

Proper license/certificates/resumes of all personal 32.99

Business reference and pertinent certificates of design-builders and

subcontractors

13.40

6 Office location and familiarity with the locality 17.53

Office location and local participation 11.34

Familiarity with the locality (local practice, regulation, etc.) 8.25

7 Legal status 12.37

Active litigation and litigation history 12.37

8 Responsiveness to the RFQ 10.31

Adequate and compete response to the requested information 9.28

Proposal adheres to submission requirements set forth in RFQ 3.09




16

Table 3 Relative importance of qualification categories by project type

Qualification

categories

Commercial

buildings

Institutional

buildings

Industrial and

processing

Heavy civil and

highway

Renovation

Experience (%) 33.6 25.9 30.0 19.7 23.8

Approach (%) 15.0 26.1 22.1 29.5 19.5

Organization (%) 18.4 20.0 25.0 21.7 12.5

Performance (%) 18.5 11.1 12.9 13.8 28.5

Qualification (%) 6.5 10.4 6.4 14.8 13.2

Responsiveness (%) 1.9 3.6 0.7 -- 2.5

Locality (%) 6.1 1.4 2.2 0.5 --

Legal status (%) -- 1.5 0.7 -- --

Project number 8 28 7 10 4




Accep

ted M

anus

cript

Not Cop

yedit

ed

Figure 1 Relative importance of qualification categories

Figure 2 Score points of qualification categories

Figure Caption List



Figure 1 Relative importance of qualification categories

Figure 1.pdf




Figure 2 Score points of qualification categories

Figure 2.pdf




Documents

Bond University Research Repository Evaluation of design