Williamson (2012 jan 10) IT project complexity, complication, and success - research summary for PMI-SWVA

Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER

IT Project Complexity, Complication, and Success

Summary of a Research Study Conducted as part of the requirements for the degree

Doctor of Philosophy

Capella University by

David J. Williamson In cooperation with the

Project Management Institute

Information Systems Community of Practice

October, 2011

CxmCn


Summary

• Problem: IT project failure

• Causes: Many, but project complexity is

underdiagnosed

• Theory: Most PM methods are based on

the rational systems view; however, a

complex adaptive systems view more

effectively describes project behavior

• Methodology: Model distinguishing

between complexity and complication;

quantitative, correlational study using a

web-hosted survey

• Survey: New survey instrument

developed, tested, and administered to the

U.S. membership of the PMI IS CoP

• Results: ITPCx and ITPCn were positively

correlated; ITPCx had a greater negative

correlation with ITPS

• Implications: Identify and mitigate project

attributes related to IT project complexity to

increase the likelihood of IT project

success

January 10, 2012 3 Copyright © 2012, David J. Williamson


Methodology

Review Literature

Identify Variables, Factors, Elements

Build Model

Build and Test Survey

Collect Data Analyze Data

Describe Results Draw

Conclusions Suggest

Implications



Complication vs. Complexity

Attribute Complication Complexity

Cause Size, detail, number of parts Interactions between the parts

Analysis Can be decomposed, analyzed,

and described in terms of

components and parts

Cannot be analyzed and

described completely as

separate components

Behavior Consistent and predictable over

time

Adapts to environmental

influences

Response Responds linearly and

predictably to external events

Responds non-linearly and

unpredictably to external events,

evolves and changes over time,

exhibits emergent behaviors,

changes environment

Managing Can be managed with rational

systems approaches

Cannot be managed directly—

can only be accommodated or

mitigated

(Benbya & McKelvey, 2006; Cilliers, 1998; Hass, 2009)



Model: IT Project Complexity, Complication, and Success

• Identify project characteristics that

contribute to IT project complexity (ITPCx)

and IT project complication (ITPCn)

• Select a definition for IT project success

(ITPS)

• Investigate relationships among:

• ITPCx – ITPCn

• ITPCx – ITPS

• ITPCn – ITPS

• ITPCx – ITPS vs. ITPCn – ITPS

• Provide evidence that complexity and

complication are related but different sets

of project characteristics, with different

relationships to project success

IT Project

Complexity

(ITPCx)

IT Project

Complication

(ITPCn)

IT Project

Success

(ITPS)

Legend:

Primary Relationship

Secondary Relationship

October, 2011 6 © 2011, David J. Williamson

New model—first to distinguish complexity and complication


Factors and Elements of IT Project Complexity


ITPCx

Project

Complexity

ITPCx1 Objectives

ITPCx2

Opportunity

ITPCx7

Requirements

ITPCx3

Solution

ITPCx6

Schedule

ITPCx4

Team

ITPCx5

Methodology

ITPCx11

Org Change

ITPCx10

Tech Change

ITPCx8

Environment

ITPCx9

IT Complexity

ITPCx13

IT Integration

ITPCx2a Clear

ITPCx2b Familiar

ITPCx3a Familiar

ITPCx3b Available

ITPCx4a Experience

ITPCx4b Track Record

ITPCx5a Formal

ITPCx5b Consistent

ITPCx6a Reasonable

ITPCx6b Flexible

ITPCx7a Clear

ITPCx7b Stable

ITPCx8a Political

ITPCx8b Strategic

ITPCx8c Stakeholders

ITPCx8d Dependency

ITPCx8e Regulatory

ITPCx8f Legal

ITPCx9a Complexity

ITPCx9b Innovation

ITPCx11a Bus Proc

ITPCx11b Org Scope

ITPCx12

Staffing

Thirteen elements with 1 to 6 factors:

• Project objectives: Clarity

• Opportunity: Clarity and familiarity

• Solution: Familiarity and availability

• Team: Experience and track record

• Methodology: Formality and consistency

• Schedule: Reasonableness and flexibility

• Requirements: Clarity and stability

• Project environment: Political, strategic,

stakeholders, dependencies, regulatory, legal

• Information technology: Complexity and

innovation

• Technology: Degree of change

• Organizational change: Business processes

and scope

• Project staffing: Number of organizations

• Integration: Number of interfaces

(Austin, et al., 2002; Baccarini, 1996; Bardyn & Fitzgerald, 1996;

Brockhoff, 2006; Cilliers, 1998; Cooke-Davies, et al., 2007; Fitzgerald &

Bardyn, 2006; Frame, 1994; Hass, 2009; Jaafari, 2003; Singh & Singh,

2002; Whitty & Maylor, 2007, 2009)

Complexity is related to change and unknowns


Factors and Elements of IT Project Complication

Nine elements with 1 to 3 factors:

• Project leadership: Experience and competence

• Project duration: Length of original schedule

• Project team size: Number of members

• Project cost: Planned cost and flexibility

• Project scope: Flexibility

• Technology content: Percent of total scope

• Organizational support: Executives and users

• Organizational units: Number of units involved

• Contractors: Number, familiarity, and track

record

(Austin, et al., 2002; Baccarini, 1996; Bardyn & Fitzgerald, 1996;

Brockhoff, 2006; Cilliers, 1998; Cooke-Davies, et al., 2007; Fitzgerald &

Bardyn, 2006; Frame, 1994; Hass, 2009; Jaafari, 2003; Singh & Singh,

2002; Whitty & Maylor, 2007, 2009)


ITPCn

Project

Complication

ITPCn5

Scope

ITPCn2

Duration

ITPCn1a Experience

ITPCn1b Competence

ITPCn4

Cost

ITPCn3

Team Size

ITPCn1

Leadership

ITPCn7

Support

ITPCn9

Contractors

ITPCn6

Technology

ITPCn7a Executives

ITPCn7b Users

ITPCn4a Planned

ITPCn4b Flexibility

ITPCn8

Org Units

ITPCn9a Number

ITPCn9b Familiarity

ITPCn9c Track Record

Complication is related to size and familiarity


Factors and Elements of IT Project Success

Three elements with 2 to 3 factors:

• Project completion: percent completed and

percent implemented

• Project performance vs. initial baseline:

percent of schedule, budget, and scope

• Project performance vs. final baseline:

percent of schedule, budget, and scope

(Baccarini, 1999; Glass, 2006; Standish Group, 1994, 1999, 2009)


ITPS

Project

Success

ITPS1a %Completed

ITPS1b %Implemented

ITPS1

Completion

ITPS2a %Schedule

ITPS2b %BudgetITPS2

Performance (B1)ITPS2c %Scope

ITPS3a %Schedule

ITPS3b %BudgetITPS3

Performance (Bn)ITPS3c %Scope

Used a definition similar to Standish Group’s CHAOS Studies


Survey

Instrument

• No existing surveys comparing complexity

and complication

• New Internet survey instrument developed

for the study

• Not previously validated; testing required

Field Test

• Qualitative review in a PMI-SWVA seminar

on agile project management

• Quantitative review using a sample with a

feedback page

• Feedback incorporated into design,

instructions, layout, and questions

Pilot Test

• IRB approval prior to pilot testing

• Responses n = 42 exceeded the minimum

of n = 35 to 40 (Johanson & Brooks, 2010)

• Responses were well-distributed; non-

response rate was minimal


The survey was new, but tested and shown to be valid


Data Collection: Population and Sample

Population

• U.S. IT project managers

• PMI IS CoP: Over 15,000 members

worldwide, 40% to 50% in the U.S.

• Other studies have used this population

• Wallace, Keil, and Rai (2004)

• Xia and Lee (2005)

• Mishra, et al. (2009)

• Response rates ranged from 6% to 15%

Sample

• 6,000 U.S. members of the PMI IS CoP

• 100% probability sample

• Sample power analysis indicated minimum

sample size n = 115 for bivariate normal

correlation ( = .05, 1- = .95, r = .30)

• Actual response rate was 3.9% (n = 235)


The population was reasonably representative The sample was statistically reliable


Data Collection: Survey Responses

Survey

• Hosted by SurveyMonkey

• Invitations sent by PMI IS CoP

Responses

• Total qualified responses n = 235

exceeded minimum of n = 115

indicated by power analysis

• Response rate indicated data

collection period could have been

shortened

• Follow-up reminders could have

increased total response


Responses were more than adequate, could have been higher


Data Analysis: Demographics

• Responses: 235 qualified responses; organizations ranging from fewer than 10 to

more than 10,000 employees, annual budgets less than $10 million U.S. to more

than $5 billion U.S.

• Job Titles: Project Manager (46%), Program Manager (17%)

• Project Roles: Project manager (55%), program manager (25%), project team

member (7%).

• PMP Certification: 76%

• Industries: Finance, insurance, and banking (22%), information technology

(20%), healthcare (11%), other (10%) including pharmaceuticals, media,

government

• Project Types: Information technology (39%), software development (33%),

application package implementation (11%)


The sample appeared to be reasonably representative


Data Analysis: Distributions

• ITPCx and ITPCn

were reasonably

normal

• ITPS was non-normal,

required a normal

transform; NITPS

distribution was

reasonably normal


5.04.03.02.01.0

ITPCx

30

25

20

15

10

5

0

Fre

qu

en

cy

5.04.03.02.01.0

ITPCn

30

20

10

0

Fre

qu

en

cy

5.04.03.02.01.0

ITPS

40

30

20

10

0

Fre

qu

en

cy

2.01.00.0-1.0-2.0-3.0

NORMAL of ITPS using RANKIT

30

25

20

15

10

5

0

Fre

qu

en

cy

The data could be used for correlation analysis


• Statistically significant rank order

nonparametric correlations

between all pairs, with the exception

of the ITPCxITPCn and NITPS pair

• Statistically significant Pearson’s

product-moment parametric

correlations between all construct

pairs and construct-transform pairs

Data Analysis: Correlations

Paired constructs Pearson’s Kendall’s taub Spearman’s rho

r r2 p p rs p

ITPCx-ITPCn .530 .281 .000 .338 .000 .483 .000

ITPCx-ITPS -.356 .127 .000 -.256 .000 -.363 .000

ITPCn-ITPS -.247 .061 .000 -.123 .006 -.181 .005

ITPCx-NITPS -.350 .123 .000 -.256 .000 -.363 .000

ITPCn-NITPS -.228 .052 .000 -.123 .000 -.181 .005

ITPCxITPCn-NITPS -.185 .034 .004 -.051 .248 -.078 .236


Relationships existed, and they were statistically significant


Results: ITPCx and ITPCn

• Positive Pearson’s correlation existed

between ITPCx and ITPCn, r = .530, r2 =

.281, p < .001

• Positive nonparametric rank order

correlations also existed:

• Kendall’s taub = .338, p < .001

• Spearman’s rho rs = .483, p < .001.

• Finding 1: ITPCx was positively

correlated with ITPCn; p = .000 (less

than the significance level .05 for

bivariate normal correlation indicated by

power analysis)


Complexity and complication are related, but different


Results: ITPCx and ITPS

• Negative Pearson’s correlation existed

between ITPCx and ITPS, r = -.350, r2 =

.123, p < .001

• Negative nonparametric rank order


• Kendall’s taub = -.256, p < .001

• Spearman’s rho rs = -.363, p < .001

• Finding 2: ITPCx was negatively

correlated with ITPS; p = .000 (less than

the significance level .05 for bivariate

normal correlation indicated by power

analysis)


Complexity has a negative correlation with success


Results: ITPCn and ITPS

• Negative Pearson’s correlation existed

between ITPCn and ITPS, r = -.228, r2 =

.052, p < .001

• Negative nonparametric rank order


• Kendall’s taub = -.123, p < .01


• Finding 3: ITPCn was negatively

correlated with ITPS; p = .000 (less than

the significance level .05 for bivariate

normal correlation indicated by power

analysis)


Complication has a negative correlation with success


Results: ITPCx vs. ITPCn

• Pearson’s correlation for ITPCx and

ITPS, r = -.350, r2 = .123, p < .001 was

greater negative than for ITPCn and

ITPS, r = -.228, r2 = .052, p < .001.

• Nonparametric rank order correlations

for ITPCx and ITPS

• Kendall’s taub = -.256, p < .001


were also greater negative than ITPCn

and ITPS

• Kendall’s taub = -.123, p < .01


• Finding 4: ITPCx had a greater

negative correlation with ITPS than did

ITPCn


Complexity affects success more than complication


Results: Summary

Results

• ITPCx and ITPCn were positively

correlated (r = .530, r2 = .281,

p < .001) to a greater degree than either

variable and ITPS

• ITPCx and ITPS were negatively

correlated (r = -.350, r2 = .123,

p < .001) to a greater degree than were

ITPCn and ITPS (r = -.228,

r2 = .052, p < .001)

Revised model


Complexity affects success more than complication,

but complexity and complication are even more strongly related


Conclusions and Implications

Conclusions

• Complexity and complication are

related but distinct

• IT projects can have different levels

of complexity and complication

• Stronger relationship between

complexity and complication means

other factors are involved, or the

model needs to be refined

Implications

• There is a difference between

complexity and complication

• Focus on identifying, mitigating, and

accommodating IT project complexity

in order to increase the likelihood of IT

project success

January 10, 2012 Copyright © 2012, David J. Williamson 21

There is a difference between complexity and complication


Application: Effect on project performance

• The impact of complication on

project difficulty and performance

is linear

• The impact of complexity is non-

linear and unpredictable


Complication

Co

mp

lex

ity

n

x


Application: Selection of PM methods

• The choice of PM methods

should be driven by the project

characteristics

• Using an inappropriate method

increases the probability of failure

• Agile methods are more effective

for handling complexity than

traditional methods

• Agile methods are also effective

for small projects with low

complexity, if flexibility is

important



Application: Risk management

• Complication increases risk

• Complexity multiplies risk

• Mitigate complexity and risk with

frequent delivery



Study published January 6, 2012


http://gradworks.umi.com/34/81/3481367.html


Paper submitted to PMI Research and Education Conference 2012



References

Austin, S., Newton, A., Steele, J., & Waskett, P. (2002). Modelling and

managing project complexity. International Journal of Project

Management, 20(3), 191-198. doi: 10.1016/S0263-7863(01)00068-0

Baccarini, D. (1996). The concept of project complexity - a review.

International Journal of Project Management, 14(4), 201-204. doi:

10.1016/0263-7863(95)00093-3

Baccarini, D. (1999). The logical framework method for defining project

success. Project Management Journal, 30(4), 25-32.

Bardyn, J., & Fitzgerald, D. (n.d.). The uses of chaos theory in project

management. Retrieved June 7, 2005, from http://www.cyber-

office.org/writings/usesof.htm

Benbya, H., & McKelvey, B. (2006). Toward a complexity theory of

information systems development. Information Technology & People,

19(1), 12-34. doi: 10.1108/095938406106499

Brockhoff, K. (2006). On the novelty dimension in project management.

Project Management Journal, 37(3), 26-36.

Burkatzky, F. H. H. (2007). Development of measurement scales for

project complexity and system integration performance (Doctoral

dissertation). Walden University.

Cavaleri, S., & Reed, F. (2008). Leading dynamically complex projects.

International Journal of Managing Projects in Business, 1(1), 71-87.

doi: 10.1108/17538370810846423

Cilliers, P. (1998). Complexity and postmodernism: Understanding

complex systems. London, England: Routledge.

Cooke-Davies, T., Cicmil, S., Crawford, L., & Richardson, K. (2007). We're

not in Kansas anymore, Toto: Mapping the strange landscape of

complexity theory, and its relationship to project management. Project

Management Journal, 38(2), 50-61.

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design

method (2nd ed.). New York, NY: John Wiley & Sons.

Fitzgerald, D., & Bardyn, J. (2006). Chaos theory and project

management: A perspective on managing the complex project.

Retrieved August 24, 2006, from

http://www.nimblepm.com/writings/chaostheory.html

Frame, J. D. (1994). The new project management: Tools for an age of rapid

change, corporate reengineering, and other business realities (1st ed.).

San Francisco, CA: Jossey-Bass.

Glass, R. L. (2006). The Standish report: Does it really describe a software

crisis? Communications of the ACM, 49(8), 15-17. doi:

10.1145/1145287.1145301

Hass, K. B. (2009). Managing complex projects: A new model. Vienna, VA:

Management Concepts.

Jaafari, A. (2003). Project management in the age of complexity and change.

Project Management Journal, 34(4), 47-57.

Johanson, G. A., & Brooks, G. P. (2010). Initial scale development: Sample

size for pilot studies. Educational and Psychological Measurement, 70(3),

394-400. doi: 10.1177/0013164409355692

Mishra, A., Sinha, K., & Thirumalai, S. (2009). Comparative evaluation of

efficiency across distributed project organizations: A stochastic frontier

analysis. University of Minnesota. Minneapolis, MN.

Singh, H., & Singh, A. (2002). Principles of complexity and Chaos theory in

project execution: A new approach to management. Cost Engineering,

44(12), 23-33.

Standish Group. (1994). The chaos report. West Yarmouth, MA: Author.

Standish Group. (1999). Chaos: A recipe for success. West Yarmouth, MA:

Author.

Standish Group. (2009). Chaos summary 2009. Boston, MA: Author.

Wallace, L., Keil, M., & Rai, A. (2004). Understanding software project risk: A

cluster analysis. Information & Management, 42, 115-125.

Whitty, S. J., & Maylor, H. (2007). And then came complex project

management. Paper presented at the 21st IPMA World Congress on

Project Management.

Whitty, S. J., & Maylor, H. (2009). And then came complex project

management (revised). International Journal of Project Management,

27(3), 304-310. doi: 10.1016/j.ijproman.2008.03.004

Xia, W., & Lee, G. (2005). Complexity of information systems development

projects: Conceptualization and measurement development. Journal of

Management Information Systems, 22(1), 45-83.
