Embed Size (px)
Citation preview
Beyond CHAOS: Understanding Factors Affecting IT Project
Performance
Dr. Andrew GeminoSimon Fraser University
Agenda
• Part 1: Project Performance• Checking under the hood
• Part 2: UK Study • Categorizing performance• Impact of factors on performance
• Part 3: US Study • Temporal Model
• Part 4: Knowledge and Performance• How we learn affects how we do
2
Motivation for Our Research
• Application of Information Technology plays an important role in Canada’s capacity to innovate• The Info. And Comm. Technology Sector
• Contributes $61 B to Canadian GDP (in 1997 constant dollars)• Comprises 5.8% of Canadian GDP• Employs over 560,000 Canadians• Responsible for 38% of all private sector R&Dhttp://strategis.ic.gc.ca/epic/site/ict-tic.nsf/en/h_it06143e.html
• Our focus is to advance our ability to understand and influence ICT project performance.
3
The Team
• Blaize Horner Reich, Simon Fraser University• Chris Sauer, Oxford University, UK• Andrew Gemino, Simon Fraser University
• Website: www.PMPerspectives.org
• Funding: • Social Sciences and Humanities Research Council
(SSHRC) • Initiative for the New Economy (2003-2006)• Research Grant funding (2007-2010)• INE Communication Grant
• Natural Sciences and Engineering Research Council (NSERC)
• Research Grant Funding
4
Selected Output
• Gemino, A., Sauer, C. and Horner Reich, B. 2010, “Using Classification Trees to Predict Project Performance” accepted in the Journal of Decision Systems.
• Gemino, A., Horner-Reich, B. and Sauer, C. "A Temporal Model for IT Project Management", Journal of MIS, Winter 2008, Vol. 24, No. 3, pp. 9–44.
• Reich, B. Gemino, A., Sauer, C. 2008. “Modeling the Knowledge Perspective of IT Projects”, Project Management Journal, Vol. 39, S4-S14.
• Gemino, A., Reich, B. and Sauer, C. "Beyond Chaos: Examining IT Project Performance” Sauer, C., Gemino, A, and Reich, B.H. “Managing Projects for Success: The Impact of Size and Volatility on IT Project Performance”, Communications of the ACM, 60:11, Nov. 2007, pp. 79-84.
• Gemino, A., Reich, B, Sauer, C. "Examining IT project performance", Proceedings of Administrative Sciences Association of Canada (ASAC) May 24-27, 2008, Halifax, Nova Scotia
• Reich, B. H., Sauer, C. and Gemino, A. “The Influence of Knowledge Management on Business Value in IT Projects: A Theoretical Model”, 3rd Pre-ICIS International Research Workshop on Information Technology Project Management, December 12, 2008, p. 96-113.
55
PART 1: Project Performance
Checking Under the Hood
6
What do you think?
• Introduce yourself to a few people and spend five minutes talking about the following questions.
• 1) How is project performance measured at your company? (On time/on budget? Business value? Other?)
• 2) Does your company separate between project performance and project manager performance?
• Be ready to share your discussion with the group.
7
Standish Surveys 1994-2009
(complied from press releases from (complied from press releases from www.standishgroup.com ) )
8
Summary
• We are still in early stages of measuring project performance. • Difficult to measure value
• Takes time to realize benefits• Difficult to measure impact (before and after)
• Difficult to separate PM from Project evaluation• Do good PM’s get more challenging projects• Are abandoned projects a wine or loss?
• Our teechniques ill continue to improve. Until they do, recognize the reality and choose wisely.
9
Part 2: UK Study
Survey Research
• Study 1: Computer Weekly UK
• Web based survey of• Readers and PM’s
• Asked IT project managers about their last project (completed or abandoned)
• 421 full responses• Average 17 years industry experience• Average 9 years as project manager
1111
What we found
• 65% of projects in the UK• Delivered within a reasonab;le contingency
(approx 7%) of ALL targets. • Compare with 63% challenged or abandoned in
Standish
• What’s the difference? • Experienced Project Managers• Data collection
• We asked for actual variances from original goal
• Cluster Analysis• Didn’t use hard line (99% is not challenged)• “let the data speak for themselves”
1212
UK - Project Characteristics
Size Characteristi
c
Type 1:Abandon
Type 2:Budget
Challeng
Type 3:ScheduleChalleng
Type 4:Good
Type 5:Star
Budget(Median, in ₤
000’s)₤ 1,000 ₤ 625 ₤ 500 ₤ 450 ₤ 2,000
Effort(Average
Person Months)
798 557 212 89 170
Duration (Average
Elapsed Time in Months)
17.4 20.0 13.0 11.2 15.3
Team Size(Effort/
Duration) 35.7 17.7 12.9 7.3 9.8
13
IT Project Types: UK Study
PerformanceVariance
Type 1:
Abandoned
n=39
Type 2:
BudgetChallenged
n=22
Type 3:
ScheduleChallenge
dn=76
Type 4:
Good
n=250
Type 5:
Stars
n=31
Performance Variances (Actual as % of Originally Planned) – 100%
Schedule N/A +18% +46% +2% +1%
Budget N/A +40% +16% +7% -24%
Scope N/A -12% -16% -7% +15%
14
Surprises:Some IT Projects Exceed
Expectations1. The IT Performance story is not all bad
1. 2/3rds of projects are performing well2. Some IT projects exceed expectations
1. 7% of UK2. 17% of US projects
• These projects that exceed expectations are not mentioned in Standish Group Reports • We found them because we did not constrain
“success” in the data collection
1515
Factors affecting performance
Defining Underperfoming
• Performing• Good or Stars
• Underperforming• Abandoned• Budget Challenged• Schedule Challenged
17
Risk Associated with Project Size
1(a) Risk of Underperformance due to Effort
25%31%
36% 36% 38%
50%
77%
100%
0%
33%
67%
100%
25 or less 25-50 50-100 100-200 200-500 500-1000 1000+ 2400+
Effort (person months)
18
Risk Associated with Project Size
1(b) Risk of Underperformance due to Duration
24%21%
34%
25%
36%
48%
0%
33%
66%
< 3 months 3-6 months 6-9 months 9-12 months 12-18months
18+ Months
Duration (months)
19
Risk Associated with Project Size
1(c) Risk of Underperformance due to Team Size
28%
38%
26%
33%
58%
0%
33%
67%
<5 FTE 5-10 FTE 10-15 FTE 15-20 FTE >20 FTE
Team Size (Effort / Duration)
20
Risk Associated with Volatility
2(a) Risk of Underperformance due to Governance Volatility
22%33%
60%53%
74%82%
0%
33%
67%
100%
0 changes 1 change 2 changes 3 changes 4 changes 5+changes
Governance Volatility (changes in PM or Sponsor)
21
Risk Associated with Volatility
2(b) Risk of Underperformance Due to Project Target Volatility
11%
25%
34%
57%
0%
33%
67%
< 3 changes 3-6 changes 6-9 changes 9+ changes
Project Target Volatility (changes to project targets)
22
PART 3: US Study
Temporal Model of Project Performance
23
US Study
Study 2: PMI Chapters in Ohio
• Web based survey• 3 PMI chapters in Ohio
• Asked IT project managers about their last project (completed or abandoned)
• 194 full responses• Average age, 43• Average years experience, 15• Average PM training: 34 days
24
US - Project Characteristics
Size Characteristi
c
Type 1:Abandon
Type 2:Budget
Challeng
Type 3:ScheduleChalleng
Type 4:Good
Type 5:Star
Budget(Median, in
US $ 000’s)$2,000 $700 $1,042 $670 $600
Effort(Average
Person Months)
N/A 126.4 163.2 255.1 66.5
Duration (Average
Elapsed Time in Months)
N/A 16.7 16.5 15.8 9.5
Team Size(Effort/
Duration) N/A 6 11.0 16.4 6.0
25
IT Project Types: US Study
Performance Variance
Type 1:Abandoned
Projectsn=16
Type 2:
Challengedn=13
Type 3:Schedule
Challengedn=36
Type 4:
Goodn=87
Type 5:
Starsn=32
Performance Variances (Actual as % of Originally Planned – 100%)
Schedule N/A +41% +107% +4% +1%
Budget N/A +25% +43% +3% -12%
Scope N/A -12% -4% -9% +3%
26
How should performance be measured?
• Two basic outcomes in project• Process Outcomes
• On time• On budget• On specs
• Product Outcomes• Value delivered• Benefits• Quality
PM’s are often measured here
But are expected to deliver here
27
Risk Factors and Performance
28
Risk Factors and Performance
What happens in the middle?
?
29
What Affects Performance
• Risk Factors and Resources• Initial (A-priori) Factors
• Knowledge resources Team. PM, Sponsor, Clients
• Structural Factors Technical complexity, budget, duration, effort
30
What Affects Performance
• Emergent Factors• Organizational Resources
Top Management Support, User participation• Volatility
Governance changes, Target Changes, Environment changes
• Project Management Practice• Expertise Coordination
Who are knowledge leaders How can they be accessed
• Horizontal Communication• PM Methods and Tools
31
US Study: Results
Knowledge Resources
Structural Risk
(Size & Tech Complexity)
OrganizationalSupport
Rsources
Project Management
Practices
VolatilityRIsk
ProjectProcess
Performance
ProjectProduct
Performance
0.292**
0.290**
0.502**
0.298**
0.359**
R2 = 0.361
R2 = 0.269
R2 = 0.076
R2 = 0.219
R2 = 0.385
-0.321**
0.396**
0.154*
-0.209*
A-Priori Emergent Performance
32
Results
• With this model we can explain• Approximately 40% of the variance in Process
performance• Approximately 22% of the variance in Product
performance
• We can also show the strength of the relationships between risk and resource factors associated with IT Projects. • Strength and direction is given by the number
associated with each line.
33
Results
• Two Broad Forces Acting on Projects 1. Forces of Evil
• Structural risk is strongly related to volatility and volatility is negatively related to process performance
• Large projects often have a bumpy ride and are challenged in regards to being on time and budget.
34
Structural Risk
(Size & Tech Complexity)
VolatilityRIsk
ProjectProcess
Performance
0.502**
R2 = 0.361
R2 = 0.385
-0.321**
34
Results
Knowledge Resources
OrganizationalSupport
Rsources
Project Management
Practices
ProjectProcess
Performance
ProjectProduct
Performance
0.292**
0.290**
0.298**
0.359**
R2 = 0.269
R2 = 0.076
R2 = 0.219
R2 = 0.385
0.396**
0.154*
2. Forces of Good• Increased knowledge is
related to higher organizational support and higher level of PM Practices.
• Increased PM Practice is related to better process and PRODUCT performance
35
Summary
• We have some idea of the factors that affect process performance. More work needs to be done. • It is important to consider initial factors as well as
emergent factors when considering performance.
• We do not yet have a good understanding of how these factors affect product performance• More work required
36
Where do we go from here?
Part 4: Project Performance
Knowledge and Projects
• The case for a Knowledge/Business Value Approach
• Theoretical Model • Knowledge Management• Knowledge Alignment• Impact on Business Value
• Results of a Pilot Survey
38
Project as Knowledge Process
• In a complex/uncertain IT project, everyone is sharing knowledge and acquiring new knowledge:
• Sponsor• Project Manager• Vendors/Consultants• Project team members
• Seen through the K lens, the job of the PM is to facilitate the knowledge practices so that learning happens
• At the right time • For the right price
39
What is our goal?
• Explore how KM may impact Business Value• Recognizing that there are multiple influences on
BV
• Increase the R² - the ability to explain Project Performance
40
Knowledge Management
Project Managers must: • Create the enabling
environment• Acquire and maintain
the K stocks• Manage the K
practices
Enabling Environment
KnowledgePractices
KnowledgeStocks
41
Connecting Knowledge Mgmt with Business Value
Business ValueAttained
Knowledge Alignment Knowledge Mgmt
Enabling Environment
KnowledgePractices
KnowledgeStocks
42
Three Types of Knowledge
Organizational Solution
Business Value Desired
Project Based Knowledges
Business ValueAttained
Technical Solution
43
Why Separate Knowledges?
• Publications are largely focused on the technical solution K• We are pushing beyond targets to BV• The 3 knowledges are largely developed by different “teams”
44
Alignment of Knowledges
Alignment of Project Based Knowledges
Business Solution
Technical Solution
Business Value
Project Based Knowledges
Business Value
45
Impact of KM on Business Value
Alignment of Project Based Knowledges
Technical Solution
Organizational Solution
Business Value Desired
Project Based Knowledges
Business ValueAttained
Knowledge Management Knowledge Alignment
Enabling Environment
KnowledgePractices
KnowledgeStocks
46
Evidence of importance of KM
• Executive Sponsor impact • Sponsors may lack the K to lead a project
• Volatility impact• Loss of K as people leave the project
• Expertise Coordination impact • Strong positive correlation but we don’t know how
• We fail often at learning from projects• No learning in, very little out of a project
• Knowledge of the Team• When K is high, PM and Sponsors do more KM
47
Latest Survey demographics
• N = 25 • Averages with (and without) outlier
• Duration 19.5 mo, (15.2)• Budget $22.7M ($1.1M)• 261 person months (48)• 8 IT people, 11 business people on team
• Majority were in-house, custom builds
48
What we could test:3 initial regressions
Enabling Environment
KnowledgePractices
KnowledgeStocks
Business ValueAttained
1
2
3
49
Knowledge Stock Test
Construct Variable Sig.
Knowledge Stock
IT Team Knowledge (3 items) Not Sig.
Business Team Knowledge (3 items) Not Sig.
Governance Team Knowledge (3 items)
Not Sig.
External Network (2 items) Not Sig.
Absorptive Capacity (2 items) Significant
Previous Collaboration (3 items) Not Sig.
50
Enabling Environment Test
Construct Variable Significance
Enabling Environment
Governance Team Promotion of Learning Environment (1 item) Not Sig.
Project Team Knowledge and Learning Orientation (1 item) Significant
Co-Location (4 items) Not Sig.
Technology (2 items) Not Sig.
51
Knowledge Practices Test
Construct Variable Significance
Knowledge Practices
IT Team Expertise Coordination (3 items)
Not Sig.
Business Team Expertise Coordination (3 items) Not Sig.
Cross-Team Expertise Coordination (5 items)
Not Sig.
Formal Knowledge Practices (4 items)
Not Sig.
Cross-Team Knowledge Sharing (2 items)
Significant
52
Final Regression
Enabling Environment
KnowledgePractices
KnowledgeStocks
Enabling Environment
Business Value
R2 = 62 % (of the variance inSatisfaction with Business Value is explained).
53
A Conclusion
• Expertise does not matter to BV• The “lifecycle” matters:
• Hire those willing to Learn and Share• Develop a Learning Culture
Try, learn, share
• Create Knowledge Sharing Opportunities
Meetings, status reports, social gatherings)
54
Principles For IT Projects
• So what can I tell you about running effective projects….
55
Next Steps
• Global Survey of Knowledge Techniques in IT Projects with a focus on:• Knowledge alignment• Knowledge practices• Absorptive Capacity (ability to learn and adapt)• Business Value
• Kickoff in May 2010• If you are interested, please let me know.
56
Takeaways
57
Establish Trust and Sharing
58
Maintain Knowledge Levels
59
Create Channels for Knowledge
60
Push Hard for Alignment
61
Rethinking Project Management
• IS project management more than getting from point A to point B on time and on budget?
• What if project management is: • Developing a culture of learning and knowledge
sharing• Expanding knowledge in project participants• Creating channels for knowledge to flow more
freely• Easing team members towards alignment
62
Projects and Knowledge
• What would you do differently if you thought of projects from a knowledge perspective?
63
Questions?
If you are interested in more information please visit: www.PMPerspectives.org
64
