2

• U.S. DoD contractor• 24 years of experience• B.S. Computer Science• M.S. Software Engineering• D.M. Information Tech. (2007)• Four books and numerous articles• Worked in Japan, U.S., and Europe• Some experiences with agile methods

3

• The purpose of this study is to determine if the use of agile methods improves the quality of ecommerce websites

• The goal of this study is to develop an instrument to measure the use of agile methods and ecommerce website quality

• The objective of this study is to examine the links between the factors of agile methods and ecommerce website quality

4

• A software method is an approach to the analysis, design, construction, and implementation of an information system

• An agile method is a process of injecting customer feedback into a stream of working software versions to converge on a solution

• A website’s quality is the extent to which it facilitates efficient and effective shopping, purchasing, and delivery of its products

5

• The Internet is a powerful communication medium for free market-style business transactions

• The Internet shifted the balance of power away from industrial age organizations in favor of Internet firms

• The Internet presents formidable challenges for the development of Internet products and services

• Some use principles of flexible manufacturing, lean development, and adaptation to turbulent markets

• Other firms continue to use methods based on principles pioneered at the turn of the 20th century

• Our challenge is to help determine if the use of agile methods is more effective than traditional ones

6

• IT is 2nd major contributor to U.S. economy• There are 5 million IT workers in the U.S.• Today, there are over 100 million websites• Top 500 U.S. firms spend $186B on IT• U.S. ecommerce revenues are about $2T

–$1.82T is B2B–$130B is B2C

• There are about 250,000 U.S. IT projects–180,000 U.S. IT projects are failing each year–165,000 use agile methods to help reduce failures

7

• 1928 - Organismic Biology (Bertalanffy)

• 1949 - Cybernetics (Wiener)

• 1966 - Learning by doing (Levhari)

• 1969 - Involvement (Fitch)

• 1971 - Participatory design (Milne)

• 1975 - Iterative (Basili)

• 1976 - Evolutionary (Bauer)

• 1977 - Double-loop learning (Argyris)

• 1978 - Incremental (Cave)

• 1982 - Adaptive organization (Anderson)

• 1982 - Prototyping (Naumann)

• 1986 - Joint application design (IBM)

• 1987 - Rapid systems design (Gane)

• 1986 - Spiral (Boehm)

• 1991 - Cooperative design (Greenbaum)

• 1995 - Sense and response (Randall)

• 1996 - Ecosystems (Moore)

• 1997 - Probes (Brown)

• 1998 - Experimentation (Thomke)

• 1998 - Internet time (Cusumano)

8

• Cheap personal computers• Standard operating systems• Easy-to-use Internet browsers• Powerful programming languages• Explosion of computer programmers• Real-time software development cycles• Internet shopping (electronic commerce)

9

Phase System

Requirements Analysis

System Architectural

Design

Software Requirements

Analysis

Software Architectural

Design

Software Detailed Design

Software Coding

and Testing

Software Integration

Software Qualification

Testing

System Integration

System Qualification

Testing

Software Installation

Software Acceptance

Support

Product • SRS • SARAD • SRD

• SAD • SIDD (t) • DDD (t) • UDD (p) • TVPL (si)

• SDD • SIDD (d) • DDD (d) • UDD (u) • TVPL (su) • TVPL (siu)

• Software • TVPR (su) • TVRR (su) • UDD (u) • TVPL (siu)

• SOIP • TVRR (si) • UDD (u) • TVPR (sq)

• TVRR (sq) • UDD (u) • SIAR (sfc) • SIAR (spc)

• TVRR (yi) • TVPR (yq)

• TVRR (yq) • SIAR (yfc) • SIAR (ypc)

• SIP • TVRR (sa) • Training

Evaluation • Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

• Walkthru • Inspection

Record • SYRER • SYAER • SORER • SOAER • DDER • EOCR • SCTRER • SCR

• SIER • DER • SCR

• SQTER • SCR • SER • SQTARR

• SIRR • SCR

Audit • PCA • FCA

• PCA • FCA

Review System

Requirements Review

System Design Review

Software Specification

Review

Preliminary Design Review

Critical Design Review

Software Test

Readiness Review

Software Formal

Qualification Review

System Test

Readiness Review

System Formal

Qualification Review

Baseline Functional Baseline

Allocated Baseline

Developmental Configuration

Software Test

Baseline

Software Product Baseline

System Test

Baseline

System Product Baseline

12 phases — 35 documents — 62 evaluations — 17 records — 4 audits — 9 reviews — 9 baselines — 28,978 hours (10 KLOC)

10

11

Year Source Findings Responses

1998 Harvard (Thomke et al., 1998)

50% reduction in engineering effort 55% improvement in time to market 925% improvement in number of changes allowed

391

1998 Harvard (MacCormack, 1998)

48% productivity increase over traditional methods 38% higher quality associated with more design effort 50% higher quality associated with iterative development

29

1999 Boston College (Fichman et al., 1999)

38% reduction in time to produce working software 50% time to market improvement 50% more capabilities delivered to customers

28

2003 Reifer Consultants (Reifer, 2003)

20% reported productivity gains 10% reported cost reductions 53% reported time-to-market improvements

78

2003 Shine Technologies (Johnson, 2003)

49% experienced cost reductions 93% experienced productivity increases 88% experienced customer satisfaction improvements

131

2004 CIO Magazine (Prewitt, 2004)

28% had been using agile methods since 2001 85% initiated enterprise-wide agile methods initiatives 43% used agile methods to improve growth and marketshare

100

2006 Digital Focus (Digital Focus, 2006)

27% of software projects used agile methods 23% had enterprise-wide agile methods initiatives 51% used agile methods to speed-up development

136

2006 Version One (Version One, 2006)

86% reported time-to-market improvements 87% reported productivity improvements 92% reported ability to dynamically change priorities

722

2006 AmbySoft (Ambler, 2006)

41% of organizations used agile methods 44% reported improved productivity, quality, and costs 38% reported improvements in customer satisfaction levels

4,232

12

• Early studies based on primitive models• Agile methods matured after early studies• Later studies were either

– Too narrow (e.g., pair programming)– Too specific (e.g., extreme programming)– Too broad (e.g., every conceivable practice)

• Gaps in literature and need for new study– Theory covering essential factors– General-purpose measurement model– Original measurement data on agile methods

13

Method Major Factors

New development rhythm

Iterations 1, involvement 2, empowered teams 3, modularity 4, synchronization, configuration control, dependency management, performance reviews, metrics, testing, reviews

Scrum Iterative development 1, stakeholder feedback 2, self managing teams 3, prioritized requirements 3, daily team meetings 3, early architectural design 4

Dynamic systems development

Iterative development 1, frequent delivery 1, user involvement 2, stakeholder cooperation 2, empowered teams 3, simple flexible designs 4, change control, high-level requirements, tests

Synch-n-stabilize Iterations 1, daily builds 1, releases 1, customer feedback 2, small teams 3, vision statements 3, prioritized features 3, milestones 3, evolving specifications 4, parallel development

Internet time Rapid prototyping and early beta releases 1, daily incorporation of rapid market feedback 2, experienced teams 3, large investments in software architecture and design 4

Judo strategy Beta testing 1, market feedback 2, small teams 3, cross platform design 4, modular designs 4, reuse 4, flexible priorities 4, evolving features 4, parallel development, testing

Extreme programming

Releases 1, on-site customer 2, pair programming 3, simplicity 4, planning, metaphors, tests, refactoring, continuous integration, collective owners, 40-hours, open workspace, just rules

Feature driven development

Regular builds 1, domain experts 2, feature teams 3, technical architecture 4, object modeling, design by feature, class (code) ownership, inspections, configuration management, reporting

Open source software

Rapid releases 1, increased user involvement 2, prompt feedback 2, international community 3, highly-talented developers 3, evolutionary designs 4, parallel development, peer reviews

Agile manifesto Working software 1, Customer collaboration 2, individuals and interactions 3, responding to change 4

1 Iterative development — 2 Customer feedback — 3 Well-structured teams — 4 Flexibility

14

Product Dev. Institute Harvard I Harvard II National Research Council NASA LeanTEC • Customer focused

- Build in market-focused actions - Market needs/customer value - Field work by core project team - Begin early - Broaden the base

• Front-end loaded - Preliminary market assessment - Technical assessment - Supplier assessment - Market research - Concept testing - Customer value assessment - Business/financial assessment

• Spiral development - User needs and wants study - Full proposition concept test - Rapid prototype and test - Field trial and beta test

• Cross functional teams - Multi-disciplinary team - End-to-end project

responsibility - Clearly defined team leader - Excellent communications - Empowerment over resources

• Measure and improve - Put performance metrics in

place - Establish success criteria - Hold teams responsible - Continuously improve

• Maximize portfolio productivity - Strategic buckets - Product roadmaps - Project selection/prioritization - Resource allocations - Portfolio reviews

• Product innovation - Institutionalize stage-gates - Scalable/adaptable stage-gates - Automate stage-gates - Alliance-enabled stage-gates - Improve stage-gates

• Development strategy - Prioritize projects - Functional integration - Maximize efficiency - Create/improve capabilities

• Functional mapping - Marketing mapping - Engineering mapping - Manufacturing mapping - Integrative mapping

• Aggregate project planning - Research projects - Alliance-based projects - Incremental projects - Breakthrough projects - Next-generation projects

• Development funnels - Research-driven - Entrepreneurial-driven - Innovation-driven

• Development framework - Customer-focused - Disciplined - Coherence and detailed - Fit with mission - Standardized pattern

• Cross-functional integration - Cross-communication - Relationship management - Organization commitment - Management commitment - Incentives to integrate

• Cross-functional leadership - Functional teams - Lightweight teams - Heavyweight teams - Autonomous teams

• Tools and methods • Prototype test cycles • Organizational learning • Capability development

• Superior performance in time, productivity, and quality - Lead time focus - Productivity focus - Total product quality focus

• Integration in the development process - Overlapping and

communication - Small team sizes and

specialization - Simple, flatter organizations

• Integrating customer and product - Heavyweight product managers- Customer access and orientation- Leadership by concept

• Manufacturing for design - Manufacturing principles focus - Rapid prototyping - Rapid tools development

• Acceptance of risk • Flexible environment • Open communication • Organization-wide

commitment • Value innovation • Focus on end-user needs • Reduced lead times • Consistent funding • Use of COTS • Iterative development • Simple procurement and

acquisition processes • Flexible standards and

testing procedures • Decentralized decision-

making

• Schedule/budget constraints • Collocated personnel • Flat organization • Concurrent engineering • Contract outsourcing • Streamlined acquisition • COTS components • Design reuse • Flexible designs • Front-loaded funding • Simplified reviews • Minimal redundancy • Extensive testing • Public relations

• Technology transition process

• Enabling environment • Technology transition

portfolio • Project resources • Project charters • Project plans • Project contracts • Communication protocols • Collaboration protocols • Shared team experiences • Formal reviews

Common Factors • Early market feedback (e.g., on each increment). • Iterative development (e.g., 30-60-90 day increments). • Small, highly-qualified cross-functional project teams

with influential leaders (e.g., mostly PhDs). • Flexible processes and products (e.g., streamlined

processes and rapid-prototyping technologies). (What this boils down to is integrating and streamlining

research, development, and operations into a single, smaller, and more efficient organization.)

15

16

Agile Methods Website Effectiveness Electronic Commerce

Customer Feedback

Feedback solicitedFeedback receivedFeedback frequencyFeedback qualityFeedback incorporated

Iterative Development

Time boxed releasesOperational releasesSmall releasesFrequent releasesNumerous releases

Well-Structured Teams

Team leaderVision and strategyGoals and objectivesSchedules and timelinesSmall team size

Flexibility

Small sizeSimple designModular designPortable designExtensible design

H 1 (+)

H 2 (+)

H 3 (+)

H 4 (+)

Website Quality

Website designPrivacy and securityFulfillment and reliabilityCustomer service

Business-to-Consumer (B2C)

ShoppingRetailServices

17

Factor Variable Item

Time-boxed releases We develop software using time-based iterations, increments, or demonstrations

Operational releases We develop software using operational iterations, increments, or demonstrations (working code)

Small releases We develop software using small iterations, increments, or demonstrations

Frequent releases We develop software using daily, weekly, bi-weekly, or monthly iterations, increments, or demonstrations

Iterative development

Numerous releases We develop software using multiple (several) iterations, increments, or demonstrations

Feedback solicited We seek customer feedback on our software iterations, increments, or demonstrations

Feedback received We receive customer feedback on our software iterations, increments, or demonstrations

Feedback frequency We receive timely customer feedback on our software iterations, increments, or demonstrations

Feedback quality We receive a lot of (detailed) customer feedback on our software iterations, increments, or demonstrations

Customer feedback

Feedback incorporated We incorporate customer feedback into our software iterations, increments, or demonstrations

Team leader Our software teams have clear administrative or technical leaders

Vision and strategy Our software teams have clear visions, missions, or strategies

Goals and objectives Our software teams have clear goals or objectives

Schedules and timelines Our software teams have clear schedules or timelines

Well-structured

teams

Small team size Our software teams have a small size with no more than 10 people

Small size Our software is designed to be as small as possible

Simple design Our software is designed to be as simple as possible

Modular design Our software is designed to be modular or object-oriented

Portable design Our software is designed to work on multiple operating systems

Customer service

Extensible design Our software is designed to be changed, modified, or maintained

18

• Consulted with numerous experts• Analyzed 100s of papers/books (3 yrs)• Used a lot of suggestions and changes• Presented research at major conference• Solicited expert feedback on final models• Conducted cognitive interviews w/experts• Numerous refinements to survey instrument

– Prototype surveys– Multiple pilot surveys– Multiple-phased surveys

19

• Type of service– Online service (e.g., freeonlinesurveys.com)– Free trial-period, inexpensive, and easy-to-use

• Source of data– Clients of big consulting firm (e.g., QAI)– Patrons of popular journal (e.g., Dr. Dobbs)– Assisted by Scott Ambler and Jon Erickson (editor)

• Method of administration– 1st round from email survey (50 respondents)– 2nd round from editor’s blog (100 respondents)– 3rd round from editor’s newsletter (100 respondents)

• Type of survey and data collection– Respondents mostly self-selected– Respondents were provided incentives

20

21

1 2 3 4 5 6 7

Factor Variable Strongly Disagree

Disagree Somewhat Disagree

Neutral Somewhat

Agree Agree

Strongly Agree

Responses Score

Time-boxed releases 14 (6%) 23 (9%) 22 (9%) 13 (5%) 38 (15%) 59 (24%) 80 (32%) 249 5.15 / 7 (73.55%)

Operational releases 10 (4%) 18 (7%) 19 (8%) 17 (7%) 50 (20%) 84 (34%) 51 (20%) 249 5.15 / 7 (73.55%)

Small releases 10 (4%) 24 (10%) 18 (7%) 25 (10%) 34 (14%) 64 (26%) 71 (29%) 246 5.13 / 7 (73.34%)

Frequent releases 27 (11%) 30 (12%) 17 (7%) 23 (9%) 34 (14%) 50 (20%) 66 (27%) 247 4.7 / 7 (67.21%)

ITERATIVE DEVELOPMENT

Numerous releases 9 (4%) 18 (7%) 11 (4%) 26 (10%) 35 (14%) 67 (27%) 82 (33%) 248 5.38 / 7 (76.79%)

Feedback solicited 6 (2%) 14 (6%) 12 (5%) 20 (8%) 38 (15%) 72 (29%) 87 (35%) 249 5.55 / 7 (79.23%)

Feedback received 4 (2%) 16 (6%) 24 (10%) 17 (7%) 55 (22%) 77 (31%) 56 (22%) 249 5.24 / 7 (74.87%)

Feedback frequency 7 (3%) 34 (14%) 35 (14%) 37 (15%) 60 (24%) 44 (18%) 29 (12%) 246 4.45 / 7 (63.59%)

Feedback quality 25 (10%) 35 (14%) 39 (16%) 37 (15%) 44 (18%) 45 (18%) 21 (9%) 246 4.05 / 7 (57.9%)

CUSTOMER FEEDBACK

Feedback incorporated 6 (2%) 8 (3%) 12 (5%) 13 (5%) 44 (18%) 80 (32%) 86 (35%) 249 5.67 / 7 (81.01%)

Team leader 9 (4%) 11 (4%) 20 (8%) 27 (11%) 43 (17%) 84 (33%) 57 (23%) 251 5.25 / 7 (74.96%)

Vision and strategy 11 (4%) 20 (8%) 23 (9%) 29 (12%) 60 (24%) 79 (32%) 28 (11%) 250 4.82 / 7 (68.91%)

Goals and objectives 3 (1%) 12 (5%) 22 (9%) 24 (10%) 57 (23%) 95 (38%) 36 (14%) 249 5.2 / 7 (74.35%)

Schedules and timelines 6 (2%) 15 (6%) 26 (11%) 25 (10%) 59 (24%) 67 (27%) 48 (20%) 246 5.07 / 7 (72.42%)

WELL-STRUCTURED

TEAMS

Small team size 6 (2%) 10 (4%) 10 (4%) 13 (5%) 16 (6%) 63 (25%) 131 (53%) 249 5.96 / 7 (85.08%)

Small size 13 (5%) 44 (18%) 18 (7%) 61 (24%) 51 (20%) 38 (15%) 24 (10%) 249 4.22 / 7 (60.24%)

Simple design 8 (3%) 28 (11%) 21 (8%) 37 (15%) 47 (19%) 67 (27%) 41 (16%) 249 4.82 / 7 (68.79%)

Modular design 6 (2%) 11 (4%) 14 (6%) 23 (9%) 49 (20%) 87 (35%) 58 (23%) 248 5.38 / 7 (76.9%)

Portable design 38 (15%) 41 (17%) 20 (8%) 32 (13%) 36 (15%) 43 (17%) 38 (15%) 248 4.08 / 7 (58.29%)

FLEXIBILITY

Extensible design 4 (2%) 16 (6%) 23 (9%) 23 (9%) 51 (20%) 75 (30%) 59 (24%) 251 5.24 / 7 (74.84%)

Survey has good reliability and validity – Low feedback frequency and quality is systemic issue – Poor wording affected small size and portable design

22

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Time-boxed releases

Operational releases

Small releases

Frequent releases

Numerous releases

Feedback solicited

Feedback received

Feedback frequency

Feedback quality

Feedback incorporated

Team leader

Vision and strategy

Goals and objectives

Schedules and timelines

Small team size

Small size

Simple design

Modular design

Portable design

Extensible design

ITER

ATI

VED

EVEL

OPM

ENT

CU

STO

MER

FEED

BA

CK

WEL

L-ST

RU

CTU

RED

TEA

MS

FLEX

IBIL

ITY

Strongly Agree

Agree

Somewhat Agree

Neutral

Somewhat Disagree

Disagree

Strongly Disagree

Blues and greens indicate reliability and validity – Feedback frequency and quality systemic issue – Small size and portable design misinterpreted

23

PRODUCTIVITY

3%11%

13%

19%

17%

19%

18%0 to 10%

11% to 25%26% to 50%

51% to 75%

76% to 100%101% to 200%

201% or more

CYCLE TIME

8%

11%

14%

18% 13%

17%

19%

0 to 10%

11% to 25%

26% to 50%51% to 75%

76% to 100%

101% to 200%201% or more

COST

26%

19%

19%

15%

7%

11%

3%

0 to 10%

11% to 25%

26% to 50%51% to 75%

76% to 100%

101% to 200%201% or more

QUALITY

5%

15%

13%

13%19%

15%

20%0 to 10%

11% to 25%

26% to 50%51% to 75%

76% to 100%

101% to 200%201% or more

CUSTOMER SATISFACTION

18%

10%

18%

10%

25%

11%

8%0 to 10%11% to 25%

26% to 50%

51% to 75%76% to 100%

101% to 200%

201% or more

24

WEB

SITE

DES

IGN

PRIV

AC

Y A

ND

SEC

UR

ITY

FULF

ILLM

ENT

AN

DR

ELIA

BIL

ITY

CU

STO

MER

SER

VIC

E

25

IterativeDevelopment

CustomerFeedback

Well-StructuredTeams

Flexibility

WebsiteDesign

Privacyand Security

Fulfillmentand Reliability

CustomerService

Agile Methods Website QualityH 0 (+) = + 0.541 (0.094)

H 1 (+) = + 0.758 (0.039)

H 2 (+) = + 1.395 (0.095)

H 3 (+) = – 1.333 (0.051)

H 4 (+) = – 1.031 (0.030)

First three hypotheses are true and last two hypotheses are not true

26

• Agile survey questions exhibit reliability• Customer feedback data indicate issues• Flexibility variables need some refinement• “All” 20 agile/benefit variables correlated• Aggregate agile/quality factors correlated• Individual agile/quality factors not correlated

– Population building websites is too small– Should have used generalized quality model– Simpler research design could improve results

27

• Choose large/slow industry to study• Use cognitive interviews/pilot surveys• Use online survey ‘sites to collect data• Use web blogs instead of email surveys• Develop small/simple theory and survey• Utilize very simple single-phase surveys• Present minor incentives to respondents• Ask for a lot of help from industry leaders

28

• Published three items– Conference paper– Journal article– Textbook chapter

• Designed a unique general-purpose– Conceptual theory of agile methods– Measurement instrument for agile methods

• Collected original data and measurements– Collected original data using main instrument– Identified/used best-of-breed web quality model– Established a good foundation for future research