Product Software Development

Course Product Software

Dr. Slinger Jansen

Aim: To provide insight into the multitude of methods that exist for product software developers.

Artikel: Pragmatic Development

Agenda

Product software development experiences The role of software architecture

What are the differences between proDUct software development and proJEct software development?

Differences Project and Product Software

Product Software Architecture plays

strong part Variability Many releases Long lifetime Many developers

throughout lifetime …

Project Software Project, deadline, and

investment play a strong part

Some parts people never touch again

Few releases Many developers for

first release, after that maintenance

…

Hybrids are also possible

Software product lines Software implementation projects Etc.

Software Product Lines

DEVELOPING SOFTWARE

As organizations’ reliance on software grows, so do the business-related consequences of software successes and failures including: – Increase or decrease revenue –Repair or damage to brand reputation –Prevent or incur liabilities – Increase or decrease productivity

THE SYSTEMS DEVELOPMENT LIFE CYCLE (SDLC)

Systems development life cycle (SDLC) – the overall process for developing information systems from planning and analysis through implementation and maintenance

THE SYSTEMS DEVELOPMENT LIFE CYCLE (SDLC)

1. Planning phase – involves establishing a high-level plan of the intended project and determining project goals

2. Analysis phase – involves analyzing end-user business requirements and refining project goals into defined functions and operations of the intended system

• Business requirement – detailed set of business requests that the system must meet in order to be successful

THE SYSTEMS DEVELOPMENT LIFE CYCLE (SDLC)) 3. Design phase – involves describing the

desired features and operations of the system including screen layouts, business rules, process diagrams, pseudo code, and other documentation

4. Development phase – involves taking all of the detailed design documents from the design phase and transforming them into the actual system

THE SYSTEMS DEVELOPMENT LIFE CYCLE (SDLC)

5. Testing phase – involves bringing all the project pieces together into a special testing environment to test for errors, bugs, and interoperability and verify that the system meets all of the business requirements defined in the analysis phase

6. Implementation phase – involves placing the system into production so users can begin to perform actual business operations with the system

THE SYSTEMS DEVELOPMENT LIFE CYCLE (SDLC)

7. Maintenance phase – involves performing changes, corrections, additions, and upgrades to ensure the system continues to meet the business goals

SOFTWARE DEVELOPMENT METHODS

There are a number of different software development methodologies including:

– Agile – Waterfall – Rapid application development (RAD) – Extreme programming – Rational unified process (RUP) – Scrum

Waterfall Methodology

Waterfall methodology – an activity-based process in which each phase in the SDLC is performed sequentially from planning through implementation and maintenance

Agile Methods

Agile methods – aims for customer satisfaction through early and continuous delivery of components developed by an iterative process

– An agile project sets a minimum number of requirements and turns them into a deliverable product

– Iterative development – consists of a series of tiny projects

Rapid Application Development Methodology (RAD)

Rapid application development methodology (RAD) – emphasizes extensive user involvement in the rapid and evolutionary construction of working prototypes of a system to accelerate the systems development process

The prototype is an essential part of the analysis phase when using a RAD methodology

– Prototype – a smaller-scale representation or working model of the users’ requirements or a proposed design for an information system

Rapid Application Development Methodology (RAD)

Fundamentals of RAD – Focus initially on creating a prototype that

looks and acts like the desired system – Actively involve system users in the

analysis, design, and development phases – Accelerate collecting the business

requirements through an interactive and iterative construction approach

Extreme Programming Methodology Extreme programming (XP) methodology – breaks a

project into tiny phases, and developers cannot continue on to the next phase until the first phase is complete

Rational Unified Process (RUP) Methodology

Rational Unified Process (RUP) – provides a framework for breaking down the development of software into four gates – Gate One: Inception – Gate Two: Elaboration – Gate Three: Construction – Gate Four: Transition

SCRUM Methodology

SCRUM – uses small teams to produce small pieces of deliverable software using sprints, or 30-day intervals, to achieve an appointed goal

Under this methodology, each day ends or begins with a stand-up meeting to monitor and control the development effort

Implementing Agile Methodologies

The Agile Alliance Manifesto – Early and continuous delivery of valuable

software will satisfy the customer – Changing requirements are welcome – Business people and developers work

together – Projects need motivated individuals – Use self-organizing teams – Reflect on how to become more effective

DEVELOPING SUCCESSFUL SOFTWARE

Primary principles for successful agile software development include:

– Slash the budget – If it doesn’t work, kill it – Keep requirements to a minimum – Test and deliver frequently – Assign non-IT executives to software

projects

What is Product Software?

A software product is defined as a packaged configuration of software components

or a software-based service, with auxiliary materials, which is released for and traded in a specific market.

So, what is 1. The application 2. The market 3. The software components 4. The package

Potential Tradeoffs

Quality (Reliability, Adaptability, Architecture)

Scope (Features) (Functionality,

Innovation)

Time (Cost…?) (Speed, Staffing, but Brook’s Law)

28

Hewlett Packard Study (2003 IEEE Software, “Tradeoffs” article)

74% of variation in defects explained by whether projects used (1) early prototypes, (2) design reviews, and (3) integration/regression testing on each build – Releasing prototype earlier than median functionality 27%

reduction in defect rate and 35% rise in LOC productivity – Integration/regression testing at each code check-in 36%

reduction in defect rate – Design reviews 55% reduction in defect rate – Daily builds 93% rise in LOC output/programmer

29

Conclusions from HP Study Best “nominal” quality from traditional “waterfall” (fewer

cycles & late changes allowed = less bugs, of course!! As in Japan!) Best “business” balance of quality, flexibility, and cost/speed

from combining conventional & iterative/agile practices NOTE: Differences in quality between waterfall &

iterative DISAPPEAR if use a bundle of techniques: 1. Short development subcycles (subprojects/milestones) 2. Early prototypes to get customer feedback 3. Frequent builds to incorporate feedback, changes 4. Design/code reviews (check quality continuously) 5. Regression tests on each build (check for errors, late changes,

integration problems)

30

Advice in Research Update (2009 IEEE Software, “Critical Decisions” article)

First stage of a project: Do process, not product design! – Main choices (waterfall vs. iterative/agile, QC level) depend on

the project context – If high market uncertainty, then focus on iterative prototypes &

beta releases – If high platform/technology uncertainty, then focus on building

an adaptable architecture Strong process (e.g., CMM 3+, or well-defined &

repeatable Agile) can overcome negatives of offshore development, but tradeoffs: – For every 1% increase in outsourcing of tasks, decrease in

productivity by 0.7% and quality by 1.5% (more defects)

31

Project Management Strategy

Avoid sequential “waterfall” schedules (usually). Divide long projects into multiple sub-cycles or

milestones, of a few weeks or months duration. Evolve requirements incrementally: try to do spec

refinement, coding & testing concurrently. Impose a few rigid rules to force frequent

synchronizations and periodic stabilizations.

Synchronize-and-Stabilize! 32

Synchronize-&-Stabilize Product Vision/Architecture Design Functional Specification Milestone 1 Milestone 2 Milestone 3 Iterative Design Iterative Design Iterative Design Code Code Code Usability test Usability test Usability test Test Test Test Daily builds Daily builds Daily builds Test Test Test Redesign Redesign Redesign Debug Debug Debug Integrate Integrate Integrate Stabilize Stabilize Stabilize Buffer time Buffer time Buffer time Alpha release Beta release Feature complete Beta release Visual freeze CODE COMPLETE Final test Final debug Stabilize Final release

Synch-and-Stabilize Process (Microsoft, Netscape, Hewlett Packard, et al. )

33

Testing and QA Strategy Try to build-in quality continuously

– design and code reviews – gates or check points – continuous customer feedback, etc.

But continually test, fix, and integrate – Frequent builds/working versions (daily, weekly) – Especially check late design changes & bug fixes – Integration testing from as early as possible

Automate! Automate! Automate! – To test & retest design changes quickly and frequently. – But automation never eliminates the need for people. Someone

has to write and rewrite (update) the automated tests! – Also need human testers to probe real user behavior

34

Concluding Comments No “one best” software development process

– “Waterfall” easier to visualize & control, but agile/iterative much more responsive to change, the business, innovation

– But using a “bundle” of good practices eliminates differences in quality between waterfall and iterative!

– Global teams complicate iterative process & quality, so need more attention to process & product architecture (modularity)

Match process with project/business context

– Type of software, customer requirements, team experience and culture, contract needs, etc.

– Product vs. custom projects, mass-market vs. niche, individual vs. enterprise, leading-edge vs. follower, etc.

35

Key Product-Service Differences Products: schedules, features, technologies, and price

set by negotiation with product managers or executives (based on competition, partners, channels, etc.) – Prototypes and iterative design & development useful to refine

features and get potential user feedback during the project. – Waterfall may work fine if requirements are well understood,

or if features can be frozen early

Custom: schedules, requirements, price, and technologies set through negotiation with the customer. – Prototypes and iterative design & development useful to

understand and implement customer requirements. – Problem of fixed-price contracts – need a preliminary or

staged project to negotiate project scope and increments. But customers often want a waterfall process for the contract.

36

Special Challenges for Startups S. Blank, The Four Steps to the Epiphany (2006) Startups and new business units in particular need an iterative

process to discover who their customers should be and exactly how to design their product or service (e.g., for an existing vs. new market, or a specific positioning – low cost vs. niche).

Skilled product development alone not enough. Development process and organization building needs to vary. Entrepreneurs must also formally evolve the sales & marketing functions, and others, along with the product or service, in parallel.

The Four Steps: (1) Customer Discovery (who really needs your solution?), (2) Customer Validation (sales and marketing roadmap), (3) Customer Creation (how drive more sales based on early success), (4) Company Building (formalization, $$) 37

Software start-up study

Erran Carmel: 1. A Process Model for Packaged Software Development 2. Time-to-completion in software package startups

Study of 12 starting product companies software

Investigation of successful approaches in the acceleration of software production

In tailor-made software world: “Software is always late”

Papers on methods for Product Software companies This presentation:

• Carmel, E. and Sawyer, S., (1998) "Packaged Software Teams: What Makes Them So Special?," Information Technology & People, 11(1), 6-17

• Carmel, E. (1995). Time-to-completion factors in packaged software development. Information & Software Technology Vol. 37, Num. 9, 1995, pp. 515-520

Other:

• Carmel, E. (1995). Cycle Time in Package Software Firms. J. Prod Innov. Manag., 1995:12:110-123

• Carmel, E. (1996). American hegemony in packaged software trade and the “culture of software”. The Information Society, Volume 12 number 4

• Sawyer, S. and Guinan, P., (1998) "Software Development: Processes and Performance," IBM Systems Journal, 37(4), 552-569

• Erran Carmel and Shirley Becker. A Process Model for Packaged Software Development. IEEE Transactions on Engineering Management, vol.42, no. 1, pp.50-61, 1995.

• Helms, R.W., Brinkkemper, S., Oosterum, J. van, & Nijs, F. de (2005). Knowledge Entry Maps: structuring of method knowledge in the IT industry. In Y Kiyoki, H Kangasslo, H Jaakkola, & J Henno (Eds.), Proceedings of the 15 European Japanese Conference on Information Modelling and Knowledge Bases (pp. 12-25). Amsterdam: IOP Press.

• Klooster, Marnix, Sjaak Brinkkemper, Frank Harmsen, Gerard Wijers, Intranet Facilitated Knowledge Management: A Theory and Tool for Defining Situational Methods. In: A. Olivé and J.A. Pastor, Advanced Information Systems Engineering. Proceedings of the 9th International Conference CAiSE'97, Barcelona, Spain, Lecture Notes in Computer Science 1250, pp. 303-317, Springer Verlag, June 1997.

How nature protects weak products

http://lazybastard.ehuna.org/files/animation/dilbert-beta.swf

Process model for product software

Formulating idea and prototyping

Design and code until quality threshold is reached

Idea

Requirements loop

Frozen specifications

Quality loop

General release

Requirements loop

Idea

Preliminary requirements

Initial screening

Initial reqs and technical specs

Build prototype and refine

Internal review

External review

A

A

Freeze specifications

Concept Go/No GO

Satisfaction decision point

Frozen specifications

Quality loop

B

Frozen specifications

Design

Code Complete product

QA Point B

Alpha test Alpha tested product

QA Point B

Defect removal

General Release C

C

Beta test Beta tested product

QA Point B

Defect removal

Final testing

Manufacturing

Background

Time-based competition in business

Time-to-completion (time-to-market) is: measure of calendar time from beginning of product

development to product release Release of new car model: 36 months Release of new printer: 22 months Release of new software product: ?

Justification: – Fast pay-back of R&D investment – Faster cycles of product renewal

What about product software?

Productivity in software

Definitions Productivity = effort per time unit * person

Effort ≅ Lines of Code (LOC) Standard productivity: 20 LOC /personday In order to be predictable for future releases it is

imperative to collect data from the beginning Professional atmosphere Not used against persons Precision of data is not an issue

Productivity accelerators

Literature review 1.Development method (life cycle model):

• water-fall: complete stages • spiral: start from core release • incremental: build architectural chunks • evolutionary: adapt according changing wishes

2.Development team

• individual characteristics • group characteristics

3.Resources

• labor • tools, infrastructure, • office space, environment, social

Productivity accelerators (2)

4. Project Management: • managing people, milestones, resources and tasks • style and tools

5.Risk Analysis

• identifying, assessing, and managing risks

6. Incremental Innovation • changes from one product to the next

1 and 2 in software research Others from general product development

Research method

12 product software firms

Revenue: 1 – 12 M$

US: Virginia – Maryland area

Products for PC

Selected because of: – Breadth of product offering

– Product complexity

– Age of firm

– Minimal success: products were purchased

Interviews with key developers (18)

Survey to team members (30 out of 48)

Firm characteristics Firm # 1 2 3 4 5 6 7 8 9 10 11 12

Application Business/Office √ √ √

Specialized √ √ √

Systems / Utilities √ √ √ √

Graphics √ √

Distribution Mass market √ √ √ √

Wide √ √ √ √ √

Narrow/high end √ √ √

Development time (in months)

Data were rough estimates Multiples of 12 Hardly a time strategy, when asked Time-to-completion unknown Deadlines not so important Competitive pressures Market pressures

Firm First product

Subsequent product(s)

1 ? 24

3 26 In development

4 5 24

5 13 None

6 36 11,12,13,13

7 ? 6,8,12,2

8 ? 19,10,9

9 48 18

11 24 18,36

12 38 9

Avg. 26.0 14.3

Acc. 1: Development method

Development method

Degree of implementation

# Firms

Specifications None Informal, brief, a few pages Formal specifications

2 7 3

Design Informal, brief, minimal Formal and distinct stage

8 4

Prototyping No data None Little or rarely Prototyping actively used

1 3 1 7

Testing Informal or non-standard Formalized phases

6 6

Development method findings

Accidental life-cycle model: – custom system for one client is turned into software product – Methodical approach after release 1.0

Entrepreneural nature of firm – Shortage of money – Critical events – Method not in culture

Importance of software architecture – Not on paper – Known by every team member

Use of Prototyping – Proof-of-concepts, Mock-up, demos – Decreases time-to-completion – Increases quality

Overlapping phases – Poor planning (or perhaps more agile?)

Growth stage model

Idea

Version 1.0

10 employees

20 employees

Inception

First steps

Transition

Rapid growth

Development method

Ad hoc

Necessary steps Config. Mgmt Error sheets

Elements of formal method

Organization

Core team forms

Non-development staff added

Additional development staff added

Quality assurance

Minimal

Some

Formalization begins

# firms in this study

0

5

3

4

Acc 2: Development Team

Essential factor

Core team – Homogeneous

– Highly motivated

– Highly experienced

– History of working together

– Core team • Architect

• Principal programmers

• Cohesive

• Dense communication

• Whole duration

Size of team related to age Firm

Years since

release of first product

Total number of employees

Company

Development

Staff

Core team

5

<1

6

3

3

12

2

8

3

3

8

2

8

4

3

3

2

5

3

3

9

3

13

7

4

2

3

20

6

6

1

4

10

6

3

4

5

45

5

4

6

6

70

45

4

11

6

57

35

6

7

7

50

11

4

10

9

20

11

5

Avg.

4.2

26

11.6

4

Core team findings

Size – About 4

– No growth during growth of company

Composition – Most between 20 and 30

– Male, American

– Varying education: from high school to PhD

– Very high motivation levels

– Stock options

– Good wages

Quotes

There are six of us who know the intimate details of what the product does

There are at most 100 pages of design … and no one knows where that is anymore – it’s in peoples’ brains

The core team members were in one room – like one brain – no phone calls, no interruptions

All we need is at most 4-5 developers, even if we get to be a $10 million company. We couldn’t survive with any more.

More on the core team

Experience – Computer science – Application domain essential – 13 years experience in programming

Duration of co-work – Before joining core team

Team structure – Amorphous – Some with strong player – Others no dominant leader

Acc. 3: Resource allocation

Labor is a flexible resource – Hire temp help

– Relocate project members

Finding: no labor added – Extra time of team

– 87% work more than 56 h/week

– 47% work more than 71 h/week

Capital investments

Little investments in tools – Non used design tools

– Less than 1000 USD / year * developer

– Freeware/shareware utilities

– Trial versions from customers/vendors

In-house developed utilities – Test software

– More investment than on outside tools

Distrust in others’ software

Acc. 4: Project Management

Usage of PM – 6 companies use formal PM tools, scheduling,

timelines.

– 1 (of these) used PM tool all of the time

– Others use nothing

Meetings – 1 or 2 meetings per week for key developers

– Coordination in less formal channels

Acc. 5: Risk Analysis

No systematic approach to risk analysis

Risk management at the end of project – Reduction of time to completion

– Quality versus functionality

No formal policies

Acc. 6: Incremental Innovation

Many firms applied incremental innovation

Product families – Closely related products

– Derivation from other products

– New versions regular time intervals

– Manageable updates (service packs)

Niche innovation in their markets – Established new niches

Additional accelerator: Quality

Testing is complex and consumes time and resources

Testing was compromised

Quality assurance: making sure that a product fulfills certain prescribed quality properties – QA was absent

– Weak methodology

– Lost in discussions on time and functionality

Lessons from this study

Core team is the accelerator with the most impact on the time-to-completion – Recognized as an asset in companies – Cross functional, small size, selective staffing, motivation, full-

time involvement – History of working together – Homogeneous, highly motivated, in-depth experience

Other accelerators present – Use of extra effort at the end of project – Building similar products – Remaining accelerators are weak

Weak awareness of time-to-completion – Deadlines were soft – Hardly a strategy for development speed – Poor recall of time spend on project

Implications for software start-ups

Founders of start-ups – Creation of successful core team – One talented developer not enough – Interesting product ideas and features not enough – Well-mixed, experienced team

How about Europe and Asia?

– Cultural aspects – Working attitude – Seniority versus egalitarian

What is Architecture?

A high-level model of a thing – Describes critical aspects of the thing – Understandable to many stakeholders

• architects, engineers, workers, managers, customers – Allows evaluation of the thing’s properties before it is built – Provides well understood tools and techniques for constructing the

thing from its blueprint

Which aspects of a software system are architecturally relevant?

How should they be represented most effectively to enable stakeholders to understand, reason, and communicate about a system before it is built?

What tools and techniques are useful for implementing an architecture in a manner that preserves its properties?

Motivation

Software systems are rapidly and continously growing in size and complexity

Techniques and tools for developing and maintaining such systems typically play

catch-up

To deal with this problem, many approaches exploit abstraction

– Ignore all but the details of the system most relevant to a task (e.g., developing the user

interface or system-level testing

– This greatly simplifies the model of the system

– Apply techniques and tools on the simplified model

– Incrementally reintroduce information to complete the “picture”

Software architecture is such an approach

– Applicable to the task of software design

What is Software Architecture?

A software system’s blueprint – Its components – Their interactions – Their interconnections

Informal descriptions – Boxes and lines – Informal prose

A shared, semantically rich vocabulary – Remote procedure calls (RPCs) – Client-Server – Pipe and Filer – Layered – Distributed – Object-Oriented – Service Oriented

From Requirements to Architecture From problem definition to requirements specification

– Determine exactly what the customer and user want – Specifies what the software product is to do

From requirements specification to architecture

– Decompose software into modules with interfaces – Specify high-level behavior, interactions, and non-functional

properties – Consider key tradeoffs

• Schedule vs. Budget • Cost vs. Robustness • Fault Tolerance vs. Size • Security vs. Speed

– Maintain a record of design decisions and traceability – Specifies how the software product is to do its tasks

Why Software Architecture?

A key to reducing development costs – Component-based development philosophy – Explicit system structure

A natural evolution of design abstractions – Structure and interaction details overshadow the choice of algorithms and

data structures in large/complex systems

Benefits of explicit architectures

– A framework for satisfying requirements – Technical basis for design – Managerial basis for cost estimation & process management – Effective basis for reuse – Basis for consistency, dependency, and tradeoff analysis – Avoidance of architectural erosion

Components

A component is a unit of computation or a data store.

Components are loci of computation and state. – Clients – Servers – Databases – Filters – Layers – Abstract Data Types (ADTs)

A component may be simple or composite. – Composite components describe a system.

Connectors

A connector is an architectural element that models: – Interactions among components – Rules that govern those interactions

Simple interactions

– Procedure calls – Shared variable access

Complex and semantically rich interactions

– Client-Server Protocols – Database Access Protocols – Asynchronous Event Multicast – Piped Data Streams

Scope of Software Architectures

Every system has an architecture

Details of the architecture are a reflection of system requirements and trade-offs made to satisfy them

Possible decision factors – Performance

– Compatibility with legacy software

– Planning for reuse

– Distribution profile • Current and Future

– Safety, Security, Fault tolerance requirements

– Evolvability Needs • Changes to processing algorithms

• Changes to data representation

• Modifications to the structure/functionality

Sources of Architecture (1)

Architecture comes from “black magic, people having ‘architectural visions’”

3 + 1 main sources of architecture – Intuition – Method – Theft (i.e., reuse) – Blind luck

Their ratio varies according to: – Architect’s experience – System’s novelty

Theft MethodIntuition

Theft MethodIntuition

Sources of Architecture (2)

Theft –From previous similar systems –From literature

Method –Systematic and conscious –Possibly documented –Architecture is derived from requirements via transformations and heuristics

Intuition –“The ability to conceive without conscious reasoning.” –Increased reliance on intuition increases the risk

Client Server

Service Oriented

Services

Source: IBM

Mon

olith

s

Stru

ctur

ed

Clie

nt/S

erve

r

3-Ti

er

N-T

ier

Dis

trib

uted

O

bjec

ts

Com

pone

nts

Serv

ices

Peer 2 Peer

Peer 1 Peer 2

Platform Architecture

iPhone Architecture

Software Architecture has implications for the Business Model

Think for instance of: – Plug-in architectures (Eclipse, AutoCAD, MS CRM,

etc.) – SaaS (Exact online, Reeleezee, Yunoo, etc.) – App Stores (Android, BlackBerry, PS3, etc.) – Data intensive client server architectures (SAP,

Oracle, JDEdwards, etc.)

Conclusions

Software product development is different from a single project

Hybrids are possible Software architecture is fundamental to a

software product’s future A core development team can make or break a

product The software architecture influences the

business model