COSC 6431 1/5/06

Software Re-Engineering 1

Software Re-Engineering

COSC 6431

http://www.cs.yorku.ca/course/6431

V. Tzerpos

bil@cs.yorku.ca

Jan-5-06 COSC6431 2

Legacy Systems

• Older software systems that remain vital toan organization

• Software systems that are developedspecially for an organization have a longlifetime.

• Many software systems that are still in usewere developed many years ago usingtechnologies that are now obsolete.

Jan-5-06 COSC6431 3

Legacy Systems

• Legacy systems are still essential for thenormal functioning of the business.

• Many changes have been incorporated inthe system by many different peoplethroughout the years

• It is not unusual that no one has acomplete understanding of the system

Jan-5-06 COSC6431 4

Legacy System Replacement

• There is a business risk in scrapping a legacy

system and replacing it with a modern system:

– Legacy systems rarely have a complete

specification.

– Business processes rely on the legacy system.

– The system may embed business rules that are not

formally documented elsewhere.

– New software development is risky and may not be

successful.

Jan-5-06 COSC6431 5

Lehman’s Second Law

• “The entropy of a software system

increases with time unless specific work is

executed to maintain or reduce it”

• Lehman’s Law of Continuing Growth:

“The functional capability of most

software systems must be continually

increased to maintain user satisfaction over

the system lifetime”

Jan-5-06 COSC6431 6

Legacy System Change

• Systems must change in order to remain useful.

• Changing legacy systems is often expensive:

– Different parts of the system are implemented bydifferent teams.

– The system may use an obsolete programminglanguage.

– The system documentation is often out-of-date.

– The system structure may be corrupted by manyyears of maintenance.

– Techniques to save space or increase speed at theexpense of understandability may have been used.

COSC 6431 1/5/06

Software Re-Engineering 2

Jan-5-06 COSC6431 7

The Legacy Dilemma

• It is expensive and risky to replace the

legacy system.

• It is expensive to maintain the legacy

system.

• Businesses may choose to extend the

system lifetime using techniques such as

reverse engineering.

Jan-5-06 COSC6431 8

Example of a

Legacy Application System

File 1 File 2 File 3 File 4 File 5 File 6

Program 2Program 1 Program 3

Program 4 Program 5 Program 6 Program 7

Jan-5-06 COSC6431 9

After Re-engineering …

Database-centred System

Program1

Program2

Program3

Program4

Databasemanagement

system

Logical andphysical

data models

describes

Jan-5-06 COSC6431 10

Legacy System Design

• Most legacy systems were designed before

object-oriented development was used.

• Rather than being organized as a set of

interacting objects, these systems have

been designed using a function-oriented

design strategy.

Jan-5-06 COSC6431 11

Legacy System Assessment

• Organizations that rely on legacy systemsmust choose a strategy for evolving thesesystems:– Replace the old system with a new one.

– Continue maintaining the system.

– Transform the system by re-engineering to improve itsmaintainability.

• The strategy chosen should depend on thesystem quality and its business value.

Jan-5-06 COSC6431 12

System quality and business value

12

3 45

67

89

10

System quality

Business valueHigh business valueLow quality High business value

High quality

Low business valueLow quality

Low business valueHigh quality

COSC 6431 1/5/06

Software Re-Engineering 3

Jan-5-06 COSC6431 13

Legacy System Categories

• Low quality, low business value– These systems should be scrapped

• Low-quality, high-business value– Should be re-engineered or replaced.

• High-quality, low-business value– Replace, scrap, or maintain.

• High-quality, high business value– Continue in operation using normal system

maintenance.

Jan-5-06 COSC6431 14

Cost/Benefit factors for RE

• Current annual maintenance cost

• Current annual operation cost

• Current annual business value

• Predicted annual maintenance cost after RE

• Predicted annual operation cost after RE

• Predicted annual business value after RE

• Estimated re-engineering costs

• Estimated re-engineering time

• Expected life of the system

Jan-5-06 COSC6431 15

Software Maintenance

• Managing the processes of system

change

Jan-5-06 COSC6431 16

• The system requirements are likely to

change while the system is being

developed because the environment is

changing.

• When a system is installed in an

environment it changes that environment

and therefore changes the system

requirements.

Maintenance is Inevitable

Jan-5-06 COSC6431 17

• Perfective maintenance

– Adding or modifying the system’s functionality to

meet new requirements.

• Adaptive maintenance

– Changing a system to adapt it to new hardware or

operating system.

• Corrective maintenance

– Changing a system to fix coding, design, or

requirements errors.

Types of Maintenance

Jan-5-06 COSC6431 18

Distribution of Maintenance Effort

Perfectivemaintenance

(65%)

Correctivemaintenance

(17%)

Adaptivemaintenance

(18%)

COSC 6431 1/5/06

Software Re-Engineering 4

Jan-5-06 COSC6431 19

Evolving Systems

• It is usually more expensive to add functionality

after a system has been developed rather than

design this into the system:• Maintenance staff are often inexperienced and unfamiliar

with the application domain.

• Programs may be poorly structured and hard to understand.

• Changes may introduce new faults as the complexity of the

system makes impact assessment difficult.

• The structure may be degraded due to continual change.

• There may be no documentation available to describe the

program.

Jan-5-06 COSC6431 20

The Maintenance Process

• Maintenance is triggered by change requests

from customers or marketing requirements.

• Changes are normally batched and implemented

in a new release of the system.

• Programs sometimes need to be repaired without

a complete process iteration but this is dangerous

as it leads to documentation and programs getting

out of step.

Jan-5-06 COSC6431 21

System Documentation

• Requirements document

• System architecture description

• Program design documentation

• Source code listings

• Test plans and validation reports

• System maintenance guide

Jan-5-06 COSC6431 22

• Usually greater than development costs (2* to

100* depending on the application).

• Affected by both technical and non-technical

factors.

• Maintenance corrupts the software structure so

makes further maintenance more difficult.

• Aging software can have high support costs

(e.g., old languages, compilers etc.)

Maintenance Costs

Jan-5-06 COSC6431 23

• Module independence• It should be possible to change one module without

affecting others.

• Programming language• High-level language programs are easier to maintain.

• Programming style• Well-structured programs are easier to maintain.

• Program validation and testing• Well-validated programs tend to require fewer changes due

to corrective maintenance.

Maintenance Cost Factors

Jan-5-06 COSC6431 24

• Documentation• Good documentation makes programs easier to understand.

• Configuration management• Good CM means that links between programs and their

documentation are maintained.

• Application domain• Maintenance is easier in mature and well-understood

application domains.

• Staff stability• Maintenance costs are reduced if the same staff are involved

with them for some time.

Maintenance Cost Factors

COSC 6431 1/5/06

Software Re-Engineering 5

Jan-5-06 COSC6431 25

Maintenance Cost Factors

• Program age– The older the program, the more expensive it is to

maintain (usually) .

• External environment– If a program is dependent on its external environment,

it may have to be changed to reflect environmentalchanges.

• Hardware stability– Programs designed for stable hardware will not

require to change as the hardware changes.

Jan-5-06 COSC6431 26

• Control complexity:

– Can be measured by examining the conditionalstatements in the program.

• Data complexity:

– Complexity of data structures and componentinterfaces.

• Length of identifier names:

– Longer names imply readability.

• Program comments:

– Perhaps more comments mean easier maintenance.

Maintenance Measurements

Jan-5-06 COSC6431 27

• Coupling:

– How much use is made of other components or data

structures

• Degree of user interaction:

– The more user I/O, the more likely the component is

to require change.

• Speed and space requirements:

– Require tricky programming, harder to maintain.

Maintenance Measurements

Jan-5-06 COSC6431 28

Process Measurements

• Number of requests for corrective

maintenance.

• Average time taken to implement a change

request.

• Number of outstanding change requests.

• If any or all of these is increasing, this may

indicate a decline in maintainability.

Jan-5-06 COSC6431 29

Software Re-engineering

• Reorganizing and modifying existing

software systems to make them more

maintainable.

Jan-5-06 COSC6431 30

Forward engineering and re-engineering

Understanding andtransformation

Existingsoftware system

Re-engineeredsystem

Design andimplementation

Systemspecification

Newsystem

Software re-engineering

Forward engineering

COSC 6431 1/5/06

Software Re-Engineering 6

Jan-5-06 COSC6431 31

• When system changes are mostly confined

to part of the system then re-engineer that

part.

• When hardware or software support

becomes obsolete.

• When tools to support re-structuring are

available.

When to Re-engineer

Jan-5-06 COSC6431 32

Re-engineering Advantages

• Reduced risk

– There is a high risk in new software

development.

• Reduced cost

– The cost of re-engineering is often

significantly less than the costs of developing

new software.

Jan-5-06 COSC6431 33

Re-engineering Cost Factors

• The quality of the software to be re-engineered.

• The tool support available for re-engineering.

• The extent of the data conversion which isrequired.

• The availability of expert staff for re-engineering.

Jan-5-06 COSC6431 34

The re-engineering process

Reverseengineering

Programdocumentation

Datareengineering

Original data

Programstructure

improvement

Programmodularisation

Structuredprogram

Reengineereddata

ModularisedprogramOriginal

program

Source codetranslation

Jan-5-06 COSC6431 35

Source Code Translation

• Involves converting the code from one language(or language version) to another e.g., FORTRANto C

• May be necessary because of:

– Hardware platform update

– Staff skill shortages

– Organizational policy changes

• Only realistic if an automatic translator isavailable.

Jan-5-06 COSC6431 36

Reverse Engineering

• Reverse Engineering is the process of

determining how a system works by analyzing its

internal constituents and/or its external behavior.

• In the software world one would say that reverse

engineering is trying to figure out how a system

works by:– Inspecting the source code and documentation (if it exists)

– Exercising the executable programs and observing their behavior.

COSC 6431 1/5/06

Software Re-Engineering 7

Jan-5-06 COSC6431 37

The Reverse Engineering

Process

Data stucturediagrams

Program stucturediagrams

Traceabilitymatrices

Documentgeneration

Systeminformation

store

Automatedanalysis

Manualannotation

System to bere-engineered

Jan-5-06 COSC6431 38

Reverse Engineering

• Reverse engineering often precedes re-engineering but is sometimes worthwhilein its own right

– The design and specification of a system maybe reverse engineered so that they can be aninput to the requirements specification processfor the system’s replacement.

– The design and specification may be reverseengineered to support program maintenance.

Jan-5-06 COSC6431 39

Why is Reverse Engineering

Important/Necessary?

• Most software that is developed is not

“from scratch”.

• Understanding someone else’s source

code, specifications, designs, is difficult.

– Why is this so?

– What makes software more difficult to

understand than a toaster or a car?

Jan-5-06 COSC6431 40

Software Maintenance Problem

• A company hires a bright software developer to

maintain a system.

• The project manager points the developer to a

source code directory and says “become an

expert in the system as soon as possible”.

• The IBM TOBEY back-end compiler project

allowed for a 1 year learning curve (but this is

quite rare).

Jan-5-06 COSC6431 41

Program Structure Improvement

• Maintenance tends to corrupt the structureof a program. It becomes harder tounderstand.

• The program may be automaticallyrestructured to remove unconditionalbranches.

• Conditions may be simplified to makethem more readable.

Jan-5-06 COSC6431 42

Program Modularization

• The process of re-organising a program so

that related program parts are collected

together in a single module.

• Usually a manual process that is carried

out by program inspection and re-

organization.

COSC 6431 1/5/06

Software Re-Engineering 8

Jan-5-06 COSC6431 43

Recovering Data Abstractions

• Many legacy systems use shared tables and

global data to save memory space.

• This causes problems because changes have a

wide impact in the system.

• Shared global data may be converted to objects:– Analyse common data areas to identify logical abstractions.

– Create an object for these abstractions.

– Find all data references and replace them with reference to the

data abstraction.

Jan-5-06 COSC6431 44

Data Re-engineering

• Involves analyzing and reorganizing the data

structures (and sometimes the data values) in a

program.

• May be part of the process of migrating from a

file-based system to a DBMS-based system or

changing from one DBMS to another.

• Objective is to create a managed data

environment.

Jan-5-06 COSC6431 45

Data Problems

• Data naming problems– Names may be hard to understand. The same

data may have different names in differentprograms.

• Field length problems– The same item may be assigned different

lengths in different programs.

• Hard-coded literals

Jan-5-06 COSC6431 46

Reverse Engineering Research

• The focus has been primarily on the

development of tools to help software

developers understand software quicker

and with less effort.

• Not much work has been done on reverse

engineering methods, however.

Jan-5-06 COSC6431 47

Sherlock Holmes Analogy

• “We have developed good detective tools

(e.g., magnifying glasses, fingerprint

matchers, etc) but we have little insight on

how to train someone to be a good

detective (e.g., guidelines, processes, etc)”

S. Mancoridis

Jan-5-06 COSC6431 48

Progress Has Been Made In …

• Source code analysis

• Program tracing and profiling

• Automatic modularization (software

clustering)

But still a research area in its infancy …

COSC 6431 1/5/06

Software Re-Engineering 9

Jan-5-06 COSC6431 49

Lecture schedule

• Jan 04: Introduction, administrivia

• Jan 11: Static and dynamic analysis,

program representation, etc.

• Jan 18: Software clustering

• Jan 25: Evaluation of clustering techniques

• Feb 01: Intro to Design Patterns

• Feb 08: Design Pattern Detection

Jan-5-06 COSC6431 50

Lecture schedule

• Feb 22: Mining Software Repositories

• Mar 01: Refactoring

• Mar 08: Re-Engineering Patterns

• Mar 15: Special topics (e.g. data reverseengineering, binary reverse engineering,visualization)

• Mar 22: Research paper presentations

• Mar 29: Project presentations

Jan-5-06 COSC6431 51

Grading

• 10% - Participation (paper discussion)

• 20% - Assignment

• 20% - Research paper presentation

• 25% - Project report

• 25% - Project presentation

Jan-5-06 COSC6431 52

Workload

• January: 2 papers a week

• February: Assignment + 1 paper a week

• March: 1 paper to present + project