12 Computer

T E C H N O L O G Y N E W S

I n recent years, one of the loudestongoing complaints about com-puter technology has been aboutthe poor quality of commercialsoftware. The US National Insti-

tute of Standards and Technology(NIST) estimated that in 2001, soft-ware bugs cost domestic companies$59.5 billion, about 0.6 percent of thecountry’s gross domestic product.

The Sustainable Computing Con-sortium (SCC, www.sustainablecomputing.org)—an academic, govern-ment and business IT- and software-improvement initiative led by CarnegieMellon University—said defective soft-ware costs companies more like $200billion annually.

The Standish Group, a softwaremarket research firm, reported that in2002, only 34 percent of software pro-jects were completed on budget withall features and that about 17 percentwere complete failures.

Of all products, software is amongthe most error prone, said GregoryTassey, the senior economist in chargeof the NIST study.

“We’ve been writing software forabout 50 years,” said Jim Larus, seniorresearcher at Microsoft Research,“and we still produce software with ahigh number of bugs. Our tools havegotten better but the quality of ourcode doesn’t reflect this.”

With software becoming more com-plex and products becoming increas-ingly dependent on software, bugs are

affecting more people and the level ofconsumer dissatisfaction has increaseddramatically.

With this in mind, three major companies—IBM, Microsoft, and SunMicrosystems—have started separateprojects with different focuses but thesame goal: to improve software qual-ity with better tools.

These efforts also attempt to reducedevelopment costs and cut down ontedious debugging tasks.

Each project automates the testingprocess so that programmers can findsoftware errors at early stages of thedevelopment process, when it’s lessexpensive to fix them, as Figure 1shows.

According to Alan Zeichick, editorin chief of SD Times, a software devel-opment newspaper, “The general ruleof thumb is it takes $10 to fix the bugduring development, $100 to fix thebug during QA (quality assurance),$1,000 to fix the bug during beta test-

ing, and $10,000 or more to fix thebug post-deployment.”

TROUBLED SOFTWAREStandish Group chair Jim Johnson

said many software vendors have beenmaking money despite selling productswith sloppy code since the mid-1990s.

All too often, Johnson explained,vendors write programs in a hurry tohit unrealistic deadlines, and theyaccept bug fixes as a natural part of thepost-release process.

They frequently write code primarilyto provide features, check whether theapplication will run, and then start elim-inating the most serious bugs until theprogram seems acceptable for release.Other bugs are fixed after release, asusers find them. However, this can causemany problems such as features thatdon’t function consistently and vulner-abilities that hackers can exploit.

The problem has grown worse asprograms have gotten bigger and morecomplex. In fact, there is now an entireindustry devoted to tracking and fix-ing bugs, generating $1.5 billion inrevenue annually, noted TheresaLanowitz, an analyst with GartnerInc., a market research firm.

Now, though, Johnson said, “There’sa push-back for better quality softwareboth from fed-up end users and fromthe sheer cost to vendors of [fixing]bugs.”

With this in mind, major vendors arenow working on automating andimproving the debugging process.

IBM: ECLIPSEIBM is one of the leaders of Eclipse

(www.eclipse.org), a nonprofit net-work of 40 vendors. They have createdan integrated development environ-ment (IDE) for use with C++, Java, andthe Java-based NetBeans developmentplatform.

Eclipse is presently deployed pri-marily for internal Java developmentin member companies such as Intel,Oracle, and SAP.

One of the project’s principal goals isto use the open source design process

Building BetterSoftware withBetter ToolsSteven J. Vaughan-Nichols

September 2003 13

otherwise idle clock cycles. Moreover,he added, it’s most important that thetools find problems, not run quickly.

Microsoft programmers can createplug-ins to use the tools to look forerrors applicable to the particular typeof software they are writing. For exam-ple, programmers working with devicedrivers, which must operate in real time,could look specifically for timing errors.

He said Microsoft eventually hopesto release the tools to other companies,particularly manufacturers of Windows-compatible hardware.

PREfixTo help control bugs in Windows

2000, Microsoft bought Intrinsa,maker of the PREfix debugging tool, in1999. Although the tool found manybugs, it proved to be slow and proces-sor-intensive. In response, Microsofthas developed an improved version.

PREfix examines code for patternsthat match a defined list of commonsemantic (logic) and syntactic (codinggrammar) errors.

PREfix is a compile-time tool, whichsimulates a C or C++ program in oper-ation and thus catches runtime prob-lems before the program actually runs.Because the tool simulates, rather thanruns, a program, it can be used veryearly in the development process,rather than having to wait until enoughof an application has been written forit to run properly on a system.

PREfastMicrosoft uses PREfast later in the

development cycle, before beta testing,to check for syntactic and semanticerrors, as well as mistakes in codingconventions such as using a capital let-ter for a function’s name. When multi-ple programmers don’t follow the sameconventions, the lack of consistencymakes it harder for them to maintainand continue developing an application.

Microsoft currently uses PREfast asa check-in requirement, as the toolwon’t let programmers add code withmistakes to the existing code base untilthey correct the errors.

to reduce bugs, explained IBM’s SkipMcGaughey, chair of Eclipse’s boardof stewards.

“The general feeling is that when thedesign stage is done in the open withmany people participating, the odds ofgetting a design right are orders ofmagnitude better,” McGaughey said.

Beyond using an open source ap-proach, Eclipse’s main effort to perfectcode is Hyades, an integrated test, trace,and monitoring environment. This test-ing suite is based on the ObjectManagement Group’s Unified ModelingLanguage testing profile, which includesthe Test Case, Test Trace, and TestObjective and Verification modules. TheEclipse IDE runs these modules duringthe development process.

Unlike Microsoft’s and Sun’s newtools, Hyades is meant for use at thebeta and quality-assurance stages ofsoftware development and thus isdesigned to catch higher-level imple-mentation errors, rather than just cod-ing mistakes.

“The Hyades project is significantbecause it addresses quality-assurancepractices throughout development,deployment, testing, and management,including static code analysis, func-tional testing, and performance test-ing,” said Adam Kolawa, CEO ofParasoft, a vendor of software-bug-prevention tools.

Hyades differs from most other soft-ware practices because it’s open sourceand brings a common IDE/debuggingpackage to many different platforms.

MICROSOFT: PREFIX, PREFAST, AND SLAM

As the world’s highest-profile soft-ware company, Microsoft has receivedconsiderable criticism for the qualityof its software. Microsoft has devel-oped three debugging tools for inter-nal use to address these concerns.

All three tools are designed to bethorough and thus run relativelyslowly, according to Microsoft. This isnot a big problem, though, Larus said,because developers run the tools whenpossible as a background task by using

Larus said PREfix and PREfast havebeen very effective and caught about 17percent of the bugs found in Micro-soft’s Server 2003, which he said is thehighest quality operating system thecompany has released.

SLAMMicrosoft is working on another

bug-detection tool: the software, lan-guages, analysis, and model-checkingproject (http://research.microsoft.com/slam). SLAM simulates a system run-ning a piece of software to make surea given piece of program logic willwork on every potential executionpath.

SLAM simulates all possible ways aprogram might run, detecting suchcommon bugs as those that permitbuffer overruns, which are unlikely tobe found during most testing or use.

SUN: JACKPOTSun’s open source Jackpot project

(http://research.sun.com/projects/jackpot) consists of three major parts,including the Jackpot Source CodeMetrics, to help eliminate bugs.

Source: Mercury Interactive; Siebel Systems

0

10

20

25

30

35

40

45

5

15

Erro

rs c

augh

t (pe

rcen

t)

With automated testingWithout automated testing

Earlydevelopment

Implementation

1X3-6X

10X15-40X

30-70X40-1,000X

Stage

Number of times more expensive to fix

Figure 1. Using automated testing tends tosave money by catching errors earlier inthe development process.

14 Computer

don’t need to examine, explained Sun’sMichael L. Van De Vanter, Jackpot’sprincipal investigator.

“Source code metrics used to be theprovince of management and auditingtools,” Van De Vanter said. “As a result,they tended to come somewhat later inthe development cycle and have lessimpact on the actual design of the code.”

Sun says it doesn’t have a timetableyet for implementing Jackpot on a wide-spread basis for commercial projects.

Z eichick of SD Times said the IBM,Microsoft, and Sun projects arepotentially important ways to

improve software quality. However, headded, they will find only the moreobvious coding problems, such as syn-tax errors and usage inconsistent withprogramming conventions.

They won’t help much with the mostinsidious and hard-to-catch problemssuch as logic errors and uncaught excep-tions, he noted. Finding these problemsusually requires complete understand-ing of a program, he said, which isalmost impossible with huge projects

Sun is developing the JSCM as a toolthat can examine code and identifystructural problems. JSCM is currentlyonly a beta module for use with Net-Beans, but Sun plans to extend it towork with Java outside the NetBeansenvironment, as well as with other lan-guages.

The tool includes a static codechecker that uses lexical scanning (atype of program analysis that examinesthe relationships between program-ming elements to make sure the over-all coding is correct), parsing, andsemantic and dataflow analysis.

But JSCM’s code checker, unlikesome others, also finds errors by com-paring the code against a set of metricsof what is and isn’t acceptable practicein a given language. The code checkerflags errors so that a programmer canimmediately see and repair them.

Users can set the code checker sothat it detects errors of different sever-ity levels. This will let users see onlycertain errors during a debugging ses-sion, making developers more likely touse the tool by reducing the amount oftime they spend looking at errors they

that have thousands of lines of code.According to Zeichick, new soft-

ware-development paradigms, such asextreme programming, may be neces-sary to eliminate the most severe errors.

Nonetheless, major companies aretaking some steps to deliver better code.Stated the Standish Group’s Johnson,“Overall, we’re focusing more on qual-ity, and that’s a good thing for every-one.”

In the end though, said MarkReineck, president of Nth Generation,a small software development house,“Good debugging tools are helpful foreliminating bugs, but good methodsare infinitely better.” �

Steven J. Vaughan-Nichols is a free-lance technology writer based inArden, North Carolina. Contact himat sjvn@vna1.com.

T e c h n o l o g y N e w s

Editor: Lee Garber, Computer, 10662 LosVaqueros Circle, PO Box 3014, Los Alamitos,CA 90720-1314; l.garber@computer.org

Get accessto individual IEEE Computer Society

documents online.

More than 67,000 articles and conference papers

available!

$9US per article for members

$19US for nonmembers

http://computer.org/publications/dlib