8 4 I E E E S O F T W A R E P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t y 0 7 4 0 - 7 4 5 9 / 0 7 / $ 2 5 . 0 0 © 2 0 0 7 I E E E

For example, imagine that a company needsto acquire a workflow system. To make an in-formed decision, the organization wants toknow the quality of the products available inthe market. From the company’s viewpoint, aworkflow system’s quality depends on manyfactors—for example, services offered (such asmechanisms to notify users about events), non-functional characteristics (such as security poli-cies implemented), and deployment character-istics (such as licensing schemes supported).

For a particular selection process, you can or-ganize selection criteria into a criteria catalog. ACC is built for a scope, which can be either a do-main (workflow systems, mail servers, antivirustools, and so on) or a category of domains (com-munication infrastructure, collaboration soft-ware, and so on). Structurally, a CC arrangesselection criteria in a hierarchical tree-like struc-ture. The higher-level selection criteria serve toclassify more concrete selection criteria, usuallyallowing some overlap. They also serve to lever-

age the CC. The lowest-level selection criteria areobservable and measurable properties that ex-hibit the target scope’s components. In our work-flow example, some lowest-level selection cri-teria are User Notification Mechanisms,Support Rules, Security Transfer Pro-tocols, Encryption Algorithms, and Li-censing Schemes Supported. The first twocriteria are children of a higher-level one, prod-uct Suitability, while the third and fourthare linked to Security. Table 1 shows some se-lection criteria from a CC for workflow systems.

It’s worth examining how a CC’s construc-tion and use fits in the overall decision processas Anthony Finkelstein, George Spanoudakis,and Mark Ryan defined it in their seminal ar-ticle.1 According to them, software selectioncomprises four activities:

1. acquiring and specifying the requirements,2. understanding the available packages,3. assessing package compatibility with regard

featureDetermining Criteria for Selecting SoftwareComponents:Lessons Learned

Software component selection1 is growing in importance. Its suc-cess relies on correctly assessing the candidate components’ qual-ity. For a particular project, you can assess quality by identifyingand analyzing the criteria that affect it.

software quality

Juan Pablo Carvallo, Etapatelecom

Xavier Franch and Carme Quer, Universitat Politècnica de Catalunya

Componentselection relies on the suitabilityand completeness of the criteria usedfor evaluation.Experiencesfrom determiningcriteria for severalindustrial projectsprovide importantlessons.

to the requirements, and4. selecting the “best” available package.

A CC is useful in the evaluation of the compo-nents (activity 2 itself), in the matching of thosecomponents with regard to the requirements(activity 3), and in making the final decision(activity 4). This process combines quantitativereasoning (for example, using a multicriteriadecision-making technique) with qualitative ar-guments (typically, managerial decisions). Also,some negotiation might be necessary when noproduct meets some requirements. More detailsabout this process appear elsewhere.2

We therefore propose CCs as the best wayto deal with selection criteria for softwarecomponent selection. However, building CCscan be cumbersome and error prone. Here, wepresent lessons we’ve learned that amelioratethese problems. These lessons come from var-ious projects in different domains and con-texts.3 We participated in seven quality-relatedprojects that resulted in 10 CCs consisting ofhundreds of selection criteria each (120 for thesmallest and 510 for the biggest). In addition,we’ve built many CCs in an academic setting.

Lesson 1: Adopt a balanced CCMost current proposals for building CCs

use a mixed approach. In this approach, youstart with a basic, solid catalog of selectioncriteria and extend and adapt the criteria tothe target scope’s needs. Such an approach’ssuccess relies on the base catalog’s quality.

ProblemLiterally hundreds of proposed selection

CCs exist, and new ones appear continually,making it hard to choose one over anotherand even harder to reconcile them. Construct-ing a suitable base catalog is difficult: it can’tbe too narrow (useless) or too specific (diffi-cult to adapt to particular projects).

SolutionWe propose using a base catalog containing

only high-level CCs, applicable to virtually allthe quality scopes. We define it as an extensionof the catalog introduced in the ISO/IEC 9126-1 standard, which is presented as a general-purpose quality model. (More details and dis-cussion of this standard appear elsewhere.2)The original standard defines a CC of two lev-els, composed of six high-level selection criteria(called characteristics), divided into 27 second-level selection criteria (subcharacteristics). Ourbase catalog adds 60 new subcharacteristics: 34in the third level and 26 in a fourth one.

M a y / J u n e 2 0 0 7 I E E E S O F T W A R E 8 5

Table 1Some selection criteria from a criteria catalog for workflow systems,

with the evaluation and comparison of two products

Selection criteria Requirements Product 1 Product 2 Comparison

SuitabilityUser As much as Email Email Product 2 betterNotification possible Message (screen) Task pageMechanisms Message (screen)

Support At least by role By role By role Both products Rules and by user By group By user satisfy requirement

By departmentBy user

SecuritySecurity S-HTTP, SSL S-HTTP S-HTTP, SSL Product 1 failsTransferProtocols

Encryption DES RSA, CAST RSA, CAST, Product 1 failsAlgorithms DES, 3DES

BusinessLicensing Per client Per concurrent user Per concurrent user Both products failSchemes Per user groupsSupported

For example, figure 1 shows the part of thebase catalog corresponding to the Function-ality characteristic. The standard decom-poses Functionality into five subcharacter-istics. Our extension decomposes three ofthem into 17 new subcharacteristics.

ObservationsWe’ve used the original ISO catalog in all

our projects, with more than satisfactory re-sults (we’ve used 100 percent of the character-istics and approximately 80 percent of thesubcharacteristics in all the cases). The onlysubcharacteristics that we haven’t always usedare the six referring to compliance that de-compose each characteristic of the ISO catalog(such as Functionality Compliance).

Forty-two percent of the new selection cri-

8 6 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e

Selection criteria: Characteristics/subcharacteristics and attributes DescriptionFunctionality ISO/IEC 9126-1 Capability of the software product to provide functions that meet stated and implied needs when the

software is used under specified conditionsSuitability ISO/IEC 9126-1 Capability of the software product to provide an appropriate set of functions for

specified tasks and user objectivesAccuracy ISO/IEC 9126-1 Capability of the software product to provide the right or agreed results or effects

with the needed degree of precisionVerifiableness Provision of the software product of resources to allow the tracking

and verification of its right or agreed results or effects

LoggingCapabilities

Effectiveness Provision of the software product of mechanisms to determine the amount of right or agreed results or effects

Interoperability ISO/IEC 9126-1 Capability of the software product to interact with one or more specified systems

DirectInteroperability

By Means ofProtocols

By Meansof APIs

IndirectInteroperability

Capability of the software product to interact with other systems by meansof indirect mechanisms

Security ISO/IEC 9126-1 Capability of the software product to protect information and data so thatunauthorized persons or systems cannot read or modify them and authorized persons or systems arenot denied access to themApplicationSecurity

Capability of the software product to provide mechanisms to prevent theaccidental or deliberate unauthorized access to system functionality

Provision of the software product of mechanisms toprevent the accidental or deliberate unauthorizedaccess to the product functionality

DataSecurity

Capability of the software product to provide mechanisms to prevent the accidentalor deliberate unauthorized access to the data managed by the software product

HistoryControl

DataVersioning

Self TestsResults

PublishedTests

Provided bytheApplication

Provided byThirdParties

Stored Data

TransmittedData

FunctionalityCompliance

ISO/IEC 9126-1 Capability of the software product to adhere to standards, conventions,or regulations in laws and similar prescriptions relating to functionality

Capability of the software product to store/provideversions of the data managedProvision of the software product of logging mechanisms

Provision of the software product of mechanisms toperform direct tests of the right or agreed results or effectsCapability according to third-party reports of right oragreed results or effects of the software product

Capability of the software product to directly interact with specified systems

Capability of the software product to directly interactwith other systems by means API libraries provided

Provision of third-party organizations of mechanisms to prevent the accidental or deliberate unauthorized access to the product functionality

Provision of the software product of mechanisms to preventthe unauthorized access to the data stored by the product

Capability of the software product to directly interactwith other systems by means of supported protocols

Provision of the software product of mechanisms to preventthe unauthorized access to the data transmitted by the product

Capability of the software product to provide a history of the changes on the data managed

Figure 1. An excerpt of our base catalog: decompositionof the ISO 9126-1 Functionality

characteristic.

teria of our base catalog appear in all the CCswe’ve built. We decided to maintain the rest ofthe selection criteria because they’re a kind ofchecklist for not forgetting any potentially rel-evant aspect. An example of such a subcharac-teristic is Functionality/Security/DataSecurity/Transmitted Data, which isn’tof interest for scopes that don’t require datatransmission.

The Suitability subcharacteristic isn’tdecomposed in our base catalog. This is be-cause we’ve observed that selection criteriathat decompose this subcharacteristic aren’tusually reused in other scopes except for veryclosely related ones.

We don’t intend the base catalog to be static.In fact, its current form is the result of an evolu-tion that occurred during our first projects, andit will likely grow with new selection criteria.

Some methodological support for extend-ing the base catalog to get the selection crite-ria appears elsewhere.2

ConsequencesConsider our base catalog as your CC’s

starting point. Look at its subcharacteristicsand decide which ones make sense for the tar-get scope and are of interest in your project’scontext. Decompose these subcharacteristicsuntil you obtain the CC, which generally oc-curs when you obtain selection criteria thatcorrespond to observable and measurableproperties of the scope (called attributes in theISO standard).

Lesson 2: Recognize the nontechnical selection criteria’s importance

Nontechnical selection criteria such as ad-ministrative, economic, or political criteria areoften significant in software selection, some-times even more important than technical se-lection criteria.

ProblemThe catalogs available in most proposals

don’t include nontechnical selection criteria.Specifically, this is true of ISO/IEC 9126-1.Not considering these criteria compromises theundertaken activity’s success. On the otherhand, considering them apart from technicalselection criteria requires managing two differ-ent sets of criteria that are actually similar innature.

SolutionWe propose enlarging the base catalog with

nontechnical selection criteria structured inthe same way as technical ones. We provide ahierarchy of 141 subcharacteristics and attri-butes that decompose three high-level charac-teristics (Supplier, Business, and Prod-uct). As a result, we obtain a comprehensivebase catalog that we call the extended ISOcatalog, which integrates technical and non-technical selection criteria. Figure 2 shows thetwo highest levels of the nontechnical part ofthe catalog, including a decomposition of theSupplier/Reputation subcharacteristic anda few metrics.

ObservationsNontechnical selection criteria have been

important in most of our projects. For exam-ple, a university rejected a technically ade-quate candidate in a requirements manage-ment tool selection project because of the poorevaluation of the selection criteria belongingto the Supplier/Support subcharacteristic.

Most nontechnical selection criteria don’t de-pend on the domain. So, we’ve included manyof them in the CC, more than technical selectioncriteria. We even defined nontechnical selectioncriteria at the attribute level, including metrics.As a result, when we used this nontechnical partin our projects, we pruned those selection crite-ria that didn’t apply to the project at hand.

The extended ISO catalog’s technical and non-technical parts overlap somewhat. For instance,the Time in Market attribute of the Historynontechnical subcharacteristic also falls under theMaturity technical subcharacteristic. Also, thereare synergies and conflicts among technical andnontechnical attributes; for example, the Para-meterization & Customization attributes in-fluence Operability ones.

ConsequencesConsider the nontechnical part of the ex-

tended ISO catalog that’s in the CC under con-struction. Remove those selection criteria thatdon’t apply, and eventually modify the defini-tion of some criteria and even include othersthat aren’t part of the catalog.

Lesson 3: Define precisely yourselection framework

ISO/IEC 9126-1 is defined on a qualityframework that embraces many concepts.

M a y / J u n e 2 0 0 7 I E E E S O F T W A R E 8 7

Consider thenontechnicalpart of the

extended ISOcatalog that’s in the criteriacatalog underconstruction.

ProblemEven when you use a widespread quality

standard such as ISO/IEC 9126-1, you mightfind some aspects that aren’t precise enough.Can characteristics and subcharacteristics bemeasured? Are hierarchies of subcharacteristicsand attributes allowed? Is overlapping allowed?Not having precise answers to these questionsmakes it difficult to be consistent over time.

SolutionStarting from the ISO/IEC 9126-1 standard

document, we’ve clarified and defined precisely

its quality framework with a UML class diagram(see figure 3) and an associated glossary of terms,detecting and correcting some problems. This ap-proach is similar to the one that Barbara Kitchen-ham, Robert Hughes, and Stephen Linkman pro-posed.4 The answers to our previous questionsbecome clear when we analyze the class dia-gram: only attributes can be measured, subchar-acteristics and attributes can be arranged intohierarchies, and overlapping is possible.

ObservationsBecause the domain experts in our projects

8 8 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e

Selection criteria: Characteristics/subcharacteristics and attributes Description Metrics

Supplier Characteristics of the supplier that can influence the quality of the software productOrganizationalStructure

Description of the organizational structure of the supplier company

Positioning andStrength

Reputation

Description of the position and orientation of the supplier company in the market

Recognition of the capability of the supplier to perform similar projects based on pastexperiences and certificationsSupplier CompanyExistence

Years of the supplier company inthe market from its foundation

Number of years:Integer

QualityProcessCertification

Certifications of the quality of theprocess followed by the suppliercompany given by recognizedcertification authorities

Qualification: (Good,Correct, Suitable)Formula for calculatingthe value from the values of the subattributes

CMM Level Capability MaturityModel Level grantedto the supplier company

CMM Level:Integer (1 ... 5)

ISO 9000 ISO 9000 Certificategranted to the suppliercompany

ISO 9000: Boolean

OtherCertificates

Other quality processcertificates

List of (Certificate, Level)Certificate: (Spice,SixSigma, ...) Level: String

ClientRecommendations

References and recommendations of thesupplier company that other clients havegiven

List of (Client, Comments)Client: String,Comments: List of String

Services Offered Description of the services offered by the supplierSupport Description of the support mechanisms offered by the supplier company

Business Characteristics of the contract among the supplier and the client that can influence the quality of the software productLicensing Schema Description of the product-licensing options Ownership Description of the aspects in relation to the intellectual property rightsGuarantees Detail of the guarantees provided over the productLicensing Costs Description of the costs components and total cost of ownership for the different licensing

options availablePlatform Cost Estimation of the cost for the required production platformImplementation Cost Estimation of implementation costs based on similar past experiencesNetwork Cost Estimation of additional costs for network operation

Product Characteristics of the commercial aspects of the software product that can influence its qualityHistory Evolution of the product since it has been offered to the clientsDeliverables Detail of the out-of-the-box and expected postimplementation deliverablesParameterization& Customization

Description of the initial effort required for the product to operate

Figure 2. An excerpt of the nontechnical part of the extendedISO catalog.

understood the class diagram well, we couldshare a common reference model. The glos-sary of terms was also a success factor forcommunication.

The UML model has other implications. Forinstance, you can’t decompose a subcharacter-istic into other subcharacteristics and attrib-utes at the same time. Also, because we don’tinclude class multiplicities, meaning that thenumber of instances of classes aren’t restricted,we could eventually add new characteristics ifsome particular project requires them.

We can enrich the framework with new con-cepts and integrate them into the original pro-posal in a clearly stated form. For instance, we’veincorporated the concept of relationships amongselection criteria such as synergy or conflict (seethe classes in gray in figure 3). These relation-ships help us understand the addressed scope.Also, we’ve defined precisely the concepts in-volved in metrics definition (types of metrics,scale, measurement units, measure instruments,and so on), although the figure doesn’t showthis.

An added benefit of the class diagram is thatit served as the basis for developing DesCOTS

(description, evaluation, and selection of COTScomponents), a tool for constructing CCs.5 (Youcan download the tool at www.lsi.upc.edu/~gessi/DesCOTS.)

ConsequencesIn a new project, use the conceptual model

and glossary to know which concepts are rele-vant for communication among domain ex-perts and technicians and to know what youcan and can’t do. Define your CCs consistentlywith respect to this conceptual model. Use theassociated tool support to populate the data-base that stores the CC.

Lesson 4: Consider the CC’s final purpose

Depending on a CC’s life span, we can clas-sify it as nonreusable or reusable (see figure 4).Nonreusable CCs appear only in specific proj-ects, and their existence is bound to them. Re-usable CCs are persistent for a certain scope;you can reuse them in many projects.

ProblemUsually, approaches for building CCs don’t

M a y / J u n e 2 0 0 7 I E E E S O F T W A R E 8 9

Criteria Catalog Selection CriteriaconsistsOf

belongsTo

{xor}

*

*

*

* parent

**

*

1

*

**

1 *

*

*

*

*

*

*

Relationship

type: InfluenceType

<<enumeration>>InfluenceType

synergyconflict...

{disjoint, complete}

Attributechild

{disjoint, complete}{disjoint, complete}

deco

mpo

sesS

ubIn

toAt

tr

deco

mpo

sesA

ttrIn

toAt

tr

deco

mpo

sesS

ubIn

toSu

b

MetricSubcharacteristicCharacteristic

DecomposableFeature

DerivedAttribute

BasicSubcharacteristic

CompoundSubcharacteristic

BasicAttribute

1

1

Class

AssociationClass

BinaryAssociation TernaryAssociation Generalization

Composition

Aggregation

**

Figure 3. A UML classdiagram for theISO/IEC-9126-1-basedquality framework.

consider this classification as relevant. So,nonreusable CCs might include aspects thataren’t of interest for the scope or might not re-fine enough aspects that are of interest. Also,reusable CCs might be too detailed or mightnot address aspects that are of general interestfor the scope.

SolutionWhen building nonreusable CCs from

scratch, we intertwine requirements engineer-ing and CC construction.6 The initial require-ments identify selection criteria of interest; re-fining these criteria might generate moreconcrete requirements. If a reusable CC for thescope exists, CC construction consists basi-cally of bridging the gap between that CC andthe requirements (refinement).

We build reusable CCs in two different ways.We can manipulate a deployed nonreusable CCto create an associated reusable model, remov-ing the project-specific parts (abstraction). Oth-erwise, we can build a reusable CC fromscratch. In this case, we base its constructionmainly on the services offered by the softwarecomponents on the market. The criteria in thecatalog should be present in most, if not all, ofthese components; however, we can include con-cepts that aren’t yet available.

ObservationsFor coarse-grained scopes, such as some

software domain in the category of business ap-plications, you’ll need to identify, organize, andmaintain dozens or even hundreds of selectioncriteria. In these cases, it’s crucial to determinein advance the CC’s objective and which partsof it are of interest.

The construction of nonreusable CCs fin-ishes once all the requirements have becomemeasurable.

If you build nonreusable CCs by refinement,it might be worth considering all the criteria, not

just the selection criteria bound to the require-ments of interest. Analyzing all the criteriamight induce the discovery of new requirementsthat you hadn’t considered before.

You can build both kinds of CCs startingfrom an existing model of the same kind (ad-aptation). For nonreusable CCs, this processwill likely compromise reusability. However,by analyzing the adapted CC, you might beable to identify very particular requirements(for example, the kind of encryption algorithmto use and the required encryption key’s num-ber of bits).

ConsequencesIn a new project, decide first which kind of

CC you need. In any case, check whether a CCfor the same domain is available. Dependingon this search’s result, determine the most ap-propriate construction process, according tofigure 4. Once you’ve finished the construc-tion of a nonreusable CC, decide whether ab-stracting it into a reusable CC is appropriate.

Lesson 5: Organize softwarescopes hierarchically

You can construct a CC for hundreds of dif-ferent scopes. The frontiers among these scopesaren’t always clear and change continually asnew organizational needs arise or technologyevolves.

ProblemTo construct a CC, you must identify the

target scope, which isn’t easy for the reasonswe just mentioned.

SolutionWe propose organizing the scopes hierar-

chically. We use classification attributes7 asclassification criteria; for each value that aclassification attribute may take, we create achild.

The root in our taxonomy (see an excerpt of itin figure 5) stands for the universal context Soft-ware Applications. It has a classification at-tribute bound, Mission of Application.

This root’s children are in turn roots of largecategories of software applications. Two ofthese categories correspond to the BusinessApplications taxonomy (for example, enter-prise-resource-planning systems and supply-chain-management systems) and SoftwareDevelopment Tools taxonomy (integrated de-

9 0 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e

From scratch

Nonreusablecriteria catalog

From scratch

Reusablecriteria catalogAbstraction

RefinementAdaption Adaption

Figure 4. Constructingnonreusable andreusable CCs.

velopment environments, requirements manage-ment tools, and so on). For these taxonomies, theMission of Application classification attri-bute has the values Support to OrganizationBusiness Processes and Support to Soft-ware Development, respectively.

Leaves of the taxonomy represent soft-ware domains, such as Mail Server Toolsor Workflow Tools.

We group similar domains into categories.For instance, the Team Support Software ca-tegory has a classification attribute Main Ob-jective that might take, among others, the val-ues Coordination of Business Processesand Management of Shared Resources thatyield nodes for the Workflow Tools and Ver-

sioning and Concurrency Control Tools

domains, respectively. We can also group cate-gories to form a multilevel taxonomy.

ObservationsSoftware domain taxonomies are usually

proposed and maintained by IT consultantcompanies, commercial Web sites, academicorganizations, and so on. We’ve used one suchtaxonomy as a source.7 We took a taxonomy(Business Applications) made by one ma-jor IT consultant company with 60 domainsand 18 categories, with an average width of4.33. After goal-oriented analysis, we ar-ranged it into a new taxonomy of 72 domains

and 48 categories, with an average width of2.78 (so, domain identification was more di-rected than before).

We don’t recommend building a whole tax-onomy from the beginning for each projectunless you have good reasons (for example,your organization already has a thoroughknowledge of the addressed market segment).Instead, make it grow incrementally with eachnew project.

We’ve used the taxonomy to enhance selec-tion criteria reuse by binding reusable CCs toits nodes (see figure 5). Each node inherits itsparent’s CC and adds those selection criteriathat are common to all its descendants’scopes. Taxonomies with reusable CCs are agood infrastructure to implement classic reuseframeworks such as the experience factory.8

ConsequencesWhen you start a project, identify the do-

main involved. To do so, you can use the clas-sification attributes to traverse the hierarchydownward until you reach the domain. In thiscase, you use the reusable CC bound to thenode as the starting point. If the search fails,you must create a node for this domain, possi-bly with some path starting at the node thatmight be considered its closest ancestor. For allthe new nodes, create reusable CCs before,during, or after the project.

M a y / J u n e 2 0 0 7 I E E E S O F T W A R E 9 1

Team SupportSoftware

CommunicationSoftware

BusinessApplications

Category

Type ofCollaboration

Type ofCollaboration =Communication

Type ofCollaboration =Coordination

ClassificationAttribute =

Value 1

ClassificationAttribute =

Value 2

MainObjective

MainObjective

=Protection

MainObjective

=SendMail

ClassificationAttribute

MainObjective

MainObjective =Managementof SharedResources

Main Objective= Coordinationof BusinessProcesses

CollaborationSoftware

WorkflowTools

Domain

MailServerTools

Versioningand ConcurrencyControl Tools

FirewallTools

Extended ISOcatalog

Criteriacatalog

Figure 5. A taxonomycorresponding to theCollaboration

Software category,which is a subcategoryof the Business Applications

taxonomy.

Some scenarios of useHere are several scenarios that might bene-

fit from our lessons learned. They are based onour experiences but have been abstracted to ob-tain a setting of interest to more people.

Call-for-tenders caseThe National Finance Department wants

to acquire document management software. Itknows exactly what type of services it needs.Owing to regulations, the bidding processmust be highly transparent. The departmentrequests its software lab to write a call-for-tenders document expressing the require-ments precisely, such that evaluation of can-didate solutions can be as objective aspossible. Because these regulations are recent,it’s the first time that the lab has had to writesuch a document.

One lab employee is an IEEE member whohas recently read an article on lessons learnedabout selection criteria. She convinces her bossto follow this approach. Figure 6a shows thisprocess. They end up with a highly structuredcall for tenders (following the CC’s layout) list-ing the measurable requirements for which thecandidate suppliers must provide information.

Product selection caseOur university’s software engineering depart-

ment wants to acquire a meeting scheduler tool.The department head has some idea of whatservices the tool must offer, but she would liketo know which other aspects of MSTs couldserve the department’s purposes. She knows thatwe’ve participated in some software selectionprojects and therefore asks us to participate asconsultants in the selection. We accept.

Figure 6b shows the selection process. Fromour previous experiences, we have an ongoingtaxonomy with the scopes identified so far; inparticular, we take advantage of projects we’veundertaken in domains similar to MST.3 Fur-thermore, our DesCOTS tool facilitates ouranalysis. We produce a comparison of severalproducts and a report on additional servicesthat you can expect from the MST. Finally, wemake the constructed CC reusable, and we up-date our taxonomy, binding this reusable CCto a new node for the MST domain.

Quality consultancy caseThe Acme consultant company has a depart-

ment specializing in consultancy on software

component quality. That department maintainsits own version of a business applications tax-onomy that was referenced in an issue of IEEESoftware. The EMCA company’s software labasks Acme to participate in the requirementselicitation of a new component-based softwaresystem for supporting its supply chain manage-ment. Because the decision to implement anSCM system was political, the EMCA SoftwareDepartment isn’t sure about which types ofcomponents it needs or which requirements itmight state over these components.

Acme accepts the consultancy and followsthe process in figure 6c. Acme uses the taxon-omy to determine the candidate scopes inwhich the client organization might be inter-ested. The CCs bound to these scopes serve toinform the client about the type of services andother expected characteristics. Also, Acme sum-marizes information about budget, licenses,and so on from the evaluation of the productsof that scope. The Software Department usesthis information to prepare a report for com-pany management describing the different op-tions and their costs.

Information brokerage caseThe Component Description Provider com-

pany offers characterization of domains throughthe Internet to people interested in knowingwhat aspects to consider when looking at someproduct in a certain domain. It offers the char-acterizations as CCs and classifies the domainsas a taxonomy, in both cases following theguidelines presented in an IEEE Software articleon CCs.

Recently, this company’s employees havebeen aware of a new domain: tools for mak-ing conceptual maps (for example, Visual-Mind and MindGenius). They have exam-ined the existing taxonomy, discovered thecategory to which this domain belongs, up-dated the taxonomy, and created a reusablecatalog. Figure 6d summarizes this process.As a result, they have their taxonomy con-stantly updated, including the latest tech-nologies, which adds value to the company’sbusiness.

T he concept of selection criteria hassimilarities with concepts such as per-sistent software attributes9 and qual-

ity characteristics.10 While these concepts focus

9 2 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e

More importantthan theconcreteartifacts

we’ve presented(the CCs, the

taxonomy, andso on) are the

underlyingideas.

Reusable criteria catalog

Understand ourselection framework

(lesson 3)

Take the extendedISO catalog

(lessons 1 and 2)

Determine whichselection criteria

of the initialcriteria catalogcorrespond to

the requirements(lesson 4)

Write the callfor tenders

Refine therelevanttechnicalselection

criteria until therequirements

aremeasurable

(lessons 1 and 4)

Prune thenontechnical

selectioncriteria that

aren't relevantfor the

requirements(lessons 2 and 4)

ExtendedISO catalog

Project criteriacatalog

Measurablerequirements

Call for tenders

Locate in the taxon-omy the target

domain's closestancestor (lesson 5)

Add the newreusable

criteria catalog tothe domain in the

taxonomy(lesson 5)

Abstract thecriteriacatalog

(lesson 4)

Determine whichselection criteriaof the departingcriteria catalog

correspond to theproject require-ments (lesson 4)

Refine therelevanttechnicalselection

criteria until therequirements are

measurable(lessons 1 and 4)

Prune thenontechnical

selectioncriteria

that aren'trelevant for therequirements

(lessons 2 and 4)

Identify newrequirementsfrom selection

criteria notrelated to theinitial projectrequirements

(lesson 4)

Evaluate availableproducts

Report services thatmight be expectedfrom the domain

Compare theproducts

Ancestorcriteriacatalog

Reusable criteriacatalog

Project criteriacatalog

(a)

C2

C0

C1

D3D2

C3

Initial taxonomy

C2

C0

C1

D3D1 D2

C3

Enlarged taxonomy

Requirements

1. –––––2. ––––– –––––3. –––––4. ––––

Additionalservices offeredby the domain

Completerequirements

(d)

(b)

Initial taxonomyStudy the domain

informationsources

Locate in thetaxonomy the

target domain’sclosest ancestor

(lesson 5)

Create a new leafcorresponding to thedomain. Take as the

departing criteriacatalog the criteriacatalog of the directancestor category

(lesson 5)

Refine the selectioncriteria until youobtain a suitabledescripton of the

domain(lessons 1 and 4)

Add the new reusablecriteria catalog to thenew domain in the

taxonomy (lesson 5)

C2

C0

C1

D3D2

C3

C2

C0

C1

D3D2D1

C3

Enlarged taxonomy

Domaininformation

sources(products,tutorials,

standards,and so on)

(c)

Criteriacatalog

Look at thedomains that de-

compose thecategory (lesson 5)

Determine whichof these domains

might berelevant for

the newsoftware system

(lesson 5)

Take thecriteria catalogcorresponding

to eachrelevantdomain

(lesson 5)

Prepare a report with options andeach option’s cost

Criteriacatalog

C2

C0

C1

D3D2D1

C3

TaxonomyLocate thecategory in

the taxonomy(lesson 5)

Report withoptions and costs

1. –––––2. ––––– –––––3. –––––

Initialrequirements

1. –––––2. ––––– –––––3. –––––

Ancestor criteriacatalog

Create a new leaf corre-sponding to the domain.

Take as the departing criteriacatalog the criteria catalog

of the direct ancestorcategory (lesson 5)

Comparisonof products

2 4 7 10 5 9 11 2 8 6 10 8 95 3 3 1 7 4 10

Figure 6. Four scenarios that benefit from the lessons learned about CCs: (a) call-for-tenders case, (b) product selection case, (c) quality consultancy case, and (d) information brokerage case.

9 4 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e

on controlling and monitoring system develop-ment and operation for quality assessment pur-poses, selection criteria are used to describe be-forehand the elements desired for the system.

More important than the concrete artifactswe’ve presented (the CCs, the taxonomy, and soon) are the underlying ideas. In other words, anorganization may use its own base catalog orarrange different kinds of taxonomies that bet-ter fit its particular needs.

We’ve been using the presented approach inour own industrial experiences with satisfactoryresults. Specifically, we’ve found its highly incre-mental nature to be very valuable. This featurewill let you construct the artifacts we’ve pre-sented as you need them for a particular project,while supporting easy knowledge transfer fromone project to the next.

AcknowledgmentsThis work has been done in the framework of the

research project UPIC TIN2004-07461-C02-01.

References1. A. Finkelstein, G. Spanoudakis, and M. Ryan, “Soft-

ware Package Requirements & Procurement,” Proc.Int’l Workshop Software Specification and Design(IWSSD), IEEE CS Press, 1996, pp. 141–145.

2. X. Franch and J.P. Carvallo, “Using Quality Models inSoftware Package Selection,” IEEE Software, vol. 20,no. 1, 2003, pp. 34–41.

3. J.P. Carvallo, X. Franch, and C. Quer, “Managing Non-Technical Requirements in COTS Components Selec-tion,” Proc. 14th IEEE Int’l Conf. Requirements Eng.(RE 06), IEEE CS Press, 2006, pp. 323–328.

4. B.A. Kitchenham, R.T. Hughes, and S.G. Linkman,“Modeling Software Measurement Data,” IEEE Trans.Software Eng., vol. 27, no. 9, 2001, pp. 788–804.

5. G. Grau et al., “DesCOTS: A Software System for Se-lecting COTS Components,” Proc. 30th EUROMICRO

Int’l Conf. (EUROMICRO 04), IEEE CS Press, 2004, pp.118–126.

6. C. Alves et al., “Using Goals and Quality Models toSupport the Matching Analysis During COTS Selec-tion,” Proc. Int’l Conf. COTS-Based Software Systems(ICCBSS), Springer, 2005, pp. 146–156.

7. J.P. Carvallo et al., “Characterization of a Taxonomyfor Business Applications and the Relationships be-tween Them,” Proc. Int’l Conf. COTS-Based SoftwareSystems (ICCBSS), Springer, 2004, pp. 221–231.

8. V.R. Basili, G. Caldiera, and H.D. Rombach, “The Ex-perience Factory,” Encyclopedia of Software Eng., J.Marciniak, ed., John Wiley & Sons, 1994, pp. 469–476.

9. T. Bollinger, J. Voas, and M. Boasson, “Persistent Soft-ware Attributes,” IEEE Software, vol. 21, no. 6, 2004,pp. 16–18.

10. J. Bøegh et al., “A Method for Software Quality Plan-ning, Control, and Evaluation.” IEEE Software, vol.16, no. 2, 1999, pp. 69–77.

For more information on this or any other computing topic, please visit ourDigital Library at www.computer.org/publications/dlib.

About the AuthorsJuan Pablo Carvallo is the manager of the Informatics department of Etapatelecom,a telecommunications company based in Cuenca, Ecuador, and a part-time teacher at AzuayUniversity’s Systems Engineering Department. His research interests include requirements en-gineering, COTS-based system development, and COTS selection, evaluation, and certification.He received his PhD in informatics from the Universitat Politècnica de Catalunya. Contact himat Calle Larga 113 y Av. Huayna Cápac, Edif. Banco Central Cuenca, Ecuador; jpcarvallo@etapatelecom.net.

Xavier Franch is an associate professor at the Universitat Politècnica de Catalunya’s De-partment of Software. His main interests include requirements engineering, COTS-based devel-opment, and software quality. He’s a member of the IEEE. He received his PhD in informaticsfrom the Universitat Politècnica de Catalunya. Contact him at UPC-Campus Nord, Edif. Omega,despatx 122, Jordi Girona 1-3, 08034 Barcelona, España; franch@lsi.upc.edu.

Carme Quer is an associate professor at the Universitat Politècnica de Catalunya’s Depart-ment of Software. Her research interests are selection of COTS components, software quality,and component-based software development. She received her PhD in informatics from theUniversitat Politècnica de Catalunya. Contact her at UPC-Campus Nord, Edif. Omega, despatx119, Jordi Girona 1-3, 08034 Barcelona, España; cquer@lsi.upc.edu.

IEEE Pervasive Computing

delivers the latest peer-reviewed develop-ments in pervasive, mobile, and ubiquitous

computing to developers, researchers, andeducators who want to keep abreast ofrapid technology change. With contentthat’s accessible and useful today, thispublication acts as a catalyst for progress inthis emerging field, bringing together theleading experts in such areas as

V I S I T www.computer.org/pervasive

SubscribeNow!

• Hardware technologies

• Software infrastructure

• Sensing and interactionwith the physical world

• Graceful integration of human users

• Systems considerations, including scalability,security, and privacy