AMS Special Topics

Writing DecisionPapers

Designing a system, analyzing aclient's operations, planning a computercenter upgrade—decisions, decisions!Software developers, consultants, andmanagers at AMS often need to evaluatealternative courses of action and make arecommendation.

It has been a time-honored traditionat AMS to write a paper when confrontedwith an issue that doesn't have an obvioussolution. Once called an issue paper, nowmore commonly referred to as a decisionpaper, it serves to clarify the issue, docu-ment the analysis, and provide a soundbasis for making a decision.

This is a paper on how to write decisionpapers. It originated with a study that theauthor, Milt Hess, performed in 1981 for alarge AMS development project. Tasked toevaluate alternative database managementsystems for the project, he formulated theapproach that this paper describes. Pleasedwith the results, he wrote the initial version ofthis paper and circulated it to a few people,who found it useful. The original version hasbeen revised several times to clarify theconcepts and to provide specifics on how toexecute the analysis steps, and these earlierversions have been distributed and usedwidely within AMS.

Writing Decision Papers

Perhaps your client has asked for a strategy tocut the cost of running the inventory system. Or,your project manager has asked for a recommenda-tion on a software package for the new financialsystem. How will you organize the study, identifyand analyze the options, and present your recom-mendations?

You know that you will have to write a paperof some sort. Such papers are variously called issuepapers, working papers, or decision papers. I preferthe term decision paper because it is more resultsoriented.

Based on my experience reading and writingdecision papers, I have found a strategy that consis-tently produces good results. The strategy is basedon a few key ideas:

• Identify a rich set of options by exploring thedesign space,

• Evaluate all options against the same criteria,

• Identify the discriminators—the criteria that mostsharply differentiate the options, and

• Reduce the analysis to a clear choice among theoptions based on the discriminators.

Following a discussion of the defects thatdecision papers often contain, I describe the recom-mended strategy for performing the analysis andwriting the report.

The Usual ApproachesSometimes, issues are analyzed and docu-

mented using a straightforward three-step ap-proach:

1. Define the options,

2. Describe the strengths and weaknesses of eachoption, and

3. Draw conclusions and present a recommenda-tion.

This approach has a serious defect whichderives from the failure to explicitly define theevaluation criteria. When the analyst focuses only onthe highlights for each option, he often leaves the

reader with the uncomfortable feeling that someimportant issue has either been omitted or givenshort shrift. His conclusions sound like a compari-son of apples and oranges, and they are usuallyexpressed in vague, unquantified terms.

Another common approach, also unsatisfac-tory, goes to the opposite extreme. The analystidentifies a large number of criteria and assigns anumerical weight to each to signify its importance inthe analysis. He then evaluates the options andassigns a numerical rating for how well each optionmeets each criterion. He then multiplies the ratingsand the weights, and sums them to derive a score foreach option. The best score wins.

Although rigorous, this is in fact an arbitraryapproach that replaces analysis by arithmetic. It isbased on two assumptions that are rarely true and,in any case, impossible to validate:

1) That the weights accurately reflect the relativeimportance of the criteria,

2) That the weighted sum of the ratings is a mean-ingful measure of the option.

These strategies are subject to yet anotherdeficiency. They fail to address the sensitivity of theresults to the assumptions. For example, whathappens in a cost/benefit analysis if the estimatedrate of inflation is wrong? The best choice may bethe option that minimizes the downside exposure,not the one that offers the most benefits if everythingworks perfectly.

The Goal of a Decision PaperThe goal of a decision paper is not necessarily a

single answer. After all, if you are asked to write adecision paper you are probably not the person whohas to make the decision. The goal of your paper isto provide the decision maker with a sound basis forthe decision. The paper is successful if the decisionmaker understands the trade-offs among the mostattractive options—what is gained and what is lostwhen the decision is implemented.

Consider an everyday example, the so-calledcost/benefit analysis. A cost/benefit analysis issimply a decision paper where one criterion is costand the others are the potential benefits. Who is tosay whether the benefits of an expensive option are

worth the cost? The decision maker, that's who. Asthe analyst, your job is to enable him or her to makethe decision with confidence:

• That there is not yet another option with an evenbetter cost/benefit profile,

• That all the costs and key benefits are representedaccurately.

If you have done this, you have done your job well.

The same is true for any decision paper. Yourgoal is to identify all the best options and theapplicable criteria, and to accurately assess theoptions. If one of the options is obviously better thanthe others, that makes the decision maker's jobeasier. Your mettle as an analyst will only be proven,however, when you have to do a decision paperwhere the options present a difficult choice. If youcan define the issue clearly enough to support aninformed decision, you will have met the test.

Doing the AnalysisBefore you can write the paper, you have to do

the analysis. This is a seven step process:

• Define the issue,• Identify the options,• Identify the evaluation criteria,• Describe each option,• Compare and score the options,• Identify the discriminators,• Summarize the trade-offs among the options.

Only rarely will you perform these steps in se-quence. Unless you are intimately familiar with theissue at hand, you will have to revisit some of thesesteps as you proceed with the analysis. For example,the process of describing and comparing the optionsoften discloses additional criteria that should beconsidered. Nevertheless, it is important to performeach of these steps consciously. They should not beskipped or blurred together.

Define the Issue

Document your understanding of the problemas precisely as you can. Describe the background ofthe issue—how it arose, who will be affected by thedecision, who will make the decision, any con-straints on the solution, and anything else of impor-tance.

It is tempting to perform this step mentally,without writing the information down. Don't. Thereason for this step is to make sure you have theissue defined correctly. If it is only in your head, youcan't be sure.

Identify the Options

Prepare a list of the possible solutions to theproblem. Do not evaluate them yet or even describethem in detail; that comes later.

Generating options—potential solutions to theproblem—is the most creative and importantactivity in the analysis. To be successful in resolvingan issue, you have to include the "right" answersamong your options. If you overlook them, you willoffer only inferior options to the decision maker.

Brainstorming is a popular technique for gen-erating options, but it is unreliable. How can you becertain that the "right" answers will surface in abrainstorming session? You can't. A more reliablemethod is needed.

Imagine a "design space" of several dimen-sions. For example, suppose that you have to eval-uate design strategies for a new inventory system.The design space has at least three dimensions:

• How much of the existing system to retain- All- Some- None

• Degree of centralization vs. distribution- Full}- centralized- Centralized database with remote workstations- Fully decentralized

• Hardware vendors- IBM- DEC- Tandem

Options can be generated by selecting onechoice from each dimension, for example:

• Fully centralized IBM system retaining some ofthe existing system,

• Fully decentralized Tandem system retainingnone of the existing system.

This systematic method of generating optionsis called "exploring the design space." First, identifythe dimensions of the design space; then select"points" along each dimension; finally, define all thefeasible combinations. The richness of the optionsderived in this way may surprise you.

Still more options can be generated by forminghybrids—creative combinations of other options. Forexample, instead of choosing between buying aPorsche or buying a limousine, one could buy aPorsche and rent a limousine when necessary.

After you have identified all the feasible op-tions, the next step is usually to eliminate some. Tokeep the analysis manageable, you should evaluateno more than three or four options in detail. Thereare two reasons for eliminating an option fromfurther consideration:

• It fails to satisfy a constraint.

• It is clearly inferior to the other options.

A constraint is a filter for eliminating unaccept-able options, not merely a very important criterion.It is useless to consider further any option that has afatal defect. If, for example, an accounting modelfails to satisfy your auditor's requirements, or asoftware package does not run on your hardware—reject it immediately. The rest of the analysis will beshorter and therefore easier to understand. But doensure that the requirement is really a requirement.Sometimes an option is eliminated because someonethinks the decision maker won't accept it. Don't con-strain the options unnecessarily.

Although they satisfy the constraints, someoptions may be clearly inferior to others for a varietyof reasons—technology, cost, risk, etc. Eliminatethem as well. Document briefly the options that youhave eliminated, in case any of your assumptionschange later.

Identify the Criteria

Define clear and unambiguous criteria forevaluating the options. To keep the rest of theanalysis manageable, include no more than five toseven major criteria. If there are many more thanthis, as is sometimes the case in a technical evalu-ation, group the criteria into categories. For example,in a DBMS evaluation there may be a dozen or morespecific questions related to restart/recovery andconcurrent update; group them into a categorycalled "Data Integrity." Besides simplifying theanalysis and the presentation, this approach helps

you to perform a completeness proof on the criteria.It is difficult at best to demonstrate that an un-clustered set of dozens of individual criteria definesthe goal of the decision.

Include as criteria the factors that will influ-ence the decision. If a factor will not influence thedecision, leave it out of the analysis. For example,although "number of current users" is often a factorin a software package evaluation, who really makessuch a decision based on this criterion? Vendorcredibility and stability are real concerns, however,and would be valid criteria.

There is a straightforward technique for identi-fying the "right" criteria—the ones that will influ-ence the decision. Considering each option in turn,define what is good and what is bad about it. If thisproves to be difficult, consider the options in pairs—why A is better than B, and why B is better than A.Consolidate the resulting list of candidate criteria byeliminating duplicates.

The final step is to ensure that the criteria areindependent and at an appropriate level of abstrac-tion. Criteria are independent if they describe com-pletely different characteristics of the options.Review the list of candidate criteria and combineany that are not independent into a more general cri-terion. For example, if an analysis of motor vehiclesgenerated "number of seats" and "passenger capa-city" as criteria, you would have to combine theminto a single criterion; they are not independent.

As you combine criteria in this way, you form-ulate more abstract criteria. Be careful that the finallist comprises criteria that are at a comparable, usefullevel of abstraction. For example, "number of seats"and "luggage capacity" are at the right level ofabstraction; each addresses a major issue in buying acar, and they are independent. The next higher levelof abstraction, perhaps "capacity," is too broad to beuseful.

Do not try to assign weights to the criteria,even subjectively. Just write them down and definethem as precisely as you can, then proceed to thenext step.

Describe Each Option

Having identified a set of viable options andsignificant criteria, you are now ready to evaluatehow well each option satisfies each criterion. If youthink of the analysis in terms of a matrix—optionsversus criteria—this step addresses each columnseparately (see Figure 1A on next page).

Options

1 2 3 4

Criteria c

Options

1 2 3 4

Criteria c

Figure 1A

Describe Each Option

Your purpose in this step is to describe eachoption as objectively as possible in terms of thecriteria. Write down all of the important facts foreach "cell" in the matrix. Use numerical valueswhere possible: for example, cost ($), speed (MIPS),service life (years).

Most evaluations, unfortunately, result insubjective ratings such as "excellent," "good," "fair,"and "poor." There would be no problem if everyoneagreed on the meanings of such ratings, but that israrely the case.

To solve this dilemma, you need to define theratings as explicitly as possible. Once again, there isa strategy that will lead you to your goal in anorderly way. For each criterion that generatessubjective ratings, define a set of descriptors that spanthe range from best to worst. For example, inevaluating the features of a software package, thefollowing set of descriptors might be appropriate:

Best Includes all required features, plus addi-tional useful features

\

Includes all required features, but littlemore

Includes most required features, withminor exceptions

Figure IB

Compare and Score the Options

For an automobile evaluation, seating comfortmight be evaluated with the following descriptors:

Best

tWorst

Good support, good cushioning, goodposition relative to controls—no discom-fort after 1 hour

Good support, cushioning, position, butsome discomfort after 1/2 hour

Some discomfort apparent after 5-10minutes

Uncomfortable immediately

Worst Excludes some important requiredfeatures

To simplify the subsequent comparison of theoptions and presentation of the results, limit thenumber of descriptors to four.

These examples demonstrate two importantpoints. First, it is important to make the descriptorsmutually exclusive. In the software package ex-ample, what happens if a package has most of therequired features and some additional features? Mydescriptors don't address this possibility explicitly.One approach is to refine the descriptors to includethis case. Another is to recognize that we really havetwo criteria here—required features and additionalfeatures—and to separate them. Each situation isunique, so you have to use the results of the analysisto tailor the criteria and the descriptors to thesituation.

Second, you can sometimes transform a set ofqualitative descriptors into a numerical value. AfterI created the seating comfort descriptors for thisexample, I realized that they could be discarded infavor of the objective criterion "elapsed time untilnoticeable discomfort." This is a striking example ofhow an ambiguous criterion can be transformed intoone that can be measured and that everyone willunderstand.

Do not hesitate to iterate the previous steps. Ifyou discover that you have included a criterion thatdoesn't matter, or omitted one that does, go backand revise the list of criteria.

When you have finished gathering and docu-menting this information, you are ready to do thecomparisons.

Compare and Score the Options

Your goal in this step is to identify the relativestrengths and weaknesses of the options for each ofthe criteria. In terms of the matrix, this step ad-dresses each row separately (see Figure IB). If youwere able to quantify the results for any of thecriteria, use the values directly. Otherwise, use thedescriptors as the basis for the evaluation.

If you have done a thorough job of describingthe options, it will be easy to compare and scorethem. If not, you may have to do more research togather missing data.

First, define the scale that you will use topresent the results. A simple scale with a few clearlydefined gradations is best (e.g., 0-3). In an effort toadd precision, analysts sometimes make the mistakeof using too fine a rating scale. Using a scale of 0-10,for example, implies that there is a meaningfuldifference between a 6 and a 7.

This is why you should try to limit the numberof descriptors to four. If you need more than this todescribe the options accurately, then you shouldcluster them into four categories for comparison andpresentation. Likewise, if you have a criterion withnumerical scores (e.g., cost ($)), define four rangesfor comparison and presentation.

If the differences among the options are small,define the ranges and the descriptors so that yourevaluation will highlight the differences. Do not,however, exaggerate minor differences among

perfectly satisfactory options so that the worstappears to be awful. The discussion of discrimina-tors in the next section will help you to see when toemphasize the differences and when not to.

A graphic format often is the best way to pre-sent the results of the analysis. Figure 2 on the nextpage, an example from the system concept for a mu-nicipal financial system, shows how incisive thisformat can be.

Identify the Discriminators

Recall that you were advised not to assignweights to the criteria. Now you will learn why. Per-haps surprisingly, the importance of each criteriondepends on the results of the analysis. If a criteriondoes not discriminate the options, it is not importantto the decision even though it may be critical to thesuccess of the project. For example, although dataintegrity is crucial in most DBMS applications, it willnot affect the selection decision if all the optionssatisfy all the data integrity objectives.

The decision will be based on the discrimina-tors—the criteria that most sharply differentiate theoptions. In this phase of the analysis your task is toidentify the discriminators and to review the de-scriptions of the options to make certain that eachdiscriminator is discussed fully.

Start by eliminating the criteria that have sim-ilar, acceptable scores for all options (e.g., criterion 6,"Flexibility to Change" in Figure 2).

Next, eliminate any options that are clearly in-ferior. In Figure 2, option 3 dominates option 1 forevery criterion except 4 and 5, where option 3 is atleast good; option 1 is clearly inferior to option 3.

Finally, identify the criteria that differentiatethe remaining options. In Figure 2, these are primar-ily criteria 4 and 5; the differences among the othersare interesting but won't drive the decision.

If there are truly no discriminators, don't fabri-cate one. "I don't care" is a valid conclusion for theanalysis.

Summarize the Trade-offs Among theOptions

Having identified the discriminators, andhaving clear distinctions among the options in termsof the discriminators, you are now ready to framethe issue for the decision maker. Mavbe the differ-

Criteria

1. Auditability

2. Control

3. Reporting

4. Management Incentive

5. Agency Needs

6. Flexibility to Change

7. Training

8. Cost

Option 1(Decentralized)

•

•

• • •

• • •

• •

•

Option 2(Central)

• • •

• • •

• •

• •

• • •

• • •

Option 3(Mixed)

• •

• •

• • •

• •

• •

• • •

• •

• •

Figure 2

ences are technically complex, or the benefits of themore costly approach are subtle. You have tosummarize the trade-offs in terms that the decisionmaker will understand.

Here are a few of the things that you do not do.You do not add the scores to find the winner. Youdo not automatically recommend the option withthe most high scores (because just one poor scoremay be fatal).

Instead, imagine yourself preparing for abriefing where you have five minutes to explain theissue to the decision maker. Write your briefingnotes. If you are analyzing the financial systemoptions shown in Figure 2, point out the fatal flawsin options 1 and 2. Focus on the criteria that high-light those flaws. If you are performing a classiccost/benefit analysis, focus on the benefits thatmight justify the higher cost option.

Finally, if your analysis is based on assump-tions about future events or other uncertainties (e.g.,the cost savings to be realized by a course of action),perform a sensitivity analysis. What is the effect ifyour assumptions are wrong by a little (or a lot)?What is the crossover point, when an option is nolonger attractive? How likely is the contingency tooccur?

Writing the PaperPresumably you have documented the analysis

as you performed it. To write the decision paper,you now have to organize the material in the bestformat for its intended audience. Here is a goodgeneral purpose outline for a decision paper:

• Management Summary• Background• Options• Evaluation Criteria• Evaluation of the Options• Comparison of the Options• Summary and Recommendations

This outline mirrors the analysis steps, withtwo exceptions. It adds a Management Summary toprovide an introductory overview of the entireanalysis, and it lumps the last two analysis steps intothe Summary and Recommendations.

The Management Summary explains the issueand the key features of the analysis. It presents theadvantages and disadvantages of the options—thediscriminators—and succinctly defines the basis forthe decision. If a table such as Figure 2 has beenprepared, it should be included in the ManagementSummary. Remember that you are not writing anovel; the reader wants to know the ending up front.

The Background section documents the infor-mation assembled in the first step of the analysis,namely, define the issue. If there are any technical,political, or economic constraints on the resolutionof the issue, identify them here. If the issue has beenconsidered previously, include a brief description ofits history and the reasons for reopening it.

In the Options section, identify the options youanalyzed and explain how you selected them. If anyother options were dropped from consideration,describe them briefly along with the reasons foreliminating them.

In the Evaluation Criteria section, describe thecriteria and why each is important. Include a discus-sion of what is good, what is bad, and why. Presentthe descriptors—the definitions of the ratings—thatyou used for the subjective criteria. One of the rea-sons for defining the criteria explicitly is to enableothers to review the analysis. Without the rationalefor each criterion, the reader is left to guess why theauthor included it in the study. If a large number offeatures were grouped into categories, present thedetails in an appendix; don't clutter the body of thepaper with them.

In the Evaluation of the Options section, de-scribe how well each option satisfies the criteria.Group the presentation by option. If there is a lot ofdetail, put the detail in an appendix and summarizeit in the text.

In presenting the Comparison of the Options,group the presentation by criterion. Highlight thekey differences among the options for each criterion,using graphics wherever possible.

The Summary and Recommendations sectionconsolidates the information developed in the lasttwo analysis steps, namely, identify the discrimina-tors and summarize the tradeoffs. If you have basedthe analysis on any assumptions, you should presentthe results in a way that helps the decision makerjudge the validity of the assumptions.

The typical approach is to state the assump-tion, derive and present the results, and then evalu-ate the sensitivity of the results to the assumption.For example, if a buy/lease analysis assumes a costof money of 10 percent, you would present theresults and then show what happens if the cost islower or higher than 10 percent. Although straight-forward, this approach has a defect. The decisionmaker may immediately question the assumptionand then be skeptical of the rest of the analysis.

An alternative is to use a goal-seeking presen-tation format. State the desired result, then derive thevalue of the assumed variable that produces thisresult. For the buy/lease analysis, for example, youwould derive the value of the cost of money thatgenerates a break-even result—say 18 percent. Thenyou can tell the decision maker "if the cost of moneywill be under 18 percent, it's better to buy; over 18percent, it's better to lease." In effect, this allows thedecision maker to make the assumptions, and it doesit in a way that eliminates the skepticism of thetypical approach.

If appropriate, conclude the paper with a rec-ommended course of action.

ConclusionThe approach described in this paper encour-

ages a rigorous thought process in analyzing issues.Adherence to the approach prevents armwaving andother forms of muddy analysis, presumably improv-ing the effectiveness of the analyst. In a rationalworld, it will also lead to better decisions.

About the Author

Milton S. HessMilton Hess is an AMS Vice President. Since

joining AMS in 1977, he has served as a technicalspecialist, manager, and senior reviewer on largeinformation systems projects for both commercialand government clients. Prior to joining AMS, Miltspent 10 years at Bell Laboratories as a researchengineer and computer systems manager.

Throughout his career, Milt has been a prob-lem solver. He has sought strategies that would helphim, and others, find good solutions to complexproblems in an orderly, repeatable fashion. He haspublished and presented papers on strategies fordeveloping large information systems, he was a keycontributor to LPS Methodology, AMS's systemdevelopment methodology, and he has developedand presented a number of seminars within AMS onsystem design and analysis techniques.

Among his project roles, Milt has been the leadarchitect on system design projects for the ArmyMateriel Command, Army Corps of Engineers, andU.S. State Department. He has prepared softwaredevelopment and testing methodologies for GTEand the Army Information Systems Command, anda database design methodology for Standard OilCompany of Indiana.

Milt is currently Technical Director for infor-mation Technology for AMS's defense and aero-space business, where he focuses on the applicationof proven engineering strategies to system designand development.

CORPORATE HEADQUARTERSAmerican Management Systems, Inc.1777 North Kent StreetArlington, Virginia 22209(703) 841-6000(703) 841-5584 FAX

EUROPEAN HEADQUARTERSAMS Management Systems Deutschland GmbHQuerstrasse 8-106000 Frankfurt 1GERMANY49-69-245511049-69-24 55 11 99 FAX

NORTH AMERICAN SUBSIDIARIES

Data Base Management, Inc.AMS Courseware Developers300 Chapel RoadManchester, Connecticut 06040(203) 646-3264(800) 232-3264(203) 643-9629 FAX

AMS Management Systems Canada Inc.180 Elgin StreetOttawa, OntarioCANADA K2P 2K3(613) 232-7400(613) 232-0324 FAX

AMS Operations Corporation66 South Van Gordon StreetLakewood, Colorado 80228(303) 989-7065(303) 232-4618 FAX

AMS Technical Systems, Inc.1525 Wilson BoulevardArlington, Virginia 22209(703) 841-2000(703) 276-3295 FAX

NORTH AMERICAN OFFICESAlexandria, VAArlington, VABoston, MABremerton, WAChesapeake, VAChicago, ILDallas, TXDetroit, MIEmeryville, CAHouston, TXLakewood, COLos Angeles, CAManchester, CTMontreal, CanadaNew York, NYOttawa, CanadaPort Hueneme, CAPortsmouth, NHRedwood City, CARichmond, VARoseland, NJSan Diego, CASan Francisco, CASarasota, FLToronto, Canada

EUROPEAN SUBSIDIARIES

AMS Management Systems Deutschland GmbHQuerstrasse 8-106000 Frankfurt 1GERMANY49-69-245511049-69-24 55 11 99 FAX

AMS Management Systems Europe, S.A./N.V.Avenue Louise 651050 BrusselsBELGIUM32-2-535-783132-2-535-7731 FAX

AMS Management Systems U.K. Ltd.Garden House18 Finsbury CircusLondon EC2M 7BPENGLAND44-71-638-323244-71-382-9776 FAX

EUROPEAN OFFICESBrussels, BelgiumDusseldorf, GermanyFrankfurt, GermanyLondon, England

© 1992 American Management Systems, Inc.Printed in the USA. All rights reserved.