Requirements are King – Better Requirements = Better Software

World®’16

RequirementsareKing– BetterRequirements=BetterSoftwareBenjaminJohnson-Ward– SeniorConsultant,Presales– CATechnologies

DO5X23E

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ForInformationalPurposesOnlyTermsofthisPresentation

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.


Abstract

Countlessresearchandcasestudieshavelongpointedtoacold,hardtruth:therequirements,notdevelopersortesters,introducethevastmajorityofdefectsthatwreakhavocinproduction.Ratherthansolvethisissue,thedevelopmentpracticesthathavearisenaround“Agile”haveintroducedanewchallenge,astestershavetopiecetogetherandreacttoaconstantbarrageofdisparateuserstoriesandchangerequests.What’smore,therequirementsremainstatic,sothattestershavetomanuallyconvertthemintotests.Thisisaslow,unsystematicprocess,whichcreatesdefectswhentheambiguousrequirementsaremisinterpreted.Inorderfortestinganddevelopmenttokeepupwithconstantlychanginguserneeds,requirementsmustbecapturedinaformatwhichminimizesmiscommunication,canreacttochange,andenablessubsequenttestassetstobederivedautomatically.§ Thissessionwillsetouthow“active”flowchartmodellingcan:§ Provideaformatalreadyfamiliartobusinessanalyststocaptureuserneeds§ Provideaformatfortesterstogenerateoptimizedtestcases,testscripts,dataandexpectedresults

automatically.§ AutomaticallyreflectchangesmadetotherequirementsinthetestsassetsInthisrequirements-drivenapproach,testingtherebybecomesalargelyautomatedcomparisonofhowasystemshouldfunctiontothesystemthathasbeendelivered.

BenjaminJohnson-WardCATechnologiesSeniorConsultant,Presales


Agenda

WHATISAREQUIREMENT?

WHYFOCUSONREQUIREMENTS?

Q&A

HOWABOUTSOFTWAREENGINEERING?

HOWABOUTREQUIREMENTSENGINEERING?

INTRODUCINGTHEREQUIREMENT-DRIVENAPPROACH

1

2

3

4

5

6


WhatisaRequirement?


Definition:Requirement

Requirement:singulardocumentedphysicalandfunctionalneed thataparticulardesign,product

orprocessmustbeabletoperform.

ISO/IEC/IEEE29148:2011

Wikipedia

Requirement:statementwhichtranslatesorexpressesaneed anditsassociatedconstraintsand

conditions.


WhatisaRequirement?

ErikSimmons,nuCognitive,“21st CenturyRequirements”,11/08/11

Arequirementisastatementof:

1. Whatasystemmustdo(functionalrequirement)

2. Howwellthesystemmustdowhatitdoes(qualityorperformancerequirement)

3. Aknownresourceordesignlimitation (constraint)

Moregenerally,

Arequirementisanythingthatdrivesadesign choice


ThePurposeofRequirements

Thepurposeofrequirementsistohelpestablishaclear,common,andcoherentunderstandingofwhatthesystemmustaccomplish.

Sowhyfocusonrequirements?ErikSimmons,nuCognitive,“21stCenturyRequirements”,11/08/11

Clear

Allstatementsareunambiguous,complete,

andconcise

Common

Allstakeholderssharethesameunderstanding

Coherent

Allstatementsareconsistentandforma

logicalwhole


WhyFocusonRequirements?


WhataretheMainSoftwareChallengesYouareFacing?

Pre-webinarsurvey,“TestingImperativesintheWorldofAgileDevelopmentandContinuousDelivery”,HuwPrice&JonathonWright,28/07/15

0% 10% 20% 30% 40% 50% 60% 70%

Time/resourcesintestdatacompliance(PII)

Defectsstemmingfromambiguousrequirements

Testinginnefficienciesleadingtohighercost

Lackoftestcoveragecreatingdefects/rework

Difficultyfindingtherightdataforaparticulartest

Manualtestingleadingtoprojectdelays


VAGUENESS

ExamplesofAmbiguityandIncompleteness

Poor ScopingAmbiguous Reference

Omission Acronyms

Aliasing

Sentence Structure

DanglingElse

Incomplete Glossary

Implicit Constraints

I.E. or E.g.?

LexicalAmbiguity

Polysemy

Passive Sentences


WhatMakesaGoodRequirement?

§ complete§ consistent§ correct§ modifiable§ ranked§ traceable§ unambiguous§ understandable§ testableNASA,AutomatedRequirementsMeasurementTool,1999,http://satc.gsfc.nasa.gov/tools/arm (archive.org)


WhatistheProblemwithRequirements?

§ Thoughaplethoraoftechniquesexist,mostarewritteninambiguousnaturallanguage

§ Therequirementsare“static”–theyoffernowaytoderivetestsdirectlyfromthem…

§ …nowaytoupdatetestswhentherequirementschange– thishastobedonemanually

BenderRBT,2009


UserTheenduserknowswhattheywant

TheSDC:What’sSupposedtoHappen

BusinessAnalystTheanalystspecifieswhatthatis

DeveloperTheprogrammerwritesthecode

TesterThetesterteststheprogram


TheFundamentalIssueWithThisModel

§ Ateachstagethatinformationispassedonandtransformedtoanotherform,thereisanaturaldegradationinquality

§ ThetypicalSDLChasseveralofthese“hops”:1. Requirementsareconvertedtocode2. Requirementsareconvertedtotestcases,tobeappliedtothecode3. Testcasesareconvertedintoautomatedtests4. Datahastobefoundorcreated,basedontherequirements,for

developmentandtesting

InformationHops:Enterprise-Wide“ChineseWhispers”

Theimpactonqualityiscreatedbyuncertainty.Some(inherent)uncertaintyisunavoidable.Someisavoidable,andshouldbeavoided.


“RumsfeldMatrix”– AvoidablevsUnavoidableUncertainty§ InformationintheSDLCcanbebrokendowninto:

– KnownKnowns– UnknownKnowns– KnownUnknowns– UnknownUnknowns

Knowns Unknowns

Known Information AvoidableUncertaintyàObservableDefects

Unknown AvoidableUncertaintyàObservableDefects

Unavoidable Uncertainty

Inanidealworld,testerswillbeprovidedwithalltheinformationtodiscoverobservabledefects.Whateverpreviouslyunknowableinformationtheydiscoverwilladdtothe“knownknowns”


Unambiguous,IncompleteRequirements

§ Poorqualityrequirementsintroduceuncertaintyatthebeginning,whichisthenperpetuatedthroughouttheSDLC

§ Asystemcanbebuildaroundmisunderstanding,sothat:– Atleast56%ofdefectsstemfromambiguityinrequirements1 – some

placethisashighas59%2 oreven65%3

– 64%oftotaldefectcost3originateintherequirementsanalysisanddesignphase– someplacethisashighas80%1

– 40-50%ofprojectcosts areexpendedinrework4

– Onaverage,only69%ofdesiredfunctionalityisactuallydelivered5

Introduceavoidableuncertaintyattheverybeginning

1– BenderRBT,2009

2– ITUniveristy,Copenhagen,2001

3- HyderabadBusinessSchool,20124– Critical-Logic,20145– StandishGroup’sChaosManifesto2014


Unambiguous,IncompleteRequirementsAnexampleambiguityreview:


AmbiguousNaturalLanguage

§ “Timeflieslikeanarrow;fruitflieslikeanapple”

§ “BuffalobuffaloBuffalobuffalobuffalobuffaloBuffalobuffalo”– Agrammaticallycorrectsentence,whichmeans“InthecityofBuffalo,

bisonwhoarebulliedbybison,arethemselvesbullyingbison”

§ Wesawherduck

Sentencestructure,homonyms,polysemy,lexicalambiguity,etc.


TheDanglingElse

§ IFATHENIFBTHENCELSED– IFATHEN(IFBTHEN(C)ELSED)– IFATHEN(IFBTHEN(C))ELSED

§ IFRAINTHENIFWINDYTHENCOATELSET-Shirt– Ifitisraining,thenwearacoatifitisalsowindy.Ifitisrainingbutnot

alsowindy,wearaT-Shirt.– Ifitisraining,andifitiswindy,thenwearacoat.OtherwisewearaT-

Shirt

Incompletesystemlogicstemmingfromsyntacticalambiguity


ThoughaPlethoraofTechniquesExist,MostAreWritteninAmbiguousNaturalLanguage


WhatAretheChallengesWithRequirements?

1– BenderRBT,20092– ITUniversity,Copenhagen,20013- HyderabadBusinessSchool,20124– Critical-Logic,20145– StandishGroup’sChaosManifesto2014

§ Asystemcanbebuildaroundmisunderstanding,sothat:– Atleast56%ofdefectsstemfromambiguityinrequirements1 – some

placethisashighas59%2 oreven65%3

– 64%oftotaldefectcost3originateintherequirementsanalysisanddesignphase– someplacethisashighas80%1

– 40-50%ofprojectcosts areexpendedinrework4

– Onaverage,only69%ofdesiredfunctionalityisactuallydelivered5


TheDanglingElse

§ IFATHENIFBTHENCELSED– IFATHEN(IFBTHEN(C)ELSED)– IFATHEN(IFBTHEN(C))ELSED

§ IFRAINTHENIFWINDYTHENCOATELSET-Shirt– Ifitisraining,thenwearacoatifitisalsowindy.Ifitisrainingbut

notalsowindy,wearaT-Shirt.– Ifitisraining,andifitiswindy,thenwearacoat.Otherwisewear

aT-Shirt

Incompletesystemlogicstemmingfromsyntacticalambiguity


TheMissingElseAcommonmenacewhenusingBehaviour-DrivenDevelopment,GherkinandRSPEC

GIVENCurrentAccount

AndGIVENnegativebalance

WHENcustomerattemptspayment

THENgointooverdraft


Requirements

Asmanyas56%ofdefectsareavoidedindevelopment

§ TestcasescanbeAuto-generatedandre-generatedfromrequirements

§ Theyareoptimizedtoprovide100%coverageinfewertests

§ Changescanbequicklyandeasilymade

§ Requirementsandtestcasesarelinkedtodatawith100%coverage,includingnegativeandfuturescenarios § Automatedtests

aregeneratedfromtestcases

§ Areupdatedifrequirementschange

§ Exhaustivelytestapplication

§ Pallet-basedUIAutomation

Requirementsaremodelledtoacomplete,unambiguous,“Active”Flowchart

ReduceUncertainty– ClearRequirementsandFewer“Hops”Code TestCases TestData Automation


APossibleSolution:Model-BasedTesting

1. Mappingrequirementstoanunambiguous,activeflowchart– Codersknowwhatneedstobecoded

2. Auto-generatetestcasesandAutomatedtests–nomore“Hops”– Executethesetestsagainstthecode,testingthatitmatches

thefunctionalityspecifiedintherequirements

3. Updatethemodelastherequirementschange,orastestinguncoversfurther,previously“unknown”information

Testasystemagainstanagreedupon,cleardefinitionofwhatthefunctionalityshouldbe


HowAboutSoftwareEngineering?


WhatisEngineering?

Engineeringistheapplicationofheuristicsunderuncertaintytocausethebestpossiblechangewithintheavailableresources(BillyVaughnKoen*)§ Aheuristicisanythingthatprovidesaplausibleaidordirectioninthe

solutionofaproblem§ Useofheuristicsdoesnotguaranteeasolutiontoaproblem,anddifferent

heuristicscanofferconflictingadviceforanygivensituation§ Heuristicsworkbyexploitingthestructureofanenvironmenttosimply

decisionmakingandnarrowthesolutionsearchspace

ErikSimmons,nuCognitive,11/09/16adaptedfrom“adiscussionofthemethod”,BillyVaughnKoenOxfordUniversityPress2003


HowDoesThisApplytoSoftwareEngineering(SDLC)?Simpleexampleoflackofclarityandvisionwithinthesoftwaredevelopmentlifecycle

BusinessAnalyst Programmer TesterUser

TheUserKnowswhattheywantTheAnalystspecifieswhatthatis

TheProgrammerwritesthecode

TheTesterteststheprogram

Clarity&Vision


HowAboutWithinLean&Agile?

DeanLeffingwell,ScaledAgileFrameworkFoundations(v4.0.6),“EmbracingLean-Agilevalues”,2016

HouseofLean AgileManifesto

LEADERSHIP

Res

pect

for

peop

le a

nd c

ultu

re

Flow

Inno

vatio

n

Rel

entle

ssim

prov

emen

t

VALUE

Valueinthesustainablyshortestleadtime

Thatis,whilethereisvalueintheitemsontheright,wevaluetheitemsontheleftmore.

Weareuncoveringbetterwaysofdevelopingsoftwarebydoingitandhelpingothersdoit.Throughthisworkwehavecometovalue:

Individualsandinteractionsoverprocessesandtools

Workingsoftwareovercomprehensivedocumentation

Customercollaborationovercontractnegotiation

Respondingtochangeoverfollowingaplan


HowAboutRequirementsEngineering?


Definition:RequirementEngineeringISO/IEC/IEEE29148:2011

Wikipedia

Referstotheprocessofdefining,documentingandmaintainingrequirementstothesub-fieldsof

systemsengineeringandsoftwareengineeringconcernedwiththisprocess.

Interdisciplinaryfunctionthatmediatesbetweenthedomainsoftheacquirerandsuppliertoestablishand

maintaintherequirementstobemetbythesystem,softwareorserviceofinterest.Requirementsengineering

isconcernedwithdiscovering,eliciting,developing,analysing,determiningverificationmethods,validating,

communicating,documenting,andmanagingrequirements.


WhatisRequirementsEngineering?

Requirementsengineeringistheuseofheuristicsfordiscovering,documenting,andmaintaining asetofrequirementsforasystemorserviceAtthehighestlevel,requirementsengineeringseeksto:§ Definethenecessaryandsufficientsystemscopeandfeaturesto

enablesystemdevelopmentatanacceptablelevelofrisk§ Enablerequirements-drivenarchitecture,design,construction,

andverificationactivities§ Provideinformationaboutsystemdevelopmentprogressandstatus§ Provideinformationtoorganizationallearningandcontinuous

improvementeffortsErikSimmons,nuCognitive,11/09/16adaptedfrom“adiscussionofthemethod”,BillyVaughnKoen,OxfordUniversityPress2003


HowtoManageRequirementsEngineering?

ToGenerateTestCases

DesignandprovisionTestDataVirtualisation

ToManageChangeinTestCases

ToGenerateAutomation

Tests

ToEstimateComplexity

PopulateStoryBoards&Backlogs

ToBuildBetterRequirements

ToImproveExisting

TestCases DesignandprovisionServiceVirtualization

ImpactandDifferences

built by

CAAgileRequirementsDesigner

NativeSupport


IntroducingtheRequirement-drivenApproach


DesignOpsistheNewDeliveryParadigm:FromIdeationtoDesign,DevelopmentandTesting


ModelRequirementsasan“Active”Flowchart

§ AformalmodelthatisaccessibletothebusinesswhoalreadyuseVISIO,BPM,etc.

§ Whichisalsoamathematicallyprecisemodelofasystem,sothatiteliminatesambiguityandincompleteness

§ Itcanbeusedbytestersanddevelopers–itbringstheend-user,businessandITintoclosealignment


TestingWiththe“Active”Flowchart

§ Testerscanoverlaytheflowchartwithallthefunctionallogicanddatainvolvedinasystem

§ Testscanthereforebeautomaticallyderivedfromit


GeneratingTestCases

§ Exhaustivelytestamodel– extractallpossibleroutesfromstart->end

§ Eachroute/pathbecomesatestcase

§ Numberofroutesgrowsexponentiallywitheveryaddeddecision32nodes+62edges=1,073,741,824possibleroutes

2145Paths


Optimization/Coverage

§ Coverage,inmostcases,isrelatedtohowmuchfunctionalityisbeingcoveredinatestcase.

§ Traversethemodelbysatisfyingcoverageconstraints:19PathsEdgePairs

9PathsInOutEdges

5PathsEdges

3PathsNodes


GenerateTestData

NotReadyforTesting!

CATestDataManager+Requireddatacharacteristics

Provisionfitforpurpose dataanytimeandeverytimeProvisiondatawithorwithoutaccesstoproductionsystems

ReadyforTesting!

§ Automaticallyprofiledata,modelit,andaccuratelymeasureitscoverage§ Generaterichsyntheticdatawhichprovides100%coverage§ Covereveryoutlier,unexpectedresult,boundaryconditionandnegativepath


ProvideFullTraceabilitywithRequirements

§ Knowwhatneedstobere-testedandwhentheintegrityofasystemisatrisk…“IfIchangethis,whatwillIbreak?”

§ Theimpactofachangemadetoanindividualcomponentisidentifiedsystemwide

§ Theimpactontestcasesanduserstoriesupanddownasystemcanalsobeidentifiedautomatically


ButAllofThisStillDependsontheRequirementsBeingRight!§ Yes,youcanreaphugerewardsfrommaintainingonevery

highqualityartefact,ratherthanthousands.

§ No,nosilverbullet,modelscanbewrong,peoplecanstillmakemistakes

§ TheideabehindMBTandARDistomakethosepotentialmistakesmoreobviousandeasiertofixoncediscovered

§ Andofcourseleveragethemodel,whilstacknowledgingtheresidualrisk.


Demo


FunExample:PokémonGo

PokémonGo,HuwPrice,22/07/16,(www.grid-tools-downloads.com/huw/POKEMONGO.zip)


Summary:Heuristics,RiskandRequirementsEngineering§ Valueiskeytodeliveringsoftwarewhichmatterstousers:

– asolutionmustbefrequentlyanditerativelymeasuredforvalue

§ PoorRequirementsdegradequalitybutalsointerruptthevaluestream:– Testerscannottestrigorouslyorquicklyagainstcustomerneeds

§ Requirementscanalsothereforebekeytoimprovingthedeliveryrateofvaluablesoftware

§ Enablingrequirements-drivendesign,constructionandverification:– Modelrequirementsas“active”flowcharts– Testfromtheflowchartwithauto-generationoftests/scripts– Generatetestdatatomatchthetestcases– Updatetestswhentherequirementschange


Questions?


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5T05TTechTalk:GoingFullCircle- LinkingCodetoTeststoRequirementsandBackAroundAgain

11/16/2016at12:45pm

DO5X27S What'sNewinCAAgileRequirementsDesigner? 11/17/2016at12:45pm

DO5X40S CaseStudy:WhyRabobankWasInspired 11/17/2016at3:45pm


MustSeeDemos

UseModelBasedTesting

CAARDDevOpsTheatre5

DevOps5Theatre

AchieveMaxTestCoverage

CAARDDevOpsTheatre5

ImproveDataCompliance

CATestDataManagerDevOpsTheatre5

DeliverTestDataFaster

CATestDataManagerDevOpsTheatre5


Stayconnectedatcommunities.ca.com

Thankyou.

@CAWORLD#CAWORLD ©2016CA.AllRIGHTSRESERVED.51 @CAWORLD#CAWORLD

DevOps– ContinuousDelivery

FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu

Technology

Requirements are King – Better Requirements = Better Software