Automated Tes,ng for Embedded Products · Automated Tes,ng for Embedded Products Michael Corr June 8 ... Total Cost $60 $760 +$700 ... Need to ﬁnd the right balance v1 (C) 2016

AutomatedTes,ngforEmbeddedProducts

MichaelCorrJune8,2016

(C)2016-MichaelCorr 1

[email protected]/in/michaelgcorr

v1

HardwareisHard•  Tes,ngforhardwareproductsishard

•  Toomanyvariablestocontrol•  HardwareplaMorm,environmentalcondi,ons,mechanicalinterfaces,

sensorvalues,etc.

•  Rigorous(automated)tes,ngbecomesin,mida,ngforhardwareproductscausingteamstooOenprocras,nate,resul,nginminimaltes,ngperformed

•  Inaddi,on,tes,ngisoOenanaOerthought,andcompaniesdon’thireaformaltes,ngteamearlyenough

•  Tes,ngismostcommonlydonemanuallysinceinves,nginautomatedtoolscanbeoverwhelming

(C)2016-MichaelCorr 2v1

AddingInsulttoInjury•  Webandmobileproductsaremucheasiertoautomate

tes,ng

•  Therewasanexplosionofinnova,oninthe2000’sintoolsfordevelopmentandtestforthewebandmobilemarkets

•  Hardwaretoolswereneglected-bothdevelopmentandtes,ngtools


HardwareisBack,Baby!•  But,inearly2010’shardwarebecamemoreaccessible

•  Anincreasednumberofyoungengineersbecameinterestedinembeddedproducts

•  Withthisresurgencecamethementalityofusingmoderndevelopmenttoolsandbestprac,ces•  FileVersionControl•  Con,nuousIntegra,on(CI)•  Automatedtes,ng


NoWalkInThePark•  Automatedtes,ngiss,lldifficultforhardwareand

embeddedproducts

•  Mul,pledisparateengineeringteamsworkingonthefinalsolu,on,makescollabora,onandcoordina,ondifficult•  Electrical,Mechanical,Firmware

•  ThehardwareplaMormisoOenbeingdevelopedsimultaneouslytothefirmware–leavingnofirmgroundtostandon

•  Infrastructuretocontrolandautomatevariablesisexpensiveanddifficulttoimplement,ornon-existent


WhyBother?•  Asastartup,,meandmoneyarepreciousresources

•  YourgoalsaretoconstantlyminimizewasteandincreaseROI

•  Latenightstroubleshoo,ngortryingtore-createanissue-wastesresources

•  Fieldtripstocustomersitestotroubleshootissues-wastesresources

•  Notcatchingahardwarebugbeforegoingintoproduc,on-wastes(serious)resources

•  Usingemployeestoimplementrote,repeatabletasks-wastesresources


TheRealWorld


Step Task Who Salary Time(Hrs) Cost

1 Developerwritesautomatedtests Engineer/OriginalAuthor $120k 1 $60

2 (Regression)Testisrunevery,mecodeiscommiied

BuildServer N/A N/A Free

WriteAutomatedTestAlongWithFeatureCode

v1

TheRealWorld



1 Developerwritesautomatedtests Engineer/OriginalAuthor $120k 1 $60




1 Email/Phonesupportwithcustomer CustomerSupportRep $60k 1 $30

2 RecreateCustomer’sEnvironment Technician $60k 1 $30

3 ReviewOriginalCode Engineer/Not-OriginalAuthor $120k 2 $120

4 RecreateIssueandSolve Engineer/Not-OriginalAuthor $120k 2 $120

5 Writeautomatedtests Engineer/Not-OriginalAuthor $120k 1 $60

6 TestPatchRelease TestEngineer $80k 8 $320



8 ReleasetoCustomer ReleaseEngineer $80k 2 $80

WriteAutomatedTestAlongWithFeatureCode

CustomerDiscoversBug

v1

TheRealWorld


WriteTestwithCode CustomerFindsBug Delta

TotalTime 1hour 17hours +16hours

TotalCost $60 $760 +$700

TeamSizeU@lized 1employee 5employees +4employees

SummaryComparison

•  Ifyouhave10incidentslikethisinayear,youarewas,ng•  $7konlaborcosts•  160hours(1month)ofcalendar,me

•  Thisapproachscaleslinearlywithnumberofincidents

•  So,whynotspend$7kontestequipmentandtools,instead?It’samuchbeierROI

v1

HiddenFees•  Writethetestsconcurrentlyatthe,meofdevelopment,notattheendof

thesprint/releasecycle

•  It’smuchmoreefficientfortheoriginalauthorofproduc,oncodetowritetestsatthe,meoffeaturedevelopment•  "Whilethesuckeriscutopen”•  Codeiss,ll“intheirhead”•  Morelikelytonottakeshortcuts(lessrushed)•  Morelikelytowritemoretestsresul,nginbeiertestcoverage

•  Atestthattakes1hourtowritewhileproduc,oncodeisfreshandinthedeveloper’sheadcantake1.5to2hourstowritejustaweekortwolaterbecausetheauthorhastoremember/reviewfeature’sdetails


FeatureA FeatureB FeatureC

TestA

TestB TestC

FeatureA

TestA

TestBFeatureB TestCFeatureC

Post:

Concurrent:v1

I’mABeliever!•  Startwithastrongfounda,onof3rdpartyproducts

•  Con,nuousIntegra,onServer(e.g.Jenkins,TeamCity,CruiseControl)•  Exis,ngandpopularxUnitbasedtesttools•  TestRepor,ngtools(e.g.TestRail)

•  Removeallfric,onfromwri,ngautomatedtests.•  Talktoyourdevelopers

•  Ifwri,ngtestshasfric,on,developerswilltakeshortcutsandfollowpathofleastresistance


StartWithPolicy1.  Alltests(automatedandmanual)arerevisioncontrolled

(svn,git)

2.  Don’tmergecodeunlesstestsarewriienandpeerreviewed

3.  Createtesttemplatesformanualtests,followingtheautomatedxUnitmodel(Setup,Test,Teardown)

4.  Spend10%ofeverysprint/releaseautoma,ngtheexis,ngmanualtests

5.  Promoteacompanyculturewhere“Development”==“Produc,onCode”+“TestCode”


I’mDownWithTDD•  TestDrivenDevelopmentisthemostefficientwayto

develop*

•  Completelypairstestcodewithdevelopmentcode

•  Greatercertaintyof100%codecoverage

•  Ensuresyourproduc,oncodeisDesignedForTest(DFT)

•  Notforthefaintofheart*AddPair-Programmingandyourefficiencyimprovesevenmore


ShowMeTheMoney•  Exercise: Assignacostvaluetorunatest•  Goal: Con,nuouslyiterateandreducethecostofatest


Variable Value Cost

TestExecu@on 1.  Manual2.  Semi-Automated3.  Automated

$$$$$$

Testenvironmentsetup 1.  EveryTime2.  Once

$$$$

Tes@ngLoca@on 1.  CustomerSite2.  RemoteTestLab3.  In-HouseTestLab4.  Engineer’sDesk/BuildServer

$$$$$$$$$$

Testrepeatability 1.  AllVariablesNotUnderControl2.  AllVariables100%theSame

$$$$

TestResultEvalua@on 1.  Manual2.  Automated

$$$$

v1

Example•  Product: CommercialUAV(quadcopter)•  Test: AssertfirmwarecontrolalgorithmholdsUAV

posi,onundervariouswindcondi,ons


Variable (1)OutintheField (2)WindTunnel (3)Simulated

TestExecu@on Manual:$$$

Semi-Automated:$$

Automated:$

Testenvironmentsetup

EveryTime:$$$

EveryTime:$$$

Once:$

Tes@ngLoca@on Mul,pleRemoteSites:$$$$

In-HouseTestLab:$$

BuildServer:$

Testrepeatability NotUnderControl:$$$

AllVariablestheSame:$


TestResultInterpreta@on

Manual:$$$

Manual:$$$

Automated:$

Totals Time:2daysPeople:2pilotsFrequency:EveryReleaseCost:$16

Time:1dayPeople:1TestEngineerFrequency:EverySprintCost:$11

Time:10minPeople:0Frequency:EveryBuildCost:$5

v1






Semi-Automated:$$

Automated:$


EveryTime:$$$

EveryTime:$$$

Once:$


In-HouseTestLab:$$

BuildServer:$





Manual:$$$

Manual:$$$

Automated:$




v1






Semi-Automated:$$

Automated:$


EveryTime:$$$

EveryTime:$$$

Once:$


In-HouseTestLab:$$

BuildServer:$





Manual:$$$

Manual:$$$

Automated:$




v1

ToAutomateorNotToAutomate•  There’sliiledebate:100%Automatedtestsarepreferredto

Manuallyruntests

•  But,ittakes,meandmoneytosetupandvalidateautomatedtests

•  Canspend$1M’sinves,ngincompleteautoma,ontools,or$1k’sinmanualtes,ng.Needtofindtherightbalance


Simulate,Simulate,Simulate•  Architectyourcodetoruncross-plaMorm

•  RequiresaproperHALdesign•  It’sgoodprac,ceanyway,forfutureexpansiontonewplaMorms

•  Runapplica,onlayercodeontestserverplaMorm•  Invest,mewri,ngstubsandmocks:ExcellentROI


Applica,onCode

APIs

HardwareAbstrac,onLayer(HAL)

Drivers

TargetHardwarePlaMorm

Applica,onCode

APIs

HardwareAbstrac,onLayer(HAL)

StubsandMocks

TestHardwarePlaMorm

=

≠

TargetProduct TestPla9orm

v1

StubsandMocks•  Veryimportant

•  StartwithGPIO

•  Builduptoexternals,muli(e.g.sensorvalues)

•  Use3rdpartytools;don’tcreateyourown

•  SeeWebformoredetailsonwri,ngStubs,Mocks,Fakes,etc.


NothingBeatsTheRealThing•  Therewillalwaysbeaneedtos,llruntestsonyourhardware

plaMorm

•  NeedtorunontargetplaMorminordertotest:•  Realworld,ming•  ISRs•  RAM,Stack,Heapu,liza,on•  etc.

•  BoiomLine:Sometestsarejusttoodifficultorexpensivetoautomate


BringOnTheHard•  RuncodeontargetplaMormthroughinterfacetobuildserver

•  UsesoOwarecontrolledexternaltoolstosimulateenvironments,muli

•  Requiresbuilding:•  Customtools•  APIstotalktoyourproduct•  Interfacesbetweenyourfounda,on3rdpartytesttoolsandyour

specificproduct


github CI

Ruby:raketestAPI:RPCfunc,ons Target

PlaMorm

UART/USB

gcc:compile

v1

ForRealz•  Automatedtestfixturetosimulateenvironments,muli

(C)2016-MichaelCorr 23 It’snotalwayspre=y,butitcanbebeau@ful

v1



TargetProductsNetworkedProgrammers

It’snotalwayspre=y,butitcanbebeau@fulv1



VoltageReferences

TargetProducts

USBControlledRelays

NetworkedProgrammers

It’snotalwayspre=y,butitcanbebeau@fulv1

DFT•  DesignForTestisoOenoverlooked,whichleadstoinefficient

testsandinefficient,medevelopingtests.i.e.waste

•  FirmwareandHardwarerequire‘hooks’tosimplifytes,ng•  Valuesrequiredbyteststovalidateasser,onsneedaccessormethods•  Electricalsignalsneedeasilyaccessibletestpoints•  Produc,oncodearchitectureneedstobelayeredproperlytoeasily

stub/mockorshim

•  Eveniffollowingbestprac,cesandaddingtestsearlyinthedevelopmentcycle,s,llneedtodoitright•  Justlikeproduc,oncode,testsneedtobewriienwelland

maintained(i.e.peerreviewed,versioncontrolled,refactored,etc.)•  Ensuretestsexecutequickly•  Ensureinterfaceprovidesforextensibility


UnitTestResources•  TestFrameworksforEmbeddedCEnvironments

•  Ceedling(Unity,CMock)hip://www.throwtheswitch.org/ceedling/

•  GeneralunittestframeworksthatcanbeusedforEmbedded•  Free:CppUtest,GoogleTest,GoogleMock,Minitest•  Paid:ParasoO

•  FullList:

hips://en.wikipedia.org/wiki/List_of_unit_tes,ng_frameworks

•  Findonethat•  IsFree•  Includesstubandmocksupport•  IntegrateseasilywithyourCIbuildserver•  Isanac,veprojectandhasawellsupportedcommunity

•  JamesGrenningBlog:hips://wingman-sw.com/renaissance/ (C)2016-MichaelCorr 27v1

•  Unit•  Testssmallestunitofcode•  Small/quickteststoensureastrongcodefounda,on

•  Integra,on•  Testsinterfacesbetweenunits•  EnsuresAPIsarecorrectanddon’tregress

•  System•  Higherlevel,end-to-endtes,ng•  Catchesuserinterac,on

•  Environment•  Testacrossvariousenvironmentalcondi,ons(temp,humidity,etc.)•  Ensuresyourproductfunc,onsacrossrealworldcondi,ons

TestTypes


Who’sGotTheTime?•  Asyourlibraryofautomatedtestsgrows,the,metoexecute

willincrease

•  ThevalueofusingCon,nuousIntegra,ondiminishesifteststaketoolongtocompleteaOereverycommit

•  Con,nuousIntegra,onbuildsshouldcompletefromcodecommittofinalresult<15minutes•  Developersloseefficiencyifwai,nglongerthanthat•  Developersstoplookingatresultsiflongerthanthat(moveontonext

task)


Separa,onAnxiety•  Segregateyourtests


Category RunTime TestTypes

EveryBuild <15min •  Currentsprint/releasefeatures•  “Younger”code•  Cri,calcode

EveryNight(Start:1am)

<8hours •  “Old/Robust”code•  Lesscri,calfeatures•  Longerrunningtests

EverySprint/Release •  EnvironmentTests•  ManualTests

v1

MeasureTwice,CutOnce•  Hardwarebugsaremoreexpensiveanddifficulttofixonce

productsarereleasedtocustomers,comparedtoWebandMobileproducts

•  Hardware•  TakataAirbagrecall$360M•  MarsOrbiter$125M

vs

•  SoOware •  FacebookViewsBug:3hrs


Takeaways•  Don’toverlookthevalueof(automated)tes,ng

•  Assignacosttoimplementandexecuteeachtest

•  Con,nuetolookforwaystoreducethecostofatest

•  Timeismoney,andstartupsdon’thavemuchofeitherone-sousewhatyouhave,wisely.•  UseCostsandROItoevaluatedecisions

•  InvestmentsinautomatedtesttoolshaveahugeROIforhardwareproductdevelopment


ThankYou


[email protected]/in/michaelgcorr

v1

Documents

Automated Tes,ng for Embedded Products · Automated Tes,ng for Embedded Products Michael Corr June 8 ... Total Cost $60 $760 +$700 ... Need to ﬁnd the right balance v1 (C) 2016