Guidelines on Mobile Testing - FINAL · should be pre-installed on emulators if used. Factory default settings for physical mobile devices (as opposed to virtual machines) are appropriate

GuidelinesonMobileTesting

Copyright©2016Anti-MalwareTestingStandardsOrganization,Inc.Allrightsreserved.Nopartofthisdocumentmaybereproducedinanyform,inanelectronicretrievalsystemorotherwise,withouttheprior

writtenconsentofthepublisher.

2

NoticeandDisclaimerofLiabilityConcerningtheUseofAMTSODocuments

ThisdocumentispublishedwiththeunderstandingthatAMTSOmembersaresupplyingthisinformationforgeneraleducationalpurposesonly.Noprofessionalengineeringoranyotherprofessionalservicesoradvice isbeingofferedhereby. Therefore,youmustuseyourownskillandjudgmentwhenreviewingthisdocumentandnotsolelyrelyontheinformationprovidedherein.

AMTSObelievesthattheinformationinthisdocumentisaccurateasofthedateofpublicationalthoughithasnotverifieditsaccuracyordeterminedifthereareanyerrors.Further,suchinformationissubjecttochangewithoutnoticeandAMTSOisundernoobligationtoprovideanyupdatesorcorrections.

Youunderstandandagreethat thisdocument isprovidedtoyouexclusivelyonanas-isbasiswithoutanyrepresentationsorwarrantiesofanykindwhetherexpress, impliedorstatutory. Without limitingthe foregoing, AMTSO expressly disclaims all warranties of merchantability, non-infringement,continuousoperation,completeness,quality,accuracyandfitnessforaparticularpurpose.

InnoeventshallAMTSObeliableforanydamagesorlossesofanykind(including,withoutlimitation,any lost profits, lost data or business interruption) arising directly or indirectly out of any use of thisdocument including, without limitation, any direct, indirect, special, incidental, consequential,exemplary and punitive damages regardless of whether any person or entity was advised of thepossibilityofsuchdamages.

Thisdocument isprotectedbyAMTSO’s intellectualpropertyrightsandmaybeadditionallyprotectedbytheintellectualpropertyrightsofothers.



3

AMTSOGuidelinesonMobileTesting

ExecutiveSummary

Thepurposeof thisdocument is toexplorethe important issuesthatexistwhentestinganti-malwaresoftwareondifferentmobiledevices.

While the document focuses on mobile platforms it is important to understand that little of thisoverridesthepreviously-publishedAMTSOprinciplesofgoodtesting.Assuch,thisdocumentshouldbereadinthecontextoftheseotherdocuments.Forexample,correctsampleselectionisjustasimportantinthemobilespaceasitisonthePCplatform.

Perhapsthemost importantdifferencesbetweenPCandmobiletestingrelatetotheextraconstraintstowhichmobiledevicesaresubject.Theseincludelimiteduserprivileges,bandwidthandpower.

Withpower consumption it is important that the realnuancesofpowerdrainaremeasuredproperlyduringa test. Forexample,a significant contributor todrainingpower fromaphone is its screenandradio.Theamountofpowerusedcanvaryduetotimeofday(auto-dimmingofthescreenatnight)andnetworkavailability(thephonemaytryandboostitssignalundercertaincircumstances).Bothoftheseeffects(andothers)cancausesignificantdifferencesinbatteryusage.

Manyendusershavephoneplansthatincurchargesastheyareusedandsobandwidthuseisofgreatinterest.Furthermore,forthoseproductsthatleveragecloud-basedcapabilities,connectivitymayhaveanimpactonaproduct’sabilitiestoprotectthedevice.

Finally,theconfigurationofthephoneiscriticalandmustbedescribedcompletely.Agoodexampleofhow important this canbe iswhether thephone is “rooted”or not.A rootedphoneallowsboth themalwareauthorandtheanti-malwaresoftwaremoreaccessandcan,inprinciple,changetheresultsofatest.Suchconfigurationdetailsneedtobeprovidedtothereadersotheycanreadthetestincontext.

Ultimatelytherecommendationsarethattestersdocumenttestindetail,sothatanotherpersoncouldattempttoreplicatetheresultspresented.

Introduction

Thisdocumentcoversissuesthatarespecifictothetestingofmalwareandsecuritycountermeasuresonmobiledevices,suchthattestsaremeaningfulandrepeatable.Thedocumentoutlinesadditionalissuesinvolved in best practice testing, above and beyond other AMTSO guidelines and best practices. Thisdocumentisnotacomprehensivelistingofallsuchissues.

Unless otherwise defined herein, all terms included in this document are used with their commonmeaning.Thispaperusestheterm‘mobiledevice’torefertoAndroid-basedsmartphones,tabletsandotherpersonaldevices.



4

AMTSOdocumentsarebest read inconjunctionwith theFundamentalPrinciplesofTesting andotherdocumentsontheAMTSOdocumentspageatwww.amtso.org.

Scope

Thisdocument isdesignedto focuson the testingofanti-malwaresolutionson theAndroidplatform.We have chosen to focus on Android because of its prevalence, rapid adoption, the availability ofantimalwaresolutionsandthefactthatitisbeingactivelyattackedby“realworld”threats.

The commentsmadehere areprimarily applicable tomobiledevices though inprincipalmanyof therecommendationswillapplytoanyAndroidbasedsystem.

SimilaritieswithPCPlatforms

Inmanyways, the testing concepts that have beenwell-developed on the PC platformmay be usedwhentestingmobiledevices.AndroidphonesareessentiallyLinux-basedpersonalcomputersthatcanprovide similar levelsof computingpower to that availablewith reasonablymoderndesktopPCs. Forexample,theGoogleNexus4smartphonehasa1.5GHzquad-coreprocessorand2GBRAM,whileitwasstill possible at the time of writing to buy a similarly-specified laptop PC (albeit with an Intel/AMDprocessor).

OnemajordifferencebetweenPCsandAndroid(andAppleiOS)devicesisthatthelattermobiledevicesrestrict the user’s control over the system.However, it is possible to gain full access by ‘rooting’ thedevice,whichbypassestheinbuiltprotectionandbestowsadministrator-level(or‘root’)privilegesuponthe user. While rooting increases the possibilities for monitoring the system, there are less positiveissuesinvolvedwhentestingsecurityproducts.Formoreinformation,seethesectionentitled“RootingandElevatedPrivileges”inthisdocument.

RealWorldAnti-MalwareTesting

MalwareSamples

Itispreferabletotestinasimilarwaytotheexperiencesthatusershaveintherealworld.Thismeansusingaselectionofmalwaresamplesthatarecomingintocontactwithusers.

Productsshouldbeconfigured ina realisticwayandthird-partyappsusually foundonmobiledevicesshould be pre-installed on emulators if used. Factory default settings for physicalmobile devices (asopposedtovirtualmachines)areappropriate.

Malwareshouldbeintroducedtothetarget inarealisticway,fromappropriatethird-partyappstoresforexample.

Emulators

Emulatorsprovideawiderangeofpossibilitiesformonitoringandmanipulatingthe installedsoftware(includingmalware).However,theyarealsolimitedbyanumberoffactors.Theseincludealackoftrue



5

GSMconnectivity,whichprecludestheabilitytoconnecttheemulatedsystemtorealvoicecallorSMSservices.

Other features, such as Bluetooth and degrading battery charge level and charging states, wereunavailable at the time of writing. Battery levels can be set, but won’t expire. For more on testingmobilebatteryuseseethesectionentitled“BatteryDrainMeasurement”inthisDocument.

Emulators for malware testing are not sufficient even for legitimate software (FP) testing as not allsoftware will be fully functional in an emulated environment. For example, malware may check formobiledevicedetailsthataremissingfromemulateddevices.Themaliciousapplicationmaynotinstallasaresult.

Emulatorsareveryslowtorun,evenwhenrunonpowerfulPChardware.

BatteryCharge

Some anti-malware products disable themselves when the device’s power runs low. Testers shouldmonitor this situation and note if andwhen an anti-malware product stopsworking effectively. Oneoptionmightbe to introduce theEICAR fileor someother standarddetectable file repeatedly.Otherfeatures may also take priority. Some apps may become more aggressive when external power issupplied,which could influenceperformance impact testing.Aproductmay scanmore thoroughly orcreatebackupfiles.Thiscouldaffectdataandprocessoruse.

One solution is to ensure that the phone in continually charged. Another is to note that in real lifephones do run out of power and that some products will cease to protect them under specificcircumstances.

Testers can override the device’s own power management system and tests can be conducted atdifferent(simulated)levelsofbatterycharge.Seethesectionentitled“BatteryDrainMeasurement”inthisdocumentformoreinformation.

Pre-InstalledSoftware

Some mobile devices could come with anti-malware software preinstalled, which means that thesecurity product could be given root privileges. This in turn could change how well this productperformsonthatplatformasopposedtoothers.Itmaybemoreorlesseffectiveandthisconfigurationmustbenotedclearlyinthetestreportandtheconclusionsshouldreflectthisfact.

Somemobiledeviceversionsmaybemoreorlessvulnerable,sothistoocouldaffecttestresults.Thiscouldalsoaffectsampleselectionifusablesamplesarerequiredtotestananti-malwareproduct’sfullprotectioncapabilities.

Testersareadvisedtocompareproductswithsimilarprivileges.Ifthisisnotpossibleordesiredthentheconclusionsofthetestshouldmakeclearthatcertain(pre-installed)productsenjoyedtheadvantagesofhavinghigherprivilegesthantheother(retro-installed)anti-malwareproducts.



6

Notealsothatpre-installedanti-malwareproductsmaybeoutdated.Newerversionsmaybeavailablefromappstores.

Connectivity

Mobiledeviceshavedifferentlevelsofconnectivityvia3G,4G,Wi-Fiandslowertechnologies.Thiscanaffectthefeaturesandfunctionsofsecuritysoftwareandthreats.

Anti-malware products on desktop platforms often perform frequent queries to backend servers andalmostall require regularupdates. The same is at leastpartially true for today’smobileanti-malwareproducts. Efficacymay suffer at slower speeds, as someproductsmay either fail in their attempts tocommunicatewith the vendor’s systems ormay elect not to use up all of the bandwidth in order toimproveauser’soverallexperience.

Locationandtimecanaffectthespeedandpowerconsumptionofdevicesduetonetworkcongestion.Weatherandatmosphericconditionscanalsocausemeasurabledifferencesandshouldbeconsidered.

Formoreinformation,seethesectionentitled“Bandwidth”inthisdocument.

MobileDeviceManufacturers

Differentmobiledevicemanufacturersand/orcarriersfrequentlyhavetheirownsetsofconfigurationsfor theAndroidoperating systemdifferent fromthestandardbuild. Someof these settingscanmakethedevicemoreorlessvulnerableandthusaffectthetestresultsandconclusions.ComparedwithPCs,differences inhardware formobilearemoreprofoundso there’smoreneed for specificityandscopeduringtests.

Ideally,itisrecommendedtousethemostcommonmodelofamobiledevicewiththemostdeployedfirmwareimageofAndroid.Agoodchoicewouldbeapopularmodelthatisavailablewithafirmwarethathasnotbeenmodifiedbyamobilenetworkoperator/carrier.

Forexample,atthetimeofwritingtheGoogleNexus4phonewaspopular,inexpensiveandlackinganythird-partyappsormodificationsmadebyamobilenetworkoperator.Itfollowsthatpre-installedanti-malwaremaynotbeideal,asitis(currently)theresultofacarriermodification.

TestersmayconsidercreatinganddeployingtheirownAndroidimagesinthesamewaythatsomelargebusinessdo.Thiswouldpermitacompletelyconsistentandcontrolledtestingenvironment,inmuchthesame way as testers install relatively clean installations of Windows when testing desktop PC anti-malwareproducts.However, customAndroid installationsarenotcommon in the realworldso thosewhocreatethemfortestsshoulddocumentthoroughlyallstepstaken.

Exact version information should be provided for the firmware and hardware used in the test. Inmid2013therewerethreedominantversionsofAndroidinpopularuse:

• Gingerbread(2.3.3-2.3.7)-39.7%

• IceCreamSandwich(4.0.3-4.0.4)-29.3%



7

• JellyBean(4.1.x)-23.0%

(Source:http://developer.android.com)

CommonTestingIssues

There are many issues that could have a significant impact on real world testing. The following listhighlightssomeareasthattestersshouldpayparticularattentionto.Aswithanyreal-worldtesting,itiscriticaltoensurethatthetestingenvironmentandotherelementsofthetestapproximateascloselyaspossiblethemostcommonly-usedhardware,software,networks,configurationsandothervariables.

Attheveryleast,testersshouldnotethefollowingdetailsastheyappearinthetestenvironment,eveniftheychoosenottoaddressthem.

1. To root or not to root? See the section entitled “Rooting and Elevated Privileges” in thisdocument.

2. Allow/disallow installation of applications from third-party markets. E.g. “Unknown sources”setting“allowinstallationofappsfromsourcesotherthanthePlayStore”.

a. This affects sample selectionofmalware and legitimate software (for ‘FP’ testing)significantly.

b. Third-partymarkets are enabled by default inmany cases. Testers should explainwhytheychosetoincludeorexcludesuchmarkets.

3. DiscoverifTrustedCredentialsisturnedonoroffbydefault.

4. Non-standard third-partyapplications (e.g. remoteconfigurationsoftware)maybehiddenbutrunninginthebackground,interferingwithsecuritysoftware.Non-standardsystem,application,userinteractionandperformance-monitoringsoftwaremayinterferewithsecurityproducts.

a. Note:SomecarriersincludingAT&Tmonitorsomeactivityonthephone.

5. Non-standard system configuration may prevent security software from functioning. Theseinclude:

a. SD-cardmounting.AnAPKcreatesafileandthencreatesamounteventeachtime.This event can bemonitored by anti-malware software. In this scenario the anti-malware productmay impact on system performance as itmonitors eachmountanddismount.

b. Theuseofroamingblockingandothernon-standardsettingsprofiles.

6. Howmessaging,email,browsingandsoftwareinstallationisintegratedintheUI.Non-standardapps and configurations may affect the anti-malware product’s ability to produce effectivealerts.Non-standard configurationsmay also interferewith security products’ functionality astheseapplicationsarethemostcommonareasmonitoredsincetheyaretheprimaryvectorsofinfection.

7. Missing standard software such as the web browser, messaging apps, the system lock andGooglePlaystore.Thesesituationsmayarisewhenproviding locked-downprofilessuchasforchildren.



8

8. ThepresenceandconfigurationoftheGoogleappverificationfeature(“Verifyapps”–“Disalloworwarnbefore installationofapps thatmaycauseharm“).This ispresent inAndroid4.2andcouldbedisabledatthetimeofthiswriting.Note:ifitis“on,”itmayinterferewithtestingthedetectionabilitiesofanti-malwareproducts.

Bandwidth

Most anti-malware products utilize bandwidth to be able to function comprehensively, if notcompletely. In contrast toPCs,bandwidth canbea scarce resource inmobiledevices, especiallywithusers with budget data plans, so it can be useful to compare bandwidth usage of different securityproducts.

Asageneralobservation,anti-malwareproductsshouldaimtobesmartenoughtoavoiddatausageonlimitednetworks.Thefollowingareasorfeaturesareofinterestwhenplanningtests:

1. Cloudlookuptechnologyisheretostaysotestingtheaveragebandwidthconsumptionofeachproductislikelytobeofsignificantinterest.

2. Databaseupdatescanhaveaneffectonbandwidthuse.

3. Does the product avoid cloud lookup and database update function when the phone is inroaming mode? This information informs a usability study about bandwidth use and is alsoextremely important when conducting anti-malware protection testing. For example, if livedatabaselookupsareavoidedbecausethephoneisinroamingmodethenthisfactneedstobemadeclearinthereport.Unlessthereal-worldtestisofanti-malwareprotectionwhileroaming,itisadvisabletodisableroamingmodeinsuchcircumstances.

4. Check if the product performs database updates only onWi-Fi connections by default. If so,malware exposures should be made using the Wi-Fi connection or else the report needs toexplaintheinherentlimitationfacedbytheproduct/user.

5. Howmuchdatadoestheproductsendtothebackend?

FeaturesinMobileSecuritySuites

Android AV products may contain more security features than the traditional anti-malware andpotentiallyunwantedapplication(PUA)filedetections.Suchfeaturesmight includesafewebbrowsingand host intrusion prevention systems (HIPS) protection layers. They may also lack some featurescommonlyfoundindesktopanti-malwareproducts,suchastheabilitytoremovemaliciousapplicationsthathavebeeninstalledonthedevice.

Security features thatmaybepresent,butarenotdirectly related toanti-malware frequently includeanti-theftmeasures.Itispossibleandvaluabletofocusonthesetypesofspecificfeaturesonly,suchasin a test to determinewhich products aremost effective in locating and/or disabling a stolen or lostphone.

However,whenconductingawholeproductanti-malwaretestitisimportantnotonlytoinvestigatethefile detection capabilities of an anti-malware product but the other protection features the product



9

offers. These features canwork hand-in-hand to bring about a significantlymore effective protectionexperienceforusers.

Exposingamobiledevicetothreatsusingrealisticmethodscanexposedifferentlayersofprotection.Suchmethodsmightincludeoneormoreofthefollowingwaysinwhichtheuserobtainsanapplication:

• Browsing to a certain URL for downloading apps via exploits and/or social engineeringtechniques.

• Visitingofficialor third-partyappmarkets thathostmalicious software that spreadsvia socialengineeringtechniques.

• ‘Side-loading’theappfromaremovablemediasuchasanSDcard.

Furthermore,itwouldbeusefultomonitorandrecordhowthesecurityapplicationshandleencounterswithmalware.Thefollowinglistcontainssomepossibleresults:

• Recognizing the app before installation: Detecting and differentiating themalicious/PUA Appfromlegitimateapps.

• Recognizing the running app: Detection of possible suspicious app behaviors during itsexecution.

RootingandElevatedPrivileges

Thequestionof‘rooting’devices,inwhichausergainsalevelofcontroloverthesystemthatishigherthanallowedbyadefault installation,doesnothaveaclearanswer.At thetimeofwriting, it is likelythat most consumers are using handsets that have not been rooted. Therefore, a real-world testoperatingontheprinciplesthattheequipmentusedshouldbeasclosetothenormaspossibleshouldavoidusingrootedphones.

Applications installed by a user (such as anti-malware software) do not have sufficient rights touninstall/removeanotherinstalledprogram.Atthistimeofthiswriting,Googlecouldachievethisgoalautomatically,using itsso-called ‘killswitch’,which isabletoremoveanapplicationbut isnotabletoclean up files that the malicious application has spread across the system. However, anti-malwareprogramsmay leadauser toagree to removeamaliciousapplication–perhapsmaking that removalmoreeasythanwouldotherwisebethecase.

Some mobile devices could come with an anti-malware product pre-installed. It is generallyrecommended not to use such mobile devices for testing due to the unique advantage that thepreinstalled security product may enjoy due to its superior system privileges compared to productsinstalled by a user. Using a standalone version of the product is preferable in normal situations.However,ifitbecomescommonpracticeforhandsetsupplierstopre-installanti-malwareproductsthencomparingsuchhandsetswithdifferentproductsseemsfair.

Iftestingananti-malwareproductonarooteddeviceisnecessaryitfollowsthattheotherproductsdueforcomparisonshouldbeinstalledonrooteddevicestoo.



10

CorporateEnvironments

Corporate mobile devices may be more managed than the consumer counterparts and ought to beconfiguredtowardmaximumprotection.Suchdevicesmayhavethefollowing‘default’settingsimposedbyadministratorsorsuppliers,whichmayhaverootedthephonetoachievecertaingoals:

• Usingacarrier-specificorcustom-madebuildoftheAndroidoperatingsystem.

• Unknownsourcesnotallowed.

• Anti-malwareproductmaybepre-installed.

• Remoteconfigurationsoftwaremaybeinstalled.

When constructing a test of products and/or handsets intended exclusively for business use, testsshould consider all of the above possibilities and research common customizations made by largebusinessestomobiledevices.

BatteryDrainMeasurement

Thismethodissuitableonlyforaspecificsubsetofphoneswithremovablebatteries.

Measuringthebatteryconsumptionofmobiledevices,especiallyinthecaseofsmartphones,isnotaseasyasitlooks.Onceastrongmethodologyisinplaceitwillbepossibletocompareaphone’sbaselinepower consumption without an anti-malware product with its consumption with security softwareinstalled.

Formeasuringbatteryconsumptionofmobiledeviceswerecommendthefollowingsteps:

1. Ensureastabletestingenvironmentthatmaintainstemperatureandotherphysicalvariables.

2. UseISOcalibrateddevicesformeasurement.

3. Overrule the phone’s powermanagement to avoid influence by lighting conditions and localsignalstrength.Alsostandardizevolumelevelstoconsistentlevels.

4. Useautomatedtasks/workloadsthatarerepeatable.

5. Resultsmustbemeasuredinhighdetail,witharesolutionofnanowatts.

6. UseadedicatedWi-Finetworkforthetest.

7. UseadedicatedUMTSbasestationforthetest.

8. Measurethecurrentlostduetoresistanceinthecablesbetweenthemeasuringdeviceandthephone.

ISOCalibration

It is very important to use calibrated measurement devices, as the measurement differences couldinfluencethetestresults.Whenusingnon-ISOcalibrateddevicesensurethatthemeasuringiscorrectbyusingafrequencymeasurement.



11

MeasuretheLossofCableConnectionTomeasurethecorrectbatteryconsumptionofadeviceinhighdetail,downtoananowattlevel,andtodiscover any influence on that consumption by an anti-malware program to the device, it is veryimportanttoexcludeany influenceofcableresistance.So it isnecessarytodeterminethepower losscausedbythewiringbetweenthedeviceandthemeasuringequipment.

The following chart demonstrates the basics to get SMU specified performances to understand thewiringrequirementswhenusinganappropriatedevice.

Figure1:Wiringrequirementswhenconnectingapowermeasurementdevicetoamobiledevice(CopyrightAV-Comparatives/AgilentTechnologies)

PowerManagement

Sometimesthemobiledevicewilltrytoadjustthebrightnessofitsscreen(orotherfeatures)accordingtothelocalenvironment.Toavoidthishappening,asitwillaffectthepowerconsumption,themobiledevice’sautomaticpowermanagementsystemshouldbedisabled.

Wi-Fi

ConnectionstotheWi-Fibasestation,andthedistancebetweenthemobiledeviceandthebasestation,influencethebatteryconsumption.EnsurethatthedistanceandthenetworkconnectionstotheWi-Fibasestationremainthesameforeachtestcaseand,ideally,forallteststhatwillbecomparedwitheachotherinthefuture.



12

UMTS/3GBaseStation

UseadedicatedUMTSbasestationwhile testing.Generally-availableUMTS/3Gnetworks,asprovidedby regular mobile operators in a tester’s locality, will cause fluctuations in the power measurementresults,whichmaybecausedbyweather,otherconnectedphonesandvariousotherfactors.

SimulatingtheUser

Tomeasurebatteryconsumptionofananti-malwareproductproperly,itisessentialtosimulatetherealuser.Itdoesnotmakeanysensetomeasureonesingleoperation,suchasa30-minutephonecall,andnothingelse.Trytoanalyzeuserbehaviorandsimulatethiswhentestingbatteryconsumption.

Anexampleofauserbehavior survey isavailableatwww.av-comparatives.org. We recommend thattestersperformtheirownsurveystofittheirreaders’needs.

UseMeasurementsofNanowattResolution

InFig.2belowyoucanseetheconsumptionofasmartphonebothidleandhandlingavoicecall.

Figure 2: Measuring power consumption requires specialized hardware andsoftwareexternaltothedevicebeingtested(CopyrightAVComparatives)



13

MistakestoAvoid

NeverUseMulti-MeterBreakpointMeasuring

Measuringbreakpointswithampereandvoltreadings,andcalculatingthewattage(evenifyousetitto100pointsasecond)doesnotgiveaccurateresults.Theenergylevelofthedifferencewithandwithoutanti-malwaresoftwarepresentistoosmalltobemeasuredwithbreakpointmeasuring.

You cannot synchronize to breakpoints accurately, even if it is two-channeled device. The peakssometimesonlyoccurinmillisecondssotheywillnotalwaysbemeasured.

A battery drain analyzer is needed for long-term measurement of true battery consumption. Thosemeasurement devices are extremely expensive; however, there may be opportunities to rent therequiredequipment. AnAMTSOmemberhas indicatedthattheymaybeabletohelpotherssecureaproperdevice;formoredetailscontactAVComparatives(http://www.av-comparatives.org/contact/).

RemovetheBattery

Do not measure power consumption with a battery installed inside the phone. This influences theresultsasthechargingstatusofthebatterycanvary.

Conclusions

In this paper we have outlined some of the special considerations that are required when testingsecuritysoftwareonmobiledevicessuchassmartphonesandtablets.Theseissuesareinadditiontothetypical challenges testers face with respect to false positives and sample selection. In particular, wefocusontheroleofthenetworkinpowerconsumptionandondeviceconfiguration.

Perhapsthemostimportantpointtonoteisthatthetestmustadequatelydocumentthemethodologyused for evaluation. In addition to the usual descriptive items, such as criteria for sample selection,mobiledevices also require a fairly sophisticateddescriptionof device configuration andpowerdrainmethodologyifpowerconsumptionistobereported.

Finallywenote that, as is sooften thecase, theability todocument the testadequately so it canbeanalyzedandreproducediscriticaltomaximizingthetest’sutility.

______________________________________________________________________________

ThisdocumentwasadoptedbyAMTSOonFebruary20,2014