2013-04 CAR Browser Socially Engineered Malware 130513c

7/30/2019 2013-04 CAR Browser Socially Engineered Malware 130513c

1/18

2013NSSLabs,Inc.Allrightsreserved. 1

BROWSERSECURITYCOMPARATIVEANALYSIS

SociallyEngineeredMalwareBlocking

2013-RandyAbrams,JayendraPathak,OrlandoBarrera

TestedVendors

Apple,Google,Microsoft,Mozilla,Opera

OverviewThewebbrowseristheprimaryvectorbywhichmalwareisintroducedtocomputers.Linksinphishingemails,

compromisedwebsites,andtrojanizedfreesoftwaredownloadsalldelivermalwareviawebbrowser

downloads.

Thewebbrowserisalsothefirstlineofdefenseagainstmalwareinfection.Browsersmustprovideastronglayer

ofdefensefrommalware,especiallyinmobileoperations,ratherthanrelyinguponthird-partyanti-malware

solutionsandoperatingsystemprotections.Thistestexaminestheeffectivenessoffiveleadingwebbrowsersin

blockingsociallyengineeredmalware.

Fiveleadingbrowsersweretestedagainst754samplesofreal-worldmalicioussoftware.Majordifferencesinthe

abilitytoblockmalwarewereobserved.Datarepresentedinthisreportwascapturedover28daysthroughNSS

Labsuniquelivetestingharness.Thedataprovidesinsightintothebuilt-inprotectioncapabilitiesofmodern

browsers,includingChrome,Firefox,InternetExplorer,Opera,andSafari.

Beyondhowmuchmalwareisblocked,thedefinitionofblockedandthetechnologiesthatareusedtoachieve

protectionsmakeasignificantdifferenceintheusefulnessofthatprotectionandinitsreliability.Ifblocked

includesasituationwhereauserisissuedawarningasopposedtobeinggivennochoice,theeffectivenessof

blockingisaffected.

Ifa100%falsepositiveacceptancerateisacceptable,itistrivialtoprotectusersfromallmaliciousdownloads.

Withjustafewlinesofcode,Firefox,Safari,andOperacoulddisplaceInternetExplorerandChromeastheleaders

ofprotectionagainstsociallyengineeredmalware.However,describingeverydownloadasmaliciouswould

breaktheInternet.Findingabalancebetweenaccuracyandsafetyisthechallengeforbrowsersatthefrontof

protectiontechnology.


2/18

NSSLabs BrowserSecurityComparativeAnalysisSociallyEngineeredMalwareBlocking


BothFigure1andFigure2illustratethischallenge.

Figure1-OverallMalwareBlockRateByBrowser(HigherValuesAreBetter).

Figure1showsthatMicrosoftandGoogleareaheadofApple,Mozilla,andOperaintermsofbuilt-indownload

securityprotection;however,furtheranalysisisnecessarytoexplainadequatelythedifferencein99.96%and

83.16%protectionratesbetweenInternetExplorerandChrome.Thesedifferencesinprotectionarefarfrom

linear.

Figure2-BlockingTechnologiesUsedByBrowsers(HigherIsBetter).

MicrosoftsApplicationReputationandGooglesDownloadProtectionarefundamentallybothcontent

agnosticmalwareprotection(CAMP)schemes,howevertheextenttowhichthistechnologyisreliedonisan

importantdifferentiator,asthetechnologyisflawed.

CAMPtechnologyisbydefinitioncontentagnosticandthereforemoresusceptibletofalsepositivesanduser

error.InordertooffsetthehigherfalsepositiverateofCAMPtechnologiestheuserisgivenachoicetoblockor

allowcontentthatisflaggedaspotentiallyuntrustworthy,baseduponreputationalschemes.Goodsoftwarethat

isnotwellknownwillbeblocked.Malicioussoftwarethathasbeenengineeredtohaveexcellentreputational

aspectsmayevadeprotection.Dependingonanuntrainedusertomakethecorrectchoiceisunwise.

1.87%

9.92%

10.15%

83.16%

99.96%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Opera 12

Firefox 19

Safari 5

Chrome 25/26

Internet Explorer10

Block rate

1.87%

9.92%

10.15%

10.00%

83.17% 16.79%

73.16%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Opera 12

Firefox 19

Safari 5

Chrome 25/26

Internet Explorer10

URL Reputation Application Reputation Download Protection


3/18



Figure2showsthatwithoutCAMPtechnology,ChromedemonstratessimilareffectivenesstoSafariandFirefox.

TheuseofCAMPtechnologyallowsChrometoapproachtheprotectionratesofferedbyInternetExplorerpriorto

theincorporationofMicrosoftsownCAMP(ApplicationReputation)technology.Duringthetestingperiod,

InternetExplorer10hadameanmalwareblockrateof99.96%andChromehadameanmalwareblockrateof

83.16%.SafariandFirefox,withmeanmalwareblockratesof10.15%and9.92%respectively,providednegligible

protectionbutwerestillmorethanfivetimesmoreeffectivethanOpera,whichblockedonly1.87%ofthe

malwareinthistest.

Toputthenumbersinperspective,foreverytenwebencounterswithsociallyengineeredmalware,Firefoxand

Safariuserswillbeprotectedfromapproximatelyoneattack.Thisimpliesthatnineoutoftenbrowsermalware

encounterswilltestthedefensesofinstalledanti-virusorotheroperatingsystemdefenses.Chromeuserswillbe

protectedfromjustovereightoutoftenattacksandInternetExplorer10userswillgenerallybeafforded

protectionfromallbutabout4outof1,000sociallyengineeredmalwareattacks.Itshouldbenotedthatsomeof

thedownloadprotectionmechanismsrequireauserchoiceandthiscandecreasetheeffectivenessofthe

protections.Operausersareaffordedvirtuallynoprotectionagainstsociallyengineeredmalware.

TestedProducts

AppleSafari5

GoogleChrome25/26

MicrosoftInternetExplorer10

MozillaFirefox19

Opera12

InatestrunningfromMarch13,2013throughApril9,2013,over96,000testcaseswereusedinthedatasampling

capturedviaNSSuniqueLiveTestingharness.Aninitialsamplesetof11,296uniqueandsuspiciousURLs

enteredthesystem;754URLswerefoundactiveandmalicious,andmetthecriteriaforentryintothetest.Intotal,

550testrunswereperformedbythefivebrowsersagainsttheseunique754URLsresultinginover18,000testcasesperbrowser.

Testingwasrepeatedevery6hoursuntilthetargetURLwasnolongeractive.Samplesthatdidnotpassthe

validationcriteriawereremoved,includingfalsepositivesandadware.Ultimately,913URLtestcasespassedthe

post-validationprocessandareincludedintheresults.Eachsamplepayloadwasvalidatedinternally.

NSSLabsFindings Malwaredownloads(viawebbrowser)arethemostcommoninfectionvectorforcriminalsattemptingto

monetizemalwareviaaccount/passwordtheft,bank/financialfraud,gamingfraud,clickfraud,andbot

installation.

Theleadingbrowsersshowasignificantvarianceintheirabilitytoblockmalware.InternetExplorer10

hadthehighestmalwareblockrateat99.96%,followedbyChrome25/26at83.16%.Safari5andFirefox

19wereadistantthirdandfourth,with10.15%and9.92%respectively.Operaofferedvirtuallyno

maliciousdownloadprotection,witha1.87%score

Browserswithlowmalwareblockratesplaceconsumersatsignificantrisk.


4/18



ThedownloadprotectionofferedbyChromehascontinuedtoincrease.BothChromeandInternet

ExplorerbenefitsignificantlyfromfilereputationsystemscombinedwithURLreputationandsiteblocking

technologies.

NSSLabsRecommendations Usersshouldconsiderbrowsersecuritytobeacriticalpartoftheirsecurity.

Usersshouldselectbrowserswithhighermalwareblockratesinordertominimizerisk.

Usersoflesssecurebrowsersshouldconsiderantivirussuiteswithrobustwebreputationtechnologies. Usersshouldnotrelyuponbrowsertechnologiestoeliminatetheneedforbasicusersecurityeducation.


5/18



TableofContents

Overview................................................................................................................................1

NSSLabsFindings....................................................................................................................3

NSSLabsRecommendations...................................................................................................4

Analysis..................................................................................................................................7

SafeBrowsingvs.ApplicationReputation......................................................................................................................8

MalwareBlockPerformance......... ......... .......... .......... ......... .......... .......... ......... .......... ......... .......... .......... ......... .......... ........10

TheRealityOfApplicationReputation..........................................................................................................................11

TimeToBlockMaliciousSites......... .......... ......... .......... ......... .......... .......... ......... .......... ......... .......... .......... ......... .......... .....12

AppendixAMethodology...................................................................................................13

ClientHostDescription........................................................................................................................................................13

TestedBrowsers.....................................................................................................................................................................13

NetworkDescription.............................................................................................................................................................13

TestDuration.............................................................................................................................................................................14

TestFrequency..........................................................................................................................................................................14

SampleSetsforMalwareURLs...........................................................................................................................................14

Sources..........................................................................................................................................................................................14

CatalogURLs..............................................................................................................................................................................15

ConfirmSamplePresenceofURLs....................................................................................................................................15

ArchivalOfActiveURLContent....................................... ......................................... ........................................ ..................15

DynamicExecutionOfEachURL.......................................................................................................................................15

ScoringAndRecordingTheResults...................................... ........................................ ......................................... ...........15Pruning.........................................................................................................................................................................................16

Post-TestValidation................................................................................................................................................................16

NSSLabsTestEnvironmentandMethodology.......... ......... .......... .......... ......... .......... ......... .......... .......... ......... .......... 17


6/18



TableofFigures

Figure1-OverallMalwareBlockRateByBrowser(HigherValuesAreBetter).................................................2

Figure2-BlockingTechnologiesUsedByBrowsers(HigherIsBetter)..................................................................2

Figure3-FirefoxSafeBrowsingWarning...........................................................................................................................8 Figure4-SafariSafeBrowsingWarning.............................................................................................................................8

Figure5-ChromeSafeBrowsingWarning..........................................................................................................................9

Figure6-ChromeMaliciousFileBlocking...........................................................................................................................9

Figure7-ChromeApplicationReputationBlocking.......................................................................................................9

Figure8-InternetExplorerSmartScreenWarning.........................................................................................................9

Figure9-InternetExplorerAppRepWarning...............................................................................................................10

Figure10-MalwareBlockRateOverTime......................................................................................................................10

Figure11-TimeToBlockMaliciousSites.........................................................................................................................12

Figure12-VirtualMachineSpecifications.......................................................................................................................13

Figure13-NSSLabsBrowserTestHarness.....................................................................................................................14 Figure14-NSSLabsLiveIn-The-CloudTestFramework..........................................................................................17


7/18



AnalysisThisreportexaminestheabilityoffivedifferentwebbrowserstoprotectusersfrommalwaredownloads,also

knownassociallyengineeredmalware (SEM).

1

Modernwebbrowsersofferanaddedlayerofprotectionagainstthesethreatsbyleveragingin-the-cloud,reputation-basedmechanismstowarnusersofpotentialinfection.

However,notallvendorshavetakenthesameapproach.

AsthemostwidelyusedandubiquitousmeansofaccessingtheInternet,webbrowsersareuniquelypositionedto

preventmalwarefrombeingdownloadedorinstalled.Whenthebrowserfailstoblockathreat,itbecomesthe

burdenoftheantivirusandoperatingsystemstoprotectagainstinfection.Antivirussoftwarecanbelikenedtoa

goalkeeper;ifthedefenseallowstoomanyshotsongoal,somethingwilleventuallygetthrough.Thesesecondline

defenseshaveprovedtobeinadequatebythemselvesinprotectingagainstattacks.TheNSSanalystbrief,

CybercrimeKillChainvs.DefenseEffectiveness,demonstratesthatholesinonelayerofdefenseareoftennot

closedbysecondaryandtertiarytechnologies.

Tocomplementtraditionaldefensesandtoaddressthehighlydynamicnatureofcurrentattacksandattackdistributionmethods,modernwebbrowsersemploytechnologiesthatblockaccesstomaliciousURLs,before

loadingthecontent.BlockingaccesstomaliciousURLsisaformidablefirstlineofdefense,sinceitprovides

completeprotectionagainstmalwareenteringthesystem.Chrome,Firefox,andSafarialldemonstratethatthe

GoogleSafeBrowsingAPIaloneisnotuptothetaskofblockingmaliciousdownloads.GoogleaugmentsitsSafe

BrowsingAPIwithadditionaldownloadprotectionthatisseventimesmoreeffectivethantheSafeBrowsingAPI.

ThecombinationoftheSafeBrowsingAPIandGooglesdownloadprotectionputsChromeonaparwithInternet

ExplorersURLreputationandcomparabledownloadprotectionschemes,butMicrosoftsapplicationreputation

technologybolsterstheprotectionIEoffersagainstmaliciousdownloadsbyanadditional16.8%aboveChrome.

Browserprotectioncontainstwomainfunctionalcomponents.Thefoundationisanin-the-cloudreputation-based

systemthatscourstheInternetformaliciouswebsitesandcategorizesfilesaccordingly,eitherbyaddingthemto

ablackorwhitelist,orbyassigningthemascore(dependingonthevendorsapproach).Thiscategorizationmay

beperformedmanually,automatically,orbyusingbothmethods.Somevendorswillutilizefeedbackfromuser

agentsontheircustomersendpointstoreportbacktothereputationsystemautomatically,providinginformation

relevanttothetrustworthinessofapplicationsandfilesdownloadedfromtheInternet.Thesecondfunctional

componentresideswithinthewebbrowseritself,requestingreputationinformationfromthein-the-cloud

systemsaboutspecificURLsandthenenforcingwarningandblockingfunctions.

Whenresultsindicatethatasiteisbad,thewebbrowserredirectstheusertoawarningmessageorpage,which

statesthattheURLismalicious.IntheeventthattheURLlinkstoadownload,thewebbrowserinstructstheuser

thatthecontentislikelytobemalicious,andthatthedownloadshouldbecancelled.Conversely,whenawebsite

isdeterminedtobegood,thewebbrowsertakesnoactionandtheuserisunawarethatasecuritycheckwas

performed.

1Exploitsthatinstallmalwarewithouttheuserbeingaware(alsoreferredtoasdrive-bydownloads)arenotincludedinthisparticularstudy.
https://www.nsslabs.com/reports/cybercrime-kill-chain-vs-defense-effectiveness


8/18



SafeBrowsingvs.ApplicationReputation

ThecorefunctionalityofURLblacklistingistoprotectagainstdrive-bydownloads,asopposedtosocially

engineeredmalwaredelivery.NSSdeterminedthatGooglesSafeBrowsingAPIv2includesadditionaldownload

protectionthathasbeenintegratedintoChrome,butnotintoFirefoxorSafari.Thisfunctionalityprovides

reputationservicesforexecutablefilesor,asGoogledescribesthem,maliciousdownloads.InternetExplorer

usesadifferenttechnology,knownasApplicationReputation(AppRep),toblockmaliciousdownloads.AppRep

technologiesuseavarietyofsourcestosetathresholdofhowtrustworthyanapplicationappearstobe.Thisisnot

thesameassayingthatanapplicationisgoodorbad.Operausesseveralpartners,includingtheRussianInternet

company,Yandex,toincreasebrowsingsafety,butthesumofitseffortshasbeeninconsequential.

Figure3-FirefoxSafeBrowsingWarning.

Figure4-SafariSafeBrowsingWarning.

BothFirefoxandSafariuseGooglesSafeBrowsingAPI,andtheirblockingratesare,predictably,comparable.In

2012,NSStestingfoundtheseproductstobewithinonepercentagepointofeachother.GooglesChrome

browserwasnomoreeffectiveinitsuseoftheSafeBrowsingAPIthanApples.However,Chromedoesnotrely
https://www.nsslabs.com/reports/browser-security-comparative-analysis-socially-engineered-malware


9/18



upontheSafeBrowsingAPIalone;Googlehasaddeditsdownloadprotectiontechnologytoincreasethe

protectionofferedbyChromeagainstsociallyengineeredmalware.

Figure5-ChromeSafeBrowsingWarning.

Figure5depictsasafebrowsingalertinChrome.Therearetwoadditionalfile-basedblocksthatresultinChrome

providingsignificantlysuperiorprotectionoverFirefoxandSafari.

Figure6-ChromeMaliciousFileBlocking.

Figure7-ChromeApplicationReputationBlocking.

Incertainsituations,awebsitemaynotbeblocked;however,amaliciousfilemaybepresent.Inothercases,a

reputationsystemsuchasMicrosoftsAppRepisusedtodeterminewhetherafileisnotwellenoughknownto

establishtrust.Inthesecases,thedialogboxesinFigure6andFigure7willappear.

Figure8-InternetExplorerSmartScreenWarning.

InternetExplorersanswertoGooglesSafeBrowsingAPIincludesMicrosoftsSmartScreenaswellasURL

reputation.JustthesecomponentsaloneinInternetExplorermatchtheprotectionofChrome.


10/18



Figure9-InternetExplorerAppRepWarning.

Figure9showstheAppReptechnologythatMicrosofthasbuiltintoWindows8andintoInternetExplorer10.

WhenAppRepiscombinedwithMicrosoftsothertechnologies,InternetExplorerprovidesalmost100%

protectionagainstmaliciousdownloads.MicrosoftsAppRepisalsoavailableinInternetExplorer9runningon

Windows7.Theoretically,theunderlyingOSshouldbeirrelevantiftheprotectionsarewhollycontainedinIE10;

however,NSShasnottestedIE10onWindows7,andthereforeitcannotbeassumedthatthesamelevelof

protectionisofferedbythatcombination.

MalwareBlockPerformance

Eachbrowsersindividualblockperformancewastracked,andanoverallblockrateofallmalwarecollectedby

browserwasdeveloped.Abrowsersoverallblockrateisdefinedasthepercentageofsuccessfulblocksdividedbythetotalnumberoftestcases.Withtestsconductedevery6hours,aURLthatwasonlinefor48hourswillbe

tested8times.Abrowserblockingiton6(outofamaximum8)testrunswillachieveablockrateof75%.Figure

10showstheoverallblockperformanceofthefourbrowserstested.Asexpected,sinceFirefoxandSafariareusing

thesametechnology,theyareachievingsimilarblockrates.However,thelargedifferenceoftheaverageblock

ratebetweenbrowsersisnoteworthy,withresultsrangingfrom2%toalmost100%.

Figure10-MalwareBlockRateOverTime.

Toassesstheeffectivenessofdifferentblockingtechnologies,theNSStestharnessalsorecordsthemechanism

thatblockedaccesstoaURL.

Test Average = 41%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Test Average

Safari

Chrome( w/DownloadProtection)

Firefox

Internet Explorer 10 (w/App Rep)

Opera

Internet Explorer 10

Chrome


11/18



OfthethreebrowsersusingGooglesSafeBrowsingAPI,ChromeistheonlyonetoalsoutilizeGooglesmalicious

downloadtechnology;thistechnologyattemptstoblockmaliciousdownloadsfromsitesthatarenotblockedby

URLreputation.Figure10showstheblockperformanceoftheURLblockingcomponentandtheadditional

downloadblockcomponentusedbyGooglesChromeandInternetExplorer.TheURLblockingperformanceof

thethreeSafeBrowsingtechnologybrowserswasconsistentatabout10%.Googlesmaliciousdownload

protectionprovedtobeapproximatelyseventimesmoreeffectivethanURLblockingalone,increasingoverall

blockingperformanceby73.2%whencomparedtoURLblockingalone.Themaliciousdownloadtechnology

accountsforthemajorityoftheblockingperformanceofGoogleChrome.

ThecoreprotectiontechnologywithinInternetExplorerisSmartScreen,whichprovidesURL-basedprotection

fromattacksviaanintegrated,cloud-basedURL-reputationservice,aswellasknownmaliciousfileblocking.

MicrosoftalsousesAppReptogreatadvantagetoboostprotectionlevels.Onthefaceofit,Chromehasvirtually

identicalprotectiontoInternetExplorer10,atapproximately83%.However,Chromereliesonthearguablyless

reliableCAMPtechnologiestoachievethat.ForInternetExplorer,AppReppicksupthebulkoftheremaining17%

ofthemaliciousfilesthatwereencounteredinthetest,resultinginaprotectionlevelapproaching100%.

TheRealityOfApplicationReputation

BothGooglesandMicrosoftsapplicationreputationblockingtechnologiesarelikelytoyieldlesseffectivereal-

worldresultsthaninanautomatedtestenvironment.

Applicationreputationtechnologiesallowuntraineduserstooverridetheprotectionmechanismsusedtoprotect

againstmaliciousapplicationdownloads.Althoughtherearetimesthatthisisappropriate,thereisalsothedanger

thatsocialengineeringattackscandeceiveusersintobypassingthefileblockingandinstallingmalicioussoftware.

InNSStesting,asuccessfulblockwasalwaysassumedifaURLwaspresentedasathreat.

IfitwerearbitrarilyassumedthatuserswouldoverrideInternetExplorersapplicationreputationcomponent

about10%ofthetime,thenInternetExplorerwouldbeassumedtohavea90%blockrate,morethan10%higher

thanGooglesunmodifiedscore.Withoutempiricaltestingofuserbehavioroutsidethelab,itisnotknownhow

oftenapplicationreputationwarningsareignored.Itcannotbeassumedthattheusagerateswouldbeidentical

forChromeandInternetExplorerusers,sincetheexactwordingofthewarningmessage,aswellasthedifficultyin

overridingtheblock,willaffectabsoluterates.

Regardlessoftheshortcomingsofsystemsthatrelyuponuntraineduserstomakecorrectchoices,application

reputationisahighlysignificantandeffectiveprotectiontechnology.

GooglemarketingrecentlycollaboratedonaresearchpaperaboutGooglesContent-AgnosticMalwareProtection

(CAMP)technology.SeveralnewsorganizationsreportedthatGooglewasclaiminga99%malwaredetectionrate

forCAMP.However,closerexaminationofthepaperinquestionrevealstheactualclaimwasthatCAMPexhibits

accuracycloseto99%relativetoproprietaryVM-baseddynamicanalysis.Comparisontoaproprietaryvirtual

machineisonewayforamarketingdepartmenttoavoidhavingtopublishdisappointingresults,whileattempting

tomaketechnologylooknearlyperfect.Inrealworldtesting,thecombinationoftheSafeBrowsingAPIand

GooglesCAMPis83%effectiveatblockingmalware.Thisissignificantlylowerthanthe99%thatisclaimedby

Google;thedifferenceisexplainedbythefactthatNSScomparesempiricallyvalidatedblockratestowhatactually

evadesdetectionbythetechnologyundertest.
https://www.cs.jhu.edu/~moheeb/aburajab-ndss-13.pdfhttps://www.cs.jhu.edu/~moheeb/aburajab-ndss-13.pdfhttps://www.cs.jhu.edu/~moheeb/aburajab-ndss-13.pdfhttps://www.cs.jhu.edu/~moheeb/aburajab-ndss-13.pdfhttps://www.cs.jhu.edu/~moheeb/aburajab-ndss-13.pdfhttps://www.cs.jhu.edu/~moheeb/aburajab-ndss-13.pdf


12/18



TimeToBlockMaliciousSites

Whennewonlineattacksarecreatedanddeployed,itisvitalthattheyaredetectedasquicklyaspossible.The

followingresponsetimegraphdisplayshowlongeachofthebrowserstooktoblockathreat,oncethethreatwas

introducedintothetestcycle.Cumulativeprotectionratesarecalculatedeachdayuntilblocked.

Figure11-TimeToBlockMaliciousSites.

WhenGoogleannouncedtheacquisitionofVirusTotal,therewasspeculationthatitwouldbeusedtoenhance

Googlesdownloadprotection.Chromesperformancehasimprovedby13%sinceitsanalysisinthe2012NSSreport,BrowserSecurityComparativeAnalysis:SociallyEngineeredMalware.TheuseofVirusTotalinformation

mayalsoplayaroleinthesharpincreaseinprotectionbetweenzero-hourandday1;however,InternetExplorers

implementationofAppRepdemonstratesthatreputationisamoreeffectivebrowsersecuritytechnologythan

actualmalwaredetection.Thereareadd-onsforFirefoxandSafarithathelptoimprovesecuritybut,ingeneral,

theseprotectivetechnologiesareneitherused,norunderstoodbythenon-technicalusers.Fortheaverageuser,

InternetExplorer10orChromeisrecommended.UserschoosingSafari,Firefox,orOperawillwanttouseadd-ons

andothertechnologiestoaugmenttheirprotectionwherepossible.

0-hr 1d 2d 3d 4d 5d 6d 7d Total

Internet Explorer 10 - AppRep 98.14% 98.14% 99.07% 99.07% 99.07% 99.07% 99.07% 99.07% 99.07%

Internet Explorer 10 81.83% 81.83% 85.41% 86.21% 86.74% 87.14% 87.14% 87.27% 87.67%

Opera 12 0.80% 1.59% 1.72% 1.86% 1.86% 1.86% 1.86% 1.86% 1.86%

Chrome 25 (w/Download Protection) 48.54% 72.02% 73.08% 73.08% 73.21% 73.47% 73.61% 73.74% 74.14%

Firefox 19 7.82% 8.22% 8.22% 8.75% 8.75% 8.75% 8.75% 8.75% 8.75%

Safari 5 11.80% 13.66% 13.66% 13.93% 13.93% 13.93% 13.93% 13.93% 14.06%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Coverage
https://www.nsslabs.com/reports/browser-security-comparative-analysis-socially-engineered-malware


13/18



AppendixAMethodology

ClientHostDescriptionAlltestedbrowsersoftwarewasinstalledonidenticalvirtualmachineswiththefollowingspecifications:

MicrosoftWindows8Enterprise

4GBRAM

60GBharddrive

Figure12-VirtualMachineSpecifications.

Browsermachinesweretestedpriortothetestandduringthetest,toensureproperfunctionality.BrowsersweregivenfullaccesstotheInternettoenablethemtovisitlivesites.

TestedBrowsers

Thebrowsers,orproductsundertest,wereobtainedindependentlybyNSSLabs.Generally,availablesoftware

releaseswereusedinallcases.Eachproductwasupdatedtothemostcurrentversionavailableatthetimethat

testingbegan.Thefollowingisacurrentlistofthewebbrowsersthatweretested:

AppleSafariv5.1.7(7534.57.2)

GoogleChromeTMv25andv26

MicrosoftInternetExplorer10

MozillaFirefoxv19.0.2

OperaTMv12.14Build1738

Oncetestingbegan,theproductversionwasmonitored,andnewupdateswereappliedinarealisticpatching

methodology.Asanewversionofabrowserwasmadepubliclyavailableduringthetestingwindow,NSSwould

updatethetestharnessmachinesandrunbothversionsinparalleloverthecourseofatwo-weekphase-outofthe

priorversionofthebrowser.Thismaintainedtheintegrityofthevirtualinstancesthatwereundertest,whilestill

allowingforfreshinstancestostartwiththenewbrowserversion.ThistestrelieduponInternetaccessforthe

reputationsystemsandaccesstolivecontent.Generally,thereisaconfigurableseparationbetweensoftware

updatesanddatabaseorsignatureupdates,todrawanalogiesfromanti-virus,intrusionprevention,andgeneral

softwarepractices.

NetworkDescription

Thebrowsersweretestedfortheirabilitytoprotecttheclientinconnectedusecases.Thus,thetestsconsider

andanalyzetheeffectivenessofbrowserprotectioninNSSreal-worldliveInternettestingharness.

Thehostsystemhadonenetworkinterfacecard(NIC)andwasconnectedtothenetworkviaa1Gbswitchport.

Forthepurposesofthistest,NSSLabsutilized120desktopsystems,witheachsystemrunningawebbrowser.

ResultswererecordedintoaMySQLdatabase.


14/18



TestDuration

NSSbrowsertestwasperformedcontinuouslyfor28days.Throughoutthetest,newURLswereaddedasthey

werediscovered.

TestFrequency

Overthecourseofthetest,eachURLwasrunthroughthetestharnesseverysixhours.Regardlessofsuccessor

failure,NSScontinuedtoattempttodownloadamalwaresamplewiththewebbrowserforthedurationofthe

test.

Figure13-NSSLabsBrowserTestHarness.

SampleSetsforMalwareURLs

Freshnessofmalwaresitesisakeyattributeofthistypeoftest.Inordertoutilizethefreshest,most

representativeURLs,NSSreceivedabroadrangeofsamplesfromanumberofdifferentsources.

Sources

NSSoperatesitsownnetworkofspamtrapsandhoneypots.Thesee-mailaccountswithhigh-volumetrafficyield

thousandsofuniquee-mailsandURLsperday.Inaddition,NSSmaintainsrelationshipswithotherindependent

securityresearchers,networks,andsecuritycompaniesthatprovideaccesstoURLsandmaliciouscontent.Sample

setscontainmaliciousURLsdistributedviae-mail,instantmessaging,socialnetworks,andmaliciouswebsites.No

contentisusedfromthetestedparties.

MaliciousURLstargetingusersthroughouttheglobeareidentifiedandselectedforinclusioninthistest.Usersare

definedasindividualsresidingwithintheNorthAmerican,SouthAmerican,European,andAsia-Pacificregions,

includingArgentina,Australia,Austria,Brazil,Canada,China,France,Germany,India,Italy,Japan,Indonesia,

Mexico,NewZealand,Singapore,Spain,SouthKorea,Sweden,Thailand,theUnitedKingdom,theUnitedStatesof

America,andVietnam.ThisreportiscomprisedonlyofdatafromtheUnitedStatesofAmericasamples;future

paperswillincludeadditionaldatagathered.

Collect NewSuspicious MalicousSites from Sources

Pre-Filter, Validate,Prune & Archive

Sites

Distribute to TestClients

Test Clients Visit Site& Record Block/Allow

Results Collected &Archived


15/18



TheultimatedeterminantofwhetherornotamaliciousURLisincludedinthistestisitsparticipationinamalware

campaigntargetingusers.TheuseofamaliciousURLinacampaigntargetinganAsia-PacificoraNorthAmerican

userdoesnotnecessarilyprecludeitsuseinothercampaignstargetingusersfromotherregions.

Exploitscontainingmalwarepayloads(exploitsplusmalware),alsoknownasclickjackingordrive-bydownloads,areexcludedfromthetest.Everyeffortismadetoconsidersubmissionsthatreflectareal-world

distributionofmalware,categorically,geographically,andbyplatform.

Inaddition,NSSmaintainsacollectionofcleanURLs,includingsitesfromYahoo,Amazon,Microsoft,Google,

NSS,majorbanks,andothers.Periodically,cleanURLsarerunthroughthesystemtoverifythatthebrowsersare

notover-blocking.

CatalogURLs

NewsitesareaddedtotheURLconsiderationsetassoonaspossible.Thedateandtimeofeachsamples

introducedisnoted.Mostsourcesareimmediatelyinsertedautomatically,whilesomemethodsrequiremanual

handlingandcanbeprocessedinunder30minutes.AllitemsintheconsiderationsetarecatalogedwithauniqueNSSID,regardlessoftheirvalidity.Thisenablescorrecttrackingofeffectivenessofsamplesources.

ConfirmSamplePresenceofURLs

Timingiscriticalsincetheobjectiveistotesttheeffectivenessagainstthefreshestpossiblemalwaresites.Given

thenatureofthefeeds,andthevelocityofchange,itisnotpossibletovalidateeachsiteindepthbeforethetest,

sincethesitescouldquicklydisappear.Thus,eachofthetestitemsisgivenabriefreviewtoverifythatitispresent

andaccessibleontheliveInternet.

Inordertobeincludedintheexecutionset,URLsmustbeliveduringthetestiteration.Atthebeginningofeach

testcycle,theavailabilityoftheURLisconfirmedbyensuringthatthesitecanbereachedandisactive,suchthata

non-404webpageisreturned.

ThisvalidationoccurswithinminutesofreceivingthesamplesfromNSSsources.Note:Theseclassificationsare

furthervalidatedafterthetest,andURLsarereclassifiedand/orremovedaccordingly.

ArchivalOfActiveURLContent

TheactiveURLcontentisdownloadedandsavedtoanarchiveserverwithauniqueNSSIDnumber.Thisenables

NSStopreservetheURLcontentforcontrolandvalidationpurposes.

DynamicExecutionOfEachURL

AclientautomationutilityrequestseachoftheURLsdeemedpresent(baseduponresultsofthetestdescribed

inSection5.4)viaeachofthewebbrowsersinthetest.NSSrecordswhetherornotthemalwareisdownloaded

andifthedownloadattempttriggersawarningfromthebrowsersmalwareprotection.

ScoringAndRecordingTheResults

TheresultingresponseisrecordedaseitherAllowedorBlockedandWarned.

Success:NSSLabsdefinessuccessasawebbrowsersuccessfullypreventingmalwarefrombeingdownloadedand

correctlyissuingawarning.


16/18



Failure:NSSLabsdefinesfailureasawebbrowserfailingtopreventthemalwarefrombeingdownloadedand/or

failingtoissueawarning.

Pruning

Throughoutthetest,labengineersreviewandremovenon-conformingURLsandcontentfromthetestexecution

set.Forexample,aURLthatwasinitiallyclassifiedasmalware,butthathassincebeenreplacedwithageneric

splashpage,willberemovedfromthetest.

IfaURLsamplebecomesunavailablefordownloadduringthecourseofthetest,thesampleisremovedfromthe

testcollectionforthatiteration.NSSLabscontinuallyverifieseachsamplespresence(availabilityfordownload)

andadds/removeseachsamplefromthetestsetaccordingly.Shouldamalwaresamplethatisunavailableforone

testiterationbecomeavailableforasubsequentiteration,itwillbeaddedbackintothetestcollection.

Unavailablesamplesarenotincludedincalculationsofsuccessorfailurebyawebbrowser.

Post-TestValidation

Post-testvalidationenablesNSStoreclassifyandevenremovesamplesthatwereeithernotmaliciousornot

availablebeforetheteststarted.NSSusestheNormanAnalyzersandboxtopruneandvalidatethemalware.

Furthervalidationisperformedusingproprietarytools,systeminstrumentation,andcodeanalysisasneeded.


17/18



NSSLabsTestEnvironmentandMethodology

NSShascreatedacomplexLiveTestingenvironmentandmethodologytoassesstheprotectivecapabilitiesof

Internetbrowsersunderthemostreal-worldconditionspossible,whilealsomaintainingcontrolandverificationof

theprocedures.

Thepurposeofthestudywastodeterminehowwellcurrentwebbrowsersprotectusersfromthemostprevalent

malwarethreatsontheInternettoday.Animportantaspectinanytestofthisnatureisthetiming.Giventhe

aggressivemannerinwhichcriminalspropagateandmanipulatemaliciouswebsites,akeyobjectiveistoensure

thatthefreshestsitespossibleareincludedinthetest.

Aspartofthelivetestmethodology,web-basedthreatsarecontinuallycollectedfrommultiplesources,including

partnersandNSSownserversandhigh-interactionhoneynets.Potentialthreatsarescreenedalgorithmically

beforebeinginsertedintothetestqueue;threatsarecontinuallyinsertedandscreenedthroughoutthetest.

UniqueinthisprocedureisthatNSSvalidatesthesamplesbeforeandafterthetest.Actualtestingofthethreatsis

repeatedeverysixhoursandstartswithvalidationofthesitesexistenceandconformancetothetestdefinition.

Alltestsareexecutedinahighlycontrolledmanner,andresultsarerecordedandarchivedateachinterval.

Figure14-NSSLabsLiveIn-The-CloudTestFramework.


18/18


2013 NSS Labs Inc All rights reserved 18

2013NSSLabs,Inc.Allrightsreserved.Nopartofthispublicationmaybereproduced,photocopied,storedonaretrieval

system,ortransmittedwithouttheexpresswrittenconsentoftheauthors.

Pleasenotethataccesstooruseofthisreportisconditionedonthefollowing:

1.TheinformationinthisreportissubjecttochangebyNSSLabswithoutnotice.

2.TheinformationinthisreportisbelievedbyNSSLabstobeaccurateandreliableatthetimeofpublication,butisnot

guaranteed.Alluseofandrelianceonthisreportareatthereaderssolerisk.NSSLabsisnotliableorresponsibleforany

damages,losses,orexpensesarisingfromanyerrororomissioninthisreport.

3.NOWARRANTIES,EXPRESSORIMPLIEDAREGIVENBYNSSLABS.ALLIMPLIEDWARRANTIES,INCLUDINGIMPLIEDWARRANTIESOFMERCHANTABILITY,FITNESSFORAPARTICULARPURPOSE,ANDNON-INFRINGEMENTAREDISCLAIMEDAND

EXCLUDEDBYNSSLABS.INNOEVENTSHALLNSSLABSBELIABLEFORANYCONSEQUENTIAL,INCIDENTALORINDIRECT

DAMAGES,ORFORANYLOSSOFPROFIT,REVENUE,DATA,COMPUTERPROGRAMS,OROTHERASSETS,EVENIFADVISEDOFTHE

POSSIBILITYTHEREOF.

4.Thisreportdoesnotconstituteanendorsement,recommendation,orguaranteeofanyoftheproducts(hardwareor

software)testedorthehardwareandsoftwareusedintestingtheproducts.Thetestingdoesnotguaranteethatthereareno

errorsordefectsintheproductsorthattheproductswillmeetthereadersexpectations,requirements,needs,or

specifications,orthattheywilloperatewithoutinterruption.

5.Thisreportdoesnotimplyanyendorsement,sponsorship,affiliation,orverificationbyorwithanyorganizationsmentioned

inthisreport.

6.Alltrademarks,servicemarks,andtradenamesusedinthisreportarethetrademarks,servicemarks,andtradenamesof

theirrespectiveowners.

ContactInformation

NSSLabs,Inc.

206WildBasinRd.BuildingA,Suite200

Austin,TX78746USA

+1(512)961-5300

[email protected]

www.nsslabs.com

V.130513c

ThisreportwasproducedaspartofNSSLabsindependenttestinginformationservices.Leadingproductswere

testedatnocosttothevendor,andNSSLabsreceivednovendorfundingtoproducethisreport.

Documents

2013-04 CAR Browser Socially Engineered Malware 130513c