2010-05-102010-05-10 EICAR 2010EICAR 2010

AVerify AVerify

Towards verifiable anti-virus testingTowards verifiable anti-virus testing

2010-05-102010-05-10 EICAR 2010EICAR 2010

What’s this presentation about?What’s this presentation about?

Mostly AV testing, of courseMostly AV testing, of course

Just my personal point of view (not my Just my personal point of view (not my employer’s!)employer’s!)

The reasons behind AVerify, and the project The reasons behind AVerify, and the project goalsgoals

2010-05-102010-05-10 EICAR 2010EICAR 2010

A philosophical noteA philosophical note

The bad guys are helping each otherThe bad guys are helping each other– sharing/selling techniques, codes, …sharing/selling techniques, codes, …

Yet the AV ecosystem is fragmented, highly Yet the AV ecosystem is fragmented, highly competitivecompetitive– Non disclosure of information/samples/… gives an Non disclosure of information/samples/… gives an

edge (temporary)edge (temporary)– Cooperation exists on a small scaleCooperation exists on a small scale

2010-05-102010-05-10 EICAR 2010EICAR 2010

A philosophical noteA philosophical note Information sharing is a good thing, in generalInformation sharing is a good thing, in general

– Security through obscurity never worksSecurity through obscurity never works See: Mifare (RFID), A5/1 (GSM), …See: Mifare (RFID), A5/1 (GSM), …

– I started with ProView, thanks to McAfee :)I started with ProView, thanks to McAfee :)

Case in point: WEP, 2004Case in point: WEP, 2004– Wireless security started improving after tools Wireless security started improving after tools

became readily availablebecame readily available– Studying attacks techniques should be encouragedStudying attacks techniques should be encouraged

2010-05-102010-05-10 EICAR 2010EICAR 2010

(not my fault)(not my fault)

2010-05-102010-05-10 EICAR 2010EICAR 2010

<short rant><short rant> AV comparisons can be found everywhereAV comparisons can be found everywhere A few problems:A few problems:

– Does it still make sense to test with 16-bit DOS Does it still make sense to test with 16-bit DOS infector samples?infector samples? Nope. But look at that shiny 99.8% detection rate!Nope. But look at that shiny 99.8% detection rate!

– Results cannot be reproduced, no clear methodology Results cannot be reproduced, no clear methodology given. “Independent” ?given. “Independent” ?

– Scanning 10000+ files a realistic usage scenario?Scanning 10000+ files a realistic usage scenario?– Ad-hoc, black-box scoringAd-hoc, black-box scoring

““My AV must be better than yours, it’s got a 4-star GOLD My AV must be better than yours, it’s got a 4-star GOLD award!”award!”

2010-05-102010-05-10 EICAR 2010EICAR 2010

Let’s search “antivirus reviews” on Let’s search “antivirus reviews” on Google. The winner is…Google. The winner is…

2010-05-102010-05-10 EICAR 2010EICAR 2010

Apparently based on checking boxes:Apparently based on checking boxes:

A sound methodology!A sound methodology!

2010-05-102010-05-10 EICAR 2010EICAR 2010

““Race To Zero” (2008) ChallengeRace To Zero” (2008) Challenge

Defcon 16, Las Vegas, August 2008Defcon 16, Las Vegas, August 2008– Aim: modify known samples to bypass signature-Aim: modify known samples to bypass signature-

based detectionbased detection– Best time: nine samples obfuscated in 3 hoursBest time: nine samples obfuscated in 3 hours

Lots of media coverageLots of media coverage Precise technical details not shared with the Precise technical details not shared with the

publicpublic

2010-05-102010-05-10 EICAR 2010EICAR 2010

““Race To Zero” (2008) ChallengeRace To Zero” (2008) Challenge

Signature-based detection will fail eventuallySignature-based detection will fail eventually– Cf. Fred CohenCf. Fred Cohen

What good is this challenge?What good is this challenge?– Precise results and samples not availablePrecise results and samples not available– Not a set of single tests, more of an ad-hoc method Not a set of single tests, more of an ad-hoc method

(let’s nop/pack/obfuscate that sample until it evades (let’s nop/pack/obfuscate that sample until it evades the AV)the AV)

– Cannot be reproduce exactly, obfuscation technique Cannot be reproduce exactly, obfuscation technique not sharednot shared

Not worthless, but not earth-shattering eitherNot worthless, but not earth-shattering either

2010-05-102010-05-10 EICAR 2010EICAR 2010

The firewall “leak tests”The firewall “leak tests”

Guillaume Kaddouch, 2007Guillaume Kaddouch, 2007 Single tests, mostly network evasion & Single tests, mostly network evasion &

keyloggerskeyloggers– Disclosed exploit techniques & executablesDisclosed exploit techniques & executables– Published the results for each testPublished the results for each test

Initially, most vendors didn’t passInitially, most vendors didn’t pass– But they improved their productsBut they improved their products

Sadly, the test suite is no longer maintainedSadly, the test suite is no longer maintained– And source code is not accessibleAnd source code is not accessible

2010-05-102010-05-10 EICAR 2010EICAR 2010

““A study of anti-virus’ response to A study of anti-virus’ response to unknown threats (EICAR, 2009)”unknown threats (EICAR, 2009)”

N. Richaud and myselfN. Richaud and myself Twelve anti-virus products testedTwelve anti-virus products tested 21 single tests, oriented towards “proactive” 21 single tests, oriented towards “proactive”

(HIPS-like) detection(HIPS-like) detection Tests run by hand in Dec. 2008Tests run by hand in Dec. 2008

– Some tests did require admin rights, but not allSome tests did require admin rights, but not all Results published at EICAR 2009Results published at EICAR 2009

2010-05-102010-05-10 EICAR 2010EICAR 2010

EICAR 2009EICAR 2009

Low-level accessLow-level access2008 versions2008 versions MBR (modified MBR (modified

bootroot)bootroot)Device\PhysicalMemoryDevice\PhysicalMemory

avast!avast!

AVGAVG

AviraAvira DetectedDetected

BitDefenderBitDefender

ESETESET

F-SecureF-Secure

KasperskyKaspersky DetectedDetected

McAfeeMcAfee

NortonNorton

PandaPanda

SophosSophos

TrendMicroTrendMicro

2010-05-102010-05-10 EICAR 2010EICAR 2010

EICAR 2009EICAR 2009

Keyloggers:Keyloggers:2008 versions2008 versions WH_KEYBOARD_LLWH_KEYBOARD_LL GetRawInputDataGetRawInputData

avast!avast!

AVGAVG

AviraAvira

BitDefenderBitDefender

ESETESET

F-SecureF-Secure

KasperskyKaspersky DetectedDetected

McAfeeMcAfee

NortonNorton

PandaPanda

SophosSophos

TrendMicroTrendMicro DetectedDetected

2010-05-102010-05-10 EICAR 2010EICAR 2010

EICAR 2009EICAR 2009

Code injection:Code injection:2008 versions2008 versions CreateRemoteThreaCreateRemoteThrea

ddSetWindowsHookESetWindowsHookExx

QueueUserAPCQueueUserAPC

avast!avast!

AVGAVG

AviraAvira DetectedDetected

BitDefenderBitDefender

ESETESET

F-SecureF-Secure

KasperskyKaspersky DetectedDetected

McAfeeMcAfee

NortonNorton

PandaPanda

SophosSophos

TrendMicroTrendMicro DetectedDetected DetectedDetected

2010-05-102010-05-10 EICAR 2010EICAR 2010

EICAR 2009EICAR 2009

Lessons learned:Lessons learned:– 12 AV x 21 tests = a full week of work (non-stop! 12 AV x 21 tests = a full week of work (non-stop!

except the coffee breaks ;)except the coffee breaks ;)– Coding is fun, testing is very repetitive & boringCoding is fun, testing is very repetitive & boring– A partial view, limited to HIPS-based detectionA partial view, limited to HIPS-based detection– Tests were run on a specific configuration; Windows Tests were run on a specific configuration; Windows

Vista was ignoredVista was ignored

No real winner(s), even basic techniques were No real winner(s), even basic techniques were barely detectedbarely detected

2010-05-102010-05-10 EICAR 2010EICAR 2010

The iAWACS 2009 ChallengeThe iAWACS 2009 Challenge

Goal: to disable several anti-virus programs Goal: to disable several anti-virus programs without the user noticingwithout the user noticing

Windows XP SP3 with admin accountWindows XP SP3 with admin account

Results:Results:– Almost all AV could be disabled thanks to ring0 Almost all AV could be disabled thanks to ring0

accessaccess– Took a few minutes to a few hours (tops)Took a few minutes to a few hours (tops)

2010-05-102010-05-10 EICAR 2010EICAR 2010

The iAWACS 2009 ChallengeThe iAWACS 2009 Challenge

Lessons learned:Lessons learned:– Disabling is easy with admin rights ;)Disabling is easy with admin rights ;)

Wait, we knew that already.Wait, we knew that already. But should it be?But should it be? HIPS-like detection leads to the problem of false positives…HIPS-like detection leads to the problem of false positives…

– Many classic techniques still not blockedMany classic techniques still not blocked Including access to the PhysicalMemory deviceIncluding access to the PhysicalMemory device First published in Phrack, 2002First published in Phrack, 2002

– Once again, there was no clear winner or loserOnce again, there was no clear winner or loser

2010-05-102010-05-10 EICAR 2010EICAR 2010

The iAWACS 2010 ChallengeThe iAWACS 2010 Challenge Different rules:Different rules:

– Windows 7, user account (no admin rights at all)Windows 7, user account (no admin rights at all)– Every possible attack was considered fair gameEvery possible attack was considered fair game– Time limit: four hoursTime limit: four hours

Seven attacks proposed:Seven attacks proposed:– Many denial-of-service onesMany denial-of-service ones

One was .bat-powered ;)One was .bat-powered ;)– RansomwareRansomware

2010-05-102010-05-10 EICAR 2010EICAR 2010

The iAWACS 2010 ChallengeThe iAWACS 2010 Challenge Judging the attacks, a challenge in itselfJudging the attacks, a challenge in itself

– Some were quite sophisticatedSome were quite sophisticated– Some very basic but still effectiveSome very basic but still effective

Reiterates that signature-based detection not a Reiterates that signature-based detection not a silver bulletsilver bullet

Although a few good surprises:Although a few good surprises:– Example: KAV’s warning of an non-signed program Example: KAV’s warning of an non-signed program

launchedlaunched– Creation of a key in CurrentVersion\Run detected by Creation of a key in CurrentVersion\Run detected by

two other anti-virusestwo other anti-viruses

2010-05-102010-05-10 EICAR 2010EICAR 2010

The Guillermito caseThe Guillermito case 2001, Tegam vs. Guillermito (a French hacker)2001, Tegam vs. Guillermito (a French hacker)

– Tegam claimed to protect against all known Tegam claimed to protect against all known and and unknownunknown viruses viruses

– Guillermito then disclosed several flaws in Tegam’s Guillermito then disclosed several flaws in Tegam’s product, ViGuardproduct, ViGuard

Condemned (2006) to pay 15000 eurosCondemned (2006) to pay 15000 euros– Reverse-engineering cited as a reasonReverse-engineering cited as a reason

Tegam then went bankrupt…Tegam then went bankrupt… Preventing reverse-engineering only helps the Preventing reverse-engineering only helps the

bad guys!bad guys!

2010-05-102010-05-10 EICAR 2010EICAR 2010

Some difficulties of testingSome difficulties of testing Reverse-engineering: to be avoided?Reverse-engineering: to be avoided?

– Or just claim to have been very lucky ;)Or just claim to have been very lucky ;)– Reading the fine print, tedious but requiredReading the fine print, tedious but required

Evaluation versions can be hard to findEvaluation versions can be hard to find– Some AV companies make it easy, others notSome AV companies make it easy, others not– Could be considered a single test in itselfCould be considered a single test in itself

Finding new threats is a challenge in itselfFinding new threats is a challenge in itself– Even AV companies have trouble obtaining samplesEven AV companies have trouble obtaining samples

2010-05-102010-05-10 EICAR 2010EICAR 2010

More difficulties of testingMore difficulties of testing Can the test environment be considered Can the test environment be considered

realistic?realistic?– Many malware detect VMware and other vmMany malware detect VMware and other vm– Single tests may run for a few minutes, real users use Single tests may run for a few minutes, real users use

their computer for hourstheir computer for hours A better setup would require:A better setup would require:

– One physical machine per AV in parallel with identical One physical machine per AV in parallel with identical hardwarehardware

– ““dummy” robots mimicking usersdummy” robots mimicking users– Knowing which malware are the most commonKnowing which malware are the most common– Gaining access to new samples near release timeGaining access to new samples near release time

2010-05-102010-05-10 EICAR 2010EICAR 2010

Testing, but what?Testing, but what?

Anti-viruses are complex products with heaps Anti-viruses are complex products with heaps of featuresof features

1.1. What exactly are they supposed to be What exactly are they supposed to be protecting against?protecting against?– Malware, rootkits, spyware,Malware, rootkits, spyware,

<insert latest buzzword here>, …<insert latest buzzword here>, …

1.1. What are security mechanisms are What are security mechanisms are implemented? What should be tested exactly?implemented? What should be tested exactly?

More features not always a good thingMore features not always a good thing– Software less resilient, greater attack surface Software less resilient, greater attack surface

2010-05-102010-05-10 EICAR 2010EICAR 2010

Does CC Certification make sense?Does CC Certification make sense?

(CC = Common Criteria, an evaluation process)(CC = Common Criteria, an evaluation process) Pros:Pros:

– Full review of documentation and source codeFull review of documentation and source code– In theory, best known attacks are appliedIn theory, best known attacks are applied– Higher EAL levels offer (semi) formal proofHigher EAL levels offer (semi) formal proof

Cons:Cons:– TOE/ST is often reduced to save timeTOE/ST is often reduced to save time– Applies to a single version, but AV evolve fastApplies to a single version, but AV evolve fast– No common reference for the attacksNo common reference for the attacks depends on the evaluator’s competence depends on the evaluator’s competence

– Just another marketing gimmick?Just another marketing gimmick?

2010-05-102010-05-10 EICAR 2010EICAR 2010

A note on CSPN CertificationA note on CSPN Certification (CSPN = “Certification Sécurité Premier Niveau”)(CSPN = “Certification Sécurité Premier Niveau”) Similar to the Common Criteria, however:Similar to the Common Criteria, however:

– ““Single shot” evaluation, much shorter (1 month)Single shot” evaluation, much shorter (1 month)– Mostly focused on the attacks themselvesMostly focused on the attacks themselves

Cons:Cons:– Like CC, nothing is made publicLike CC, nothing is made public

(except the eventual certificate)(except the eventual certificate)– Perimeter is restricted as wellPerimeter is restricted as well– Cert. applies to a single versionCert. applies to a single version– Testing depends on the evaluatorTesting depends on the evaluator

2010-05-102010-05-10 EICAR 2010EICAR 2010

AVerifyAVerify Inspired by the EICAR anti-virus test fileInspired by the EICAR anti-virus test file

ButBut this project is independent from EICAR itself! this project is independent from EICAR itself! Follows the EICAR code of conductFollows the EICAR code of conduct

Provide a set of simple tests to other researchers Provide a set of simple tests to other researchers – w– with the full description & source codeith the full description & source code

This ensures:This ensures:– Independent reproducibilityIndependent reproducibility– Based on the original experimental desc.Based on the original experimental desc.– Reliably repetition of these experimentsReliably repetition of these experiments

Allows fact-based reviewing of AV programsAllows fact-based reviewing of AV programs– Instead of the hand-waving one often seesInstead of the hand-waving one often sees

2010-05-102010-05-10 EICAR 2010EICAR 2010

The basicsThe basics Define a common platformDefine a common platform

– Windows XP 32-bit still widely usedWindows XP 32-bit still widely used– Windows 7 64-bit gaining momentumWindows 7 64-bit gaining momentum– Which software to install? Firefox/IE8/Chrome…?Which software to install? Firefox/IE8/Chrome…?

Define a base privilege levelDefine a base privilege level– Admin or not admin? Or better, both.Admin or not admin? Or better, both.

Define common usage scenarios and attack Define common usage scenarios and attack vectorsvectors– Opening malicious links from email / IMOpening malicious links from email / IM– Running a malware run from an external drive:Running a malware run from an external drive:

USB key / CD-ROM / network driveUSB key / CD-ROM / network drive– Machine already infected, try do disinfectMachine already infected, try do disinfect

Evaluate the success of disinfection with a LiveCDEvaluate the success of disinfection with a LiveCD

2010-05-102010-05-10 EICAR 2010EICAR 2010

Planned tests #1: HIPS featuresPlanned tests #1: HIPS features Very similar to our 2008 testsVery similar to our 2008 tests Every test should only exercise one aspectEvery test should only exercise one aspect

– Old-school persistence through CurrentVersion\RunOld-school persistence through CurrentVersion\Run– WH_KEYBOARD Windows hookWH_KEYBOARD Windows hook– StartService driver loadingStartService driver loading– This includes firewall bypassing techniquesThis includes firewall bypassing techniques

Run each test identically on all AVRun each test identically on all AV– Define a passed/not passed checkDefine a passed/not passed check

Combination of tests to simulate real Combination of tests to simulate real threats, evaluate AV’s threshold levelthreats, evaluate AV’s threshold level

2010-05-102010-05-10 EICAR 2010EICAR 2010

Planned tests #2: resiliencePlanned tests #2: resilience Try to cause faults in the AV itselfTry to cause faults in the AV itself

– Fuzzing: file formats, IOCTLsFuzzing: file formats, IOCTLs Instrumenting AV scanners may be trickyInstrumenting AV scanners may be tricky

– Example: avast! looks simple, just call ashQuick.exe on each fileExample: avast! looks simple, just call ashQuick.exe on each file– But exception catching requires removing SSDT hooksBut exception catching requires removing SSDT hooks

– Checking and modifying ACLsChecking and modifying ACLs Files, pipes, handles, …Files, pipes, handles, …

– Attempts to disable/kill the anti-virusAttempts to disable/kill the anti-virus hosts file, connection blockinghosts file, connection blocking On-the-fly code patchingOn-the-fly code patching On-disk signature database corruptionOn-disk signature database corruption Network updates corruptionNetwork updates corruption

2010-05-102010-05-10 EICAR 2010EICAR 2010

Planned tests #3: real-world Planned tests #3: real-world malwaremalware

This is slightly harder:This is slightly harder:– Find a new threat after it is releasedFind a new threat after it is released– Repeatedly scan with each anti-virus to determine the Repeatedly scan with each anti-virus to determine the

reaction time of AV companiesreaction time of AV companies Or use any older but common threatOr use any older but common threat Install the threat, create a snapshotInstall the threat, create a snapshot

– Determine how it installs and stays persistentDetermine how it installs and stays persistent– Attempt to disinfect with each AVAttempt to disinfect with each AV– Use a LiveCD to check for successful disinfection on Use a LiveCD to check for successful disinfection on

diskdisk

2010-05-102010-05-10 EICAR 2010EICAR 2010

Planned tests #4: other stuffPlanned tests #4: other stuff Usage monitoring:Usage monitoring:

– Size on disk after installationSize on disk after installation– Size on disk after x months of updatingSize on disk after x months of updating– Amount of real memory usable in idleAmount of real memory usable in idle– CPU consumed when scanning a threatCPU consumed when scanning a threat– Bandwidth consumedBandwidth consumed

UI featuresUI features– Usefulness of error messagesUsefulness of error messages– Logging and access to log filesLogging and access to log files

SupportSupport

2010-05-102010-05-10 EICAR 2010EICAR 2010

An interesting attackAn interesting attack

““KHOBE – 8.0 earthquake for Windows desktop KHOBE – 8.0 earthquake for Windows desktop security software”security software”– They might be over-hyping it a little ;)They might be over-hyping it a little ;)

– Researchers provided full detailsResearchers provided full details– Could be re-implemented in AVerifyCould be re-implemented in AVerify

2010-05-102010-05-10 EICAR 2010EICAR 2010

Automation ftwAutomation ftw Mostly based on VMware scripting (VIX)Mostly based on VMware scripting (VIX)

– C/VB API bundled with VMware WorkstationC/VB API bundled with VMware Workstation– Also nicely documentedAlso nicely documented– Avoids errors due to manual testingAvoids errors due to manual testing– Lots of useful APIs:Lots of useful APIs:

VixVM_RevertToSnapshotVixVM_RevertToSnapshot VixVM_RunProgramInGuestVixVM_RunProgramInGuest

Cannot send raw keyboard/mouse inputCannot send raw keyboard/mouse input– However, VMware offers a VNC serverHowever, VMware offers a VNC server– VNC protocol open and easily scriptableVNC protocol open and easily scriptable

2010-05-102010-05-10 EICAR 2010EICAR 2010

Scripting a virtual machineScripting a virtual machine Checking for success or failure:Checking for success or failure:

– Depends on the test (eg. capturing keys, …)Depends on the test (eg. capturing keys, …)– VixVM_CopyFileFromGuestToHost VixVM_CopyFileFromGuestToHost – VixVM_CaptureScreenImageVixVM_CaptureScreenImage

VirtualBoxVirtualBox– VBoxManage is kinda like VIXVBoxManage is kinda like VIX

Start, stop, revert to snapshot…Start, stop, revert to snapshot…– No remote code execution & VNC server thoughNo remote code execution & VNC server though

Installing one might interfere with the testsInstalling one might interfere with the tests– One possible alternative: enable remote desktopOne possible alternative: enable remote desktop

Scripting by modification of an OSS rdp clientScripting by modification of an OSS rdp client

2010-05-102010-05-10 EICAR 2010EICAR 2010

On providing source codeOn providing source code

Is providing attack code illegal in France?Is providing attack code illegal in France?– A somewhat muddy subjectA somewhat muddy subject– See French ministry of interior vs. VUPENSee French ministry of interior vs. VUPEN

2009: Author of exploit condemned to pay 1000 E2009: Author of exploit condemned to pay 1000 E

I didn’t get jailed for publishing aircrackI didn’t get jailed for publishing aircrack– But that was in 2004/2005But that was in 2004/2005– It did include win32 binariesIt did include win32 binaries

AVerify will provide source codeAVerify will provide source code– But on a per-request basisBut on a per-request basis– No ready-to use binaries!No ready-to use binaries!

2010-05-102010-05-10 EICAR 2010EICAR 2010

On providing samplesOn providing samples

Distributing samples == distributing malware?Distributing samples == distributing malware?– Yes but…Yes but…

Very useful for other security researchersVery useful for other security researchers– vx.netlux.orgvx.netlux.org– offensivecomputingoffensivecomputing

Real-world malware samples from AVerify to be Real-world malware samples from AVerify to be made available on offensivecomputingmade available on offensivecomputing– Or any other site willing to host themOr any other site willing to host them

2010-05-102010-05-10 EICAR 2010EICAR 2010

A simple exampleA simple example Automation of the EICAR anti-virus test file Automation of the EICAR anti-virus test file

through VMware VIXthrough VMware VIX

Current problem: detecting that it workedCurrent problem: detecting that it worked– Screenshots not reliableScreenshots not reliable– AV logs?AV logs?

2010-05-102010-05-10 EICAR 2010EICAR 2010

The future of AVerify?The future of AVerify? Lots of ideas, but not much code yet!Lots of ideas, but not much code yet!

– No demo quite yet, sorryNo demo quite yet, sorry– As always, time is not on our sideAs always, time is not on our side– Right now, it’s a one-person projectRight now, it’s a one-person project

First initial target: November 2010First initial target: November 2010– Source code and initial results made publicSource code and initial results made public– Hopefully, together with a submission for the next Hopefully, together with a submission for the next

EICAR conference :)EICAR conference :) Anyone can contribute! You can too.Anyone can contribute! You can too. Please check Please check averify.orgaverify.org in a few months in a few months

2010-05-102010-05-10 EICAR 2010EICAR 2010

A final noteA final noteDuring the installation of a major AV (2010):During the installation of a major AV (2010):

Scaring your users is not the solution. Educating them is.Scaring your users is not the solution. Educating them is.

2010-05-102010-05-10 EICAR 2010EICAR 2010

Q&A Q&A

Thank you for your attention!Thank you for your attention!