Embed Size (px)
Citation preview
Building a Test Environment for Android Anti-Malware Tests
Hendrik Pilz Director Technical Lab / Mobile Security
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Agenda
• Android Malware Landscape • Real Devices or Emulator? • Preparation • Test Scenarios • Automation • Problems
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Android Malware Landscape
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
0
10.000
20.000
30.000
40.000
50.000
60.000
January 2011 August 2012
Android Malware Collection Growth
Total Number of Samples
Android Malware Landscape
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Backdoor Monitor Other
Trojan
Trojan-SMS
Trojan-Spy
Malware Categories August 2012
Android Malware Landscape
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
FakeInst
Opfake
Other
GinMaster
BaseBrid KungFu FakeDoc Kmin Plangton
Malware Families August 2012
Real Devices or Emulator Device
• Real user experience • App activation via
SMS • Real life environment
Emulator • Cost efficient,
scalable • Root privileges • Multiple API versions
and hardware configurations
• Snapshots
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Preparation
System Requirements: • PC which is capable to run the Android SDK • Android device, prepaid SIM • USB cable • WiFi-Internet for Android device
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Preparation
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
WWW
PC with Android SDK WWW
USB
Android device
Preparation
• Install Android SDK from developer.android.com/sdk
• Choose Malware Samples according to AMTSO Guidelines
• Install Anti-Malware on test device, update signatures
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Preparation
• Connect device to PC • Create device backup
$: adb backup –f <file> -apk –shared –all –system $: adb restore <file>
• Take Screenshots $: android-sdk/tools/ddms
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – On-Demand Scan
• Copy samples to device $: adb push <source> /sdcard/samples
• Perform on-demand scan, delete all malicious files
• Save remaining files $: adb pull /sdcard/samples <dest>
• Save scan reports, if possible
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – On-Demand Scan
Alternative to adb push/pull: Copy files over WiFi from/to network
share (e.g. with Astro File Manager)
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – On-Demand Scan
Some Anti-Malware apps scan installed apps only!
An On-Access Test is always required to determine accurate detection rates!
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – On-Access
• Install each sample one-by-one $: adb install <apk-file>
• Check warnings and messages of Mobile Security
• Remove or uninstall sample $: adb uninstall <package-name>
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – On-Access
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – On-Access
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – On-Access
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – False Positives
• Combination of OA & OD • Install clean apps via ADB • Run an OD-scan afterwards • Note all warnings and detections
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – False Positives
• Be aware of greyware: – Ad supported apps – Privacy risks
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – Performance
• Install clean apps from Google Play – We can‘t use ADB here, because we can‘t disable USB charging
• Monitor CPU-usage and battery discharge
• Repeat several times
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – Performance
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
70,00%
80,00%
com
.ado
be.p
smob
ile
com
.ado
be.r
eade
r
com
.alp
hons
o.pu
lse
com
.am
azon
.kin
dle
com
.cre
ativ
emob
ile.d
ragr
…
com
.dat
aviz
.doc
stog
o
com
.dro
pbox
.and
roid
com
.eba
y.m
obile
com
.est
rong
s.an
droi
d.po
p
com
.eve
rnot
e
com
.fac
eboo
k.ka
tana
com
.gam
elof
t.an
droi
d.A
N…
com
.goo
gle.
andr
oid.
apps
.…
com
.inst
agra
m.a
ndro
id
com
.rov
io.a
ngry
bird
sspa
c…
com
.sha
zam
.and
roid
com
.sky
pe.r
aide
r
com
.wet
ter.
andr
oidc
lient
com
.zin
io.m
obile
.and
roid
.…
org.
moz
illa.
fire
fox
com
.wha
tsap
p
com
.dev
uni.f
lash
light
de.s
child
bach
.oef
fi
de.a
maz
on.m
Sho
p.an
droi
d
logo
s.qu
iz.c
ompa
nies
.gam
e
de.r
adio
.and
roid
org.
gold
ennu
gget
apps
.si…
mob
i.mge
ek.T
unny
Bro
wse
r
com
.goo
gle.
zxin
g.cl
ient
.a…
com
.hal
fbri
ck.f
ruit
ninj
afre
e
com
.pic
sart
.stu
dio
com
.mpi
soft
.doo
rs
com
.am
azon
.mp3
com
.met
ago.
astr
o
CPU usage
Product A
Product B
Product C
Product D
Product E
Test Scenarios – Performance
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
0
0,05
0,1
0,15
0,2
0,25
0,3
0,35
0,4
Product A Product B Product C Product D Product E
Discharge rate in % per minute
0
50
100
150
200
250
300
350
400
Product A Product B Product C Product D Product E
Estimated battery life in minutes
Test Scenarios – Performance
• Measure impact on real-world usage – Loading websites – Sending/receiving messages – Opening apps – Playing media files – …
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – Others?
• Other functions are not common among all AV/mobile security products: – Anti-Theft – Backup, Encryption – Spam, Phishing – …
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Test Scenarios – Others?
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
0 5
10 15
20 25 30 35 40 45
Number of Products with this specific Feature
Automation
• ADB-CLI • ddmlib.jar (included in SDK)
– High Level API to control ADB
• Robotium <http://code.google.com/p/robotium/>
– GUI automation of Android apps
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Problems
• Not all apps support SD card scan • No proper reporting • No export of report files
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
Thank You!
Questions?
Building a Test Environment for Android Anti-Malware Tests www.av-test.org
