Detecting User Activities using the Accelerometeron Android Smartphones

Sauvik Das— Georgia Institute of TechnologyLaToya Green— University of Houston

Beatrice Perez— University of Puerto Rico, MayaguezMichael Murphy— FranklinW. Olin College of Engineering

Supervisor: Dr.Adrian Perrig

July 30, 2010

Contents

1 Abstract 1

2 Introduction 1

3 Background Information 2

4 Related Work 3

5 Methodology 35.1 Background . . . . . . . . . . 35.2 Data Acquisition . . . . . . . 45.3 Signal Processing . . . . . . . 5

5.3.1 Preprocessing . . . . . 55.3.2 Noise Reduction . . . 55.3.3 Linearization . . . . . 65.3.4 Smoothing . . . . . . . 6

5.4 Feature Extraction . . . . . . . 75.5 Classification . . . . . . . . . . 8

6 Results 9

7 Conclusions and Future Work 9

8 References 10

1 Abstract

The purpose of this study is to identify whethersmartphones pose a security threat to the user.The accelerometer and other sensors within thedevice can be used without the users consent.Our intent in this is to show that the accelerom-eter can be used to obtain sensitive informationabout the user. Using the magnitude of the ac-celerometer data we found that we could iden-tify general activities preformed by the user, and

even have the phone learn new activities. Mul-tiple approaches were implemented to attempt tofind the best results. With individual calibra-tion we obtained accuracy of 93%, which couldbe improved with future work.

2 Introduction

Smartphone security is becoming increas-ingly important as consumers rely moreand more on their smartphones to storepersonal information. With the devel-opment of smartphone operating systems,such as Apple’s iOs, Google’s Android andBlakcberry’s RIM operating systems on therise, it is imperative that security measuresare in place to protect the privacy of the user.This is especially true for Google’s Androidplatform, being that it has an open develop-ment environment.

What is Android?The Android platform is an open platform

for mobile devices consisting of an operat-ing system, applications and middleware1.Android gives users the opportunity tobuild and publish their own applications byproviding an open development environ-ment. Android treats all applications (na-tive and third-party) as equals2. Therefore,having such an open development environ-ment requires security measures to be takenin order to protect the integrity of the An-droid platform and the privacy of its users.

Android Security and PermissionsThe Android Platform takes advantage

of several mechanisms designed to protectthe privacy and security of Android users,

1

as well as the operating system. Thesemethods include the Android security ar-chitecture, application certificates, and ap-plication permissions. The purpose of theAndroid security architecture is to preventapplications from being able to automati-cally perform operations that could jeopar-dize the security of other applications, theoperating system or the user. Certificatesare used to identify the author of a spe-cific application and to prevent users frominstalling fraudulent software on their de-vices. Android will not install an applica-tion that has not been signed with a certifi-cate. Therefore, the origin of all publishedapplications is traceable.

Android security permissions are han-dled by the AndroidManifest.xml filepresent within all application files. Whena user downloads an application onto theirdevice, they are automatically notified ofthe permissions the application has accessto. This informs the user of what type ofinformation an application is able to collectfrom the device as well as the hardware theapplication can use.

The AndroidManifest.xml file takes careof both software and hardware permissions.But while Android does require permis-sions for the use of hardware devices suchas the camera and vibrator, it does not re-quire permissions to be set in place for theuse of any available sensors, including theaccelerometer, orientation, and gyroscopesensors3. But we have found that these sen-sors, when used alongside other tools suchas the internet and GPS, can also pose as se-curity threat to the user. And it is possible

for an application to collect user informa-tion from these sensors without the user’sknowledge.

AccelerometersThe accelerometer in Android phones

measures the acceleration of the device onthe x (lateral), y (longitudinal), and z (ver-tical) axes. Accelerometers can be used todetect movement and the rate of change ofthe speed of movement. As stated above,the use of accelerometers in Android ap-plications does not require the applicationto have permission to use it. Therefore, itis possible for an application to collect auser’s accelerometer data without the user’sknowledge. With accelerometer data andthe use of a server to collect the informa-tion, it is a fairly simple task for someone togain a user’s personal information, their lo-cation, or to figure out what a user is doingor typing.

3 Background Information

Accelerometers have been used for a varietyof uses throughout the world today, frommedical to research, from car performanceto robotics. However, with the advent ofthe iPhone and Android, accelerometers aremuch more commonplace in the world oftoday. The most commonly used accelerom-eter within our phones is the LIS311DLH, at3x3x1 millimeters, it is a tiny, low power,high performance linear accelerometer. Itsenses the forces of acceleration in the X Yand Z planes to a precision of six decimalplaces. In order to investigate the possible

2

link between cyber security and physical se-curity, we set out to see how these sensors insmart phones, specifically this accelerome-ter could be used. In our research we wishto see if this device can turn what seems tobe a harmless application on the AndroidMarketplace into a potential spying device,able to detect what actions the user is takingat the moment.

4 Related Work

Many groups have developed sensorrecording and processing devices, such asthe eWatch, a wearable sensor that is in-tended for use of monitoring elderly andsick, identify non-responsiveness and keeptaps on their position5. A Japanese com-pany, KDDI, has developed something sim-ilar to this, a system to keep track of em-ployee’s actions throughout the day, andsend it back to a central mother server.This device can detect such activities suchas walking, climbing stairs, or even clean-ing. However, little thought from the com-pany has gone into the potential securityviolations6. Many other universities haveexplored activity recognition using videoinstead of accelerometer data7. While thisis not exactly related, it also explores theidea of using a system we don’t have on ourminds as being a security threat, in fact, theopposite, a security tool - cameras - as some-thing to invade privacy. Indeed, some uni-versities also attempted activity recognitionusing cell phones as well8. Other systemshave been suggested by the world today,

but not implemented, such as a path track-ing device using only accelerometer data,or the use of accelerometer to transmit dataother than the data of acceleration. Suchphones could then be used as tracking de-vices for the creators, able to see their user’sdaily lives through accelerometer data.

Current location tracking systems offerhigh accuracy using GPS and other suchmethods; however, those require consentfrom the user. In this paper we attempt toshow such things as location and activitycan be detected and recorded without theknowledge of the user, using simply the ac-celerometer within their phone.

5 Methodology

5.1 Background

The data received from the accelerometerwas in the form of a three-valued vectorof floating point numbers that representedthe individual accelerations of the smart-phone device in the X, Y, and Z axes sub-tracted by the gravity vector G. The ac-celeration values were recorded in metersper second squared( m

s2 ). Thus, layed flaton a level surface, the expected reading ofthe accelerometer would be approximately[0,0,-9.81]. The process of converting theseacceleration vectors into known activitiesoccured in four broad phases: data acqui-sition,signal processing, feature extraction andclassification.

During the preprocessing stage, each sam-pled acceleration vector was combined into

3

a single magnitude; in the noise reductionstage, the received signal was linearized andsmoothed to reduce noise; in the feature ex-traction stage, features were extracted foreach sample window of a predeterminednumber of 512 samples; in the classificationstage, unknown patterns of data were clas-sified via feature comparison of known pat-terns of data, using the Nearest Neighborand Naive Bayes classifiers. Two versions ofthe classification stage were implemented,one online–which was done on the fly as datawas being received–and one offline–whichwas done on a server after all relevant datahad been collected.

Prior to implementation, we had decidedon attempting to classify the following fiveactivities:

• Phone Detached: The user is not cur-rently holding the phone

• Idle: The user is holding the phone inan idle state

• Walking: The user is walking

• Running: The user is running

• Jumping: The user is jumping

. Having achieved significant accuracyin classifying the aforementioned activities,two additional activities were added to thelist:

• Descending Stairs: The user is de-scending stairs

• Climbing Stairs: The user is climbingstairs

Figure 1: The data acquisition application runningon the Motorola Droid

. Following reasonable success with theseadditions, users would be allowed to entercustomized gesture patterns.

5.2 Data Acquisition

The device used was the Motorola Droid.To acquire the accelerometer data, we cre-ated an application for the Droid using theAndroid SDK. Training data was acquiredby having users perform an instructed task(e.g., walking, running, jumping) while thedata was either being exported to a server orwritten to the Droid’s SD card. The applica-tion also graphed the sampled accelerationdata, allowing for a real-time preview of thedata. Figure 1 shows a screenshot of theapplication running on the Droid.

4

Figure 2: Raw accelerometer data in the X,Y and Z.Horizontal axis units are in milliseconds, and verticalaxis units are in m

s2

5.3 Signal Processing

5.3.1 Preprocessing

The first step taken was to merge the three-dimensional input signal into one accelera-tion magnitude. The original signal can beseen in Figure 2. We found the magnitudeof the acceleration vector by taking the eu-clidean magnitude of the three individualvalues, that is:

a =√

x2 + y2 + z2 (1)

The signal run through equation (1) can beseen in Figure 3. The merger was doneto simplify feature extraction, because theoverall theme in the acceleration patternwas deemed to be sufficient for the recog-nition of our original set of activities, noneof which required distinction of directionalaccelerations. However, features from indi-vidual acceleration axes may be importantin determining activities where directional

Figure 3: Merged acceleration magnitude data.Horizontal axis units are in milliseconds, and ver-tical axis units are in m

s2

information is relevant, such as differentiat-ing between martial arts motions.

5.3.2 Noise Reduction

There were two primary sources of noise inthe received signal. The first was irregularsampling rates and the second was the noiseinherent in discrete physical sampling of acontinous function.

The accelerometer was likely sampled ir-regularly because of the Android frame-work’s implementation of the samplingmechanism. The Android API offers fourabstract sampling rates for its accelerome-ter sensor (listed from fastest to slowest):Fastest, Game, Normal, and UI. The actualphysical sampling capabilities of the ac-clerometers likely vary from device to de-vice, so these sampling rate options are usedmore as guidelines than as actual physicalsampling rates. Another reason is becauseof how application on the Android frame-

5

work receives acceleration readings. TheAndroid API only allows the acquisitionof an acceleration sample on an ”onSensor-Changed()” event, which fires whenever theAndroid OS determines that the accelerom-eter values have changed. As such, an accel-eration sampling application cannot forcea reading of the accelerometer at predeter-mined intervals. Combined with the factthat Android OS likely has varying loads ofactivity from moment to moment, the ”on-SensorChanged()” is not scheduled firmly,resulting in irregularly sampled values. Toregularize sampled values, the holes in thesignal were interpolated via a process calleddata linearization.

Outside of irregular sampling rates, addi-tional noise was periodically introduced tothe signal just from the nature of the activitypatterns performed. For example, a slightchange in the orientation of the phone isquite common even in an Idle position, butcan result in an anomalous peak in the re-sultant signal. This type of noise was han-dled by running the signal through a 5-pointsmoothing algorithm.

5.3.3 Linearization

Data Linearization was the process usedto handle the irregular sampling of the re-ceived acceleration signal. The process in-volved choosing a desired regular samplingrate, and interpolating all the holes in thedata via linear interpolation. The linearizedvalue would be calculated by finding theclosest sampled data point before the de-sired sampling time, and the closest sam-

pled data point after the desired samplingtime, and then interpolating what the valueof the desired sample time would have been.One significant problem in the linearizationprocess was to determine an ideal desiredsampling rate.

In order to ensure that not too much datawas calculated via interpolation - whichcould have resulted in a false recreation ofthe original signal - a program was createdwhich determined the mean and minimumdifferences in time (in milliseconds) of sub-sequent readings. The arithmetic averageof these two values was then calculated andanalyzed for multiple datasets in order toprovide us with a general guideline of whatmight be an ideal sampling rate. It was de-termined that approximately 1 sample ev-ery 8 milliseconds, or 125 samples per sec-ond was a good sampling rate for the data.

A signal before and after data lineariza-tion can be seen in Figures 4 and 5, respec-tively. Notice that while the X axis has al-most doubled in length, the key features ofthe signal are exactly preserved. This is ex-pected because data linearization is simplymeant to fill in the holes of a sampled signal,and should not significantly alter the signalitself. The X domain is doubled because weused the arithmetic average of the minimumand mean sampling time differences, whichin this case corresponded to about half themean difference.

5.3.4 Smoothing

The linearized signal was then run througha 5-point smoothing algorithm to reduce

6

Figure 4: An acceleration signal before data lin-earization. X-axis is the sample number, Y-axis is inms2

any additional noise. This additional noisecould have come from any of a number ofsources (e.g., idle orientation shifts, screentaps, bumps on the road while walking).The 5-point smoothing algorithm calculateseach point to be the average of its fournearest neighbors, the two nearest beforeand the two nearest after. The 5-pointsmooth was chosen so that spikes with anobservable, steady progression would bepreserved while anomalous, sudden spikeswould be eliminated.

We tried running the signal through a 5-point smooth between one and five itera-tions, with three iterations yielding the mostaccuracy. This is because over-smoothingcaused subtle distinctions between similar,but different signals - such as the Idle versusPhone Detached signal - to disappear.

Figure 6 shows the signal shown in Figure5 after smoothing. Notice how in Figure 5,the signal is very jaggedy and discrete - akey indication of a noisy signal. In Figure 6,the signal is more smooth and continous.

Figure 5: An acceleration signal after data lin-earization. X-axis is the sample number, Y-axis is inms2 . Notice that the X-domain is significantly longerin this image, though the shape of the curve is ap-proximately the same.

Revisiting data linearization, Figure 7shows the same signal smoothed, but notfirst linearized. Notice how the signals inFigures 5 and 6 are significantly different.Thus, it was determined that after smooth-ing, the differences between linearized andnon-linearized signals are noticable and sig-nificant.

5.4 Feature Extraction

Feature Extraction was the process used toextract the key elements from a processedsignal which made the signal distinct. Forexample, how could one represent the sig-nals in Figure 8 such that a classification al-gorithm could later distinguish between thethree signals? As humans, it is simple forus to make the distinction visuall; but ma-chine algorithms require concrete statistics- features.

The following features were chosen based

7

Figure 6: Smoothed acceleration signal. Comparethis signal to the signal in Figure 5.

Figure 7: Smoothed, but not linearized accelera-tion signal. Notice how this signal is significantlydifferent from the one in Figure 6.

on our own discussion and the recommen-dations of previous accelerometer based ac-tivity recognition research:

1. Fundamental Frequencies: The funda-mental frequencies of the signal. Thesewere found from the Fourier Transfor-mation of the signal over the samplewindow. The final value was the aver-age of the three dominant frequenciesof the signal.

2. Average Acceleration: The arithmetic

Figure 8: A thirty second-spread of Walking, Run-ning and Jumping accleration signals.

average of the acceleration values in thesample window.

3. Maximum Amplitude: The maximumvalue of the signal in the window.

4. Minimum Amplitude: The minimumvalue of the signal in the window.

Note that a sample window of 512 sam-ples was used. 512 samples were chosenbased on the recommendation of prior pa-pers, because it was deemed sufficient foractivity recognition and because 512 is a 2n

number, which is ideal for the Fast FourierTransformation algorithm.

5.5 Classification

During the classification stage, we labeledunknown patterns of data based on theknowledge acquired from known patternsof data. For the online implementationof the Activity Recognition application, weimplemented a 1-Nearest Neighbor classi-fier on the Motorola Droid. The 1-Nearest

8

Neighbor algorithm is a simple algorithmwhich matched unknown patterns of datawith a known pattern of data based on theeuclidean distance between the two featurevectors. The known pattern of data withthe lowest euclidean distance from the un-known pattern of data was determined tobe the best match. For the offline classifi-cation, we used the Naive Bayes classifierprovided by the Weka 3 data mining soft-ware. The Naive Bayes classifier works onthe premise that the absence of a single fea-ture does not necessarily disqualify a knownpattern of data from being the correct match.Despite this assumption, Naive Bayes hasbeen shown to be a very effective classifier.Unfortunately, Weka 3 could not be run onthe Droid because of Java Mobile Edition’slimited functionality.

6 Results

We found that using the nearest neighborclassifier the program could predict patternsor activities with 93% accuracy after it hadbeen calibrated for a particular user. Wefound that for people of the same height thephone could predict activities even if thesubject had not gone through the trainingprocess. A different classifier, for exampleNaives Bayes could complete the predic-tions for a bigger population after a min-imum amount of samples had been col-lected. Individual gestures were recognizedwith as much accuracy as activities but onceagain machine learning had to come first.

One of the limitations of our program is

that for the phone to make accurate predic-tions, the handset would have to be in thesame place as it was when it was in the train-ing mode for example if during the train-ing mode the phone was being held in thehand, then for more accurate predictions,the phone would have to be in the hand ofthe person that was using it. Having it beon the pocket or the purse could make theclassifier label an activity incorrectly.

7 Conclusions and FutureWork

The research shows that sensors in the an-droid platform could constitute a violationof privacy for the users. Still to be doneon the activity recognition implementationwould be to use a better classifier that wouldpredict for a greater amount of people with-out having to go through the training pro-cess. The accelerometer proved to be a use-ful tool in identifying activities based onyour phone’s movements and other uses ofthe accelerometer could potentially includeits use in detecting key presses and even textwithout the knowledge of the user. Anotherinteresting application would be the trans-fer of information between phones based onthe matching acceleration values betweenthem. Overall we found that the safeguardsin place are not enough and have flaws thatmust be corrected in future updates to theplatform.

9

8 References

[1] ”Android FAQ - What Is Google An-droid?” Android News - Android GooglePhone Forums. Web. 20 July 2010. 〈http ://www.talkandroid.com/google − android −f aq/〉.

[2] Meier, Reto. Professional Android2 Application Development. Indianapolis,IN: Wiley, 2010. Print.

[3] ”Security and Permissions.” AndroidDevelopers. Web. 20 July 2010. 〈http ://developer.android.com/guide/topics/security/security.html〉.

[4] ”LIS331DLH MEMS digital out-put motion sensor ultra low-powerhigh performance 3-axes ”nano” ac-celerometer.” Web, 28 July 2010. 〈http ://www.st.com/stonline/products/literature/ds/15094.pd f 〉

[5] Uwe Maurer, Anthony Rowe, AsimSmailagic, and Daniel Siewiorek. Locationand Activity Recognition Using eWatch: AWearable Sensor Platform. 2008. Print.

[6] ”Mobile that allows bosses tosnoop on staff developed ” BBC NEWS- Technology. Web. 27 July 2010. 〈http ://news.bbc.co.uk/2/hi/technology/8559683.stm/〉.

[7] Weiyao Lin, Ming-Ting Sun, RadhaPoovandran, and Zhengyou Zhang.”Human Activity Recognition for VideoSurveillance.” Web 29 July 2010 〈http ://www.ee.washington.edu/research/nsl/papers/iscas−08.pd f 〉.

[8] Nishkam Ravi and Nikhil Dandekarand Preetham Mysore and Michael L.Littman. Activity Recognition from Ac-celerometer Data. Department of ComputerScience Rutgers University

[9] Bao, L., and Intille, S. S. 2004. Activ-

ity recognition from userannotated acceler-ation data. In Proceceedings of the 2nd In-ternational Conference on Pervasive Com-puting.

10