ECE 494 Capstone Design Final Design Presentation Smartphone Based Human Behavioral Analysis Andrew...
If you can't read please download the document
ECE 494 Capstone Design Final Design Presentation Smartphone Based Human Behavioral Analysis Andrew Jackson Michael Armstrong Robbie Rosati Andy McWilliams
ECE 494 Capstone Design Final Design Presentation Smartphone
Based Human Behavioral Analysis Andrew Jackson Michael Armstrong
Robbie Rosati Andy McWilliams Aaron Stewart April 18, 2014 Advisor:
Dr. Fei Hu
Slide 2
Outline 4/18/2014Smartphone Based Human Behavioral Analysis2
Project Recap & Goal System Diagram Subsystem Breakdown &
Team Roles Sensor Data Extraction & Analysis App Development
& Activity Recognition DTW Algorithm Environment Sensors NN
Algorithm Raw Memory Extraction & Analysis HMM Algorithm
Administration
Slide 3
Project Recap & Goal 4/18/2014Smartphone Based Human
Behavioral Analysis3 Create a system for tracking and detecting a
users behavioral patterns though the phones sensors and internal
memory logs. Can be used in multiple areas: Healthcare Activity
Monitoring Homeland Security
Slide 4
System Diagram 4/18/20144Smartphone Based Human Behavioral
Analysis
Slide 5
Behavior Recognition Phone Data Sensor Data Extraction &
Analysis Accelerometer Light Sound Proximity GPS Raw Memory
Extraction & Analysis dc3dd Call Logs Text Messages Machine
Learning Algorithms Hidden Markov Model Neural Networks Dynamic
Time Warping Subsystem Breakdown & Team Roles
4/18/20145Smartphone Based Human Behavioral Analysis Andrew Jackson
Android Sensor App DTW Algorithm Aaron Stewart Raw Memory Analysis
Team Leader Robbie Rosati Machine Learning Algorithms Raw Memory
Extraction Michael Armstrong Raw Memory Extraction HMM Algorithm
Andy McWilliams HMM Algorithm Raw Memory Analysis
Slide 6
Sensor Data Extraction & Analysis Aaron Stewart
Slide 7
Android, iOS, or Windows Phone We chose to go with Android. No
development license fees Majority of market Many sensors available
Open file system Previous Android programming experience Cheaper
phone prices 4/18/20147Smartphone Based Human Behavioral
Analysis
Slide 8
Samsung Galaxy S4 Mini 4/18/2014Smartphone Based Human
Behavioral Analysis8 New phone with many sensors Fairly inexpensive
for an unlocked phone Smaller size great for testing in
pockets
Slide 9
Accelerometer LightSoundProximityGPS 4/18/2014 Smartphone Based
Human Behavioral Analysis 9 Sensors
Slide 10
Sensor Types 4/18/2014Smartphone Based Human Behavioral
Analysis10 Accelerometer Measures acceleration with gravity value
Reports data in 3 values, one for each axis Example: Could tell if
user is walking or stationary GPS Reports GPS coordinates See
changes in location Example: Could tell if user is driving
Slide 11
Sensor Types 4/18/2014Smartphone Based Human Behavioral
Analysis11 Light Intensity of light Reports data in a single value
Example: Indoors/Outdoors, Weather, Time of day Sound Detects sound
waves Reports loudness Example: Voice recognition Proximity Changes
depending on surrounding objects Boolean value Example: Device is
in tight space such as pocket or purse
Slide 12
Behavior Analysis App Version 1.0 4/18/2014Smartphone Based
Human Behavioral Analysis12 Shows values from each sensor in
real-time as the sensor updates Can save the data to a file on
phone Move file to computer to analyze with MATLAB Graph the data
Validate that it makes sense
Slide 13
Testing with App 3/10/2014Smartphone Based Human Behavioral
Analysis13 Most sensor data comes in a set of 3 points Few have
single values Tests related to typical smartphone user behaviors
Used MATLAB for data graphing and filtering
Slide 14
Light Sensor Test (Different Exposure) 4/18/2014Smartphone
Based Human Behavioral Analysis14
Slide 15
Accelerometer Test (Answer Call) 4/18/2014Smartphone Based
Human Behavioral Analysis15 XYZXYZ
Slide 16
App Development and Activity Recognition Andrew Jackson
Slide 17
4/18/2014 Smartphone Based Human Behavioral Analysis 17
Slide 18
Behavior Analysis App Version 2.0 4/18/2014 Smartphone Based
Human Behavioral Analysis 18 Integrates DTW machine learning
algorithm Allows training of the algorithm from the phone no
computer required Once trained, will recognize where the phone is
with almost 100% accuracy Visual Upgrades All features from
previous app rolled over
Slide 19
Further App Improvements 4/18/2014Smartphone Based Human
Behavioral Analysis19 Increased accuracy of activity recognition by
Increasing sensor sensitivity Polling each sensor more times a
second Tweaking DTW algorithm code Using multiple sensors at the
same time Continued adding new activities Added speech recognition
By demo day: Add in GPS coordinates Further explore light, sound,
and proximity sensor possibilities
Slide 20
Accelerometer, Light, and Proximity 4/18/2014 Smartphone Based
Human Behavioral Analysis 20 Chose to work with accelerometer first
because it can give some of the most useful data about the phone
Applying DTW on accelerometer data Next, applied light and
proximity sensors to get better results
Slide 21
Recognized Activities Last Presentation Now 4/18/2014
Smartphone Based Human Behavioral Analysis 21 Walking (with phone
in hand) Talking on phone Sitting on table Holding in hand Walking
(with phone in pocket) Running Stairs Sitting Driving
Slide 22
4/18/2014 Smartphone Based Human Behavioral Analysis 22 App
Demonstration
Slide 23
Dynamic Time Warping Andrew Jackson
Slide 24
Dynamic Time Warping Measures similarity between two sequences
which may vary in time or speed Calculates an optimal match between
the two given sequences or time series A distance-like quantity is
measured between the two series Smartphone Based Human Behavioral
Analysis4/18/201424
Slide 25
DTW Time Series 4/18/2014Smartphone Based Human Behavioral
Analysis25 Time series can be accelerated/decelerated as much as
necessary to give an optimal match. Cost: 3.3084e+05Cost:
2.7239e+06
Slide 26
Applying DTW to the Project 4/18/2014Smartphone Based Human
Behavioral Analysis26 Allows training for different activities with
disregard to time Supports three-dimensional data Provides
extremely accurate results Runs in O(n 2 ) time
Slide 27
FastDTW 4/18/2014Smartphone Based Human Behavioral Analysis27
Open-source algorithm based on original DTW Runs in O(n) time
Slightly less accurate Easy to port to Android
Slide 28
Environmental Sensors Robbie Rosati
Slide 29
Proximity Sensor 4/18/2014Smartphone Based Human Behavioral
Analysis29 Gives proximity in cm Most phones only return binary
values near and far Ability to check whether phone is pressed
against ear, in pocket, etc. neural networks
Slide 30
Light Sensor 4/18/2014Smartphone Based Human Behavioral
Analysis30 Detects ambient luminosity in lux Useful for
indoor/outdoor detection Could be included in gestures, used with
DTW
Slide 31
GPS 4/18/2014Smartphone Based Human Behavioral Analysis31 Can
get users latitude and longitude Could improve detection for if
user is driving Worst sensor with battery life Need to only use it
occasionally
Slide 32
Microphone 4/18/2014Smartphone Based Human Behavioral
Analysis32 Speaker recognition via NN algorithm Passive or active
detection Could use to detect loudness of rooms Also see things in
sound waveform like snoring
Slide 33
Neural Networks Robbie Rosati
Slide 34
Transition from Support Vector Machine to Neural Networks
4/18/2014Smartphone Based Human Behavioral Analysis34 Implemented
SVM into our Behavior Analysis app Ran too inefficiently for phone
hardware Difficult to train with the sound sensor Therefore,
decided to use an alternative algorithm that would fit our needs.
Decided on Neural Networks, a popular machine learning algorithm
for speech recognition
Slide 35
Neural Networks 4/18/2014Smartphone Based Human Behavioral
Analysis35 Algorithm used for machine learning and pattern
recognition Inspired by the way the brain recognizes objects and
sound Presented as systems of interconnected neurons that can
compute values Difficult to train in a short time so used an API
from Google to offload processing from the phone Integrates speech
recognition into the app
Slide 36
Google Speech API 4/18/2014Smartphone Based Human Behavioral
Analysis36 Open source API for speech recognition Could be coded
into our Behavior Analysis app Uses NN to interpret speech Can be
presented in text with further coding However, the app requires an
internet connection for this function since it streams audio to
remote servers
Slide 37
4/18/2014 Smartphone Based Human Behavioral Analysis 37 App
Demonstration
Slide 38
Raw Memory Extraction & Analysis Michael Armstrong
Slide 39
4/18/2014 Smartphone Based Human Behavioral Analysis 39
Slide 40
Overview 4/18/2014Smartphone Based Human Behavioral Analysis40
Retrieving the physical image of a device is our goal. Immense
variety of phones with an array of OS and applications. Current
solutions are time consuming and/or very expensive. Access to
deleted data A logical image is easier to obtain, but it omits
deleted data, and logical extraction interfaces usually enforce
access rules and may modify data upon access.
Slide 41
Samsung Galaxy S3 Mini 4/18/2014Smartphone Based Human
Behavioral Analysis41 Relatively inexpensive Compatible with teams
SIM cards SD card slot Has some sensors in case we need it as back
up for sensor testing
Slide 42
Testing Options 4/18/2014Smartphone Based Human Behavioral
Analysis42 Flashing is interpreted as a dump of the phones memory
into a format that is either hexadecimal or binary. Backup Software
Flashing box Linux Forensics Software dc3dd
Slide 43
dc3dd 4/18/2014Smartphone Based Human Behavioral Analysis43
Terminal based utility for Linux Parses a partition bit-by-bit and
creates binary image Advantages: Exactly what we need, easy to use
Disadvantage: Parses through empty space and fills it with zeroes,
creating a very large image file full of nothing however, there is
a workaround for this issue
Slide 44
Steps for Extraction 4/18/2014Smartphone Based Human Behavioral
Analysis44 Gain root access to Android using simple utility Extract
a copy of the database file Shrink the partition of a USB flash
drive to the smallest possible size and copy the database file to
the partition Use dc3dd to parse the partition and create the
binary image for the database file
Slide 45
Viewing the Image File 4/18/2014Smartphone Based Human
Behavioral Analysis45 Any hex viewer application should able to
view the data inside the image Chose GHex because it is easy to
obtain with Ubuntu and has search functions for finding the data we
are looking for
Slide 46
Finding Desired Data 4/18/2014Smartphone Based Human Behavioral
Analysis46 Messages and numbers are encoded as ASCII Locate
corresponding hex data using the search function in Ghex Verify by
comparing the located hex data to ASCII values Process should work
for any type of file because the ASCII data will there regardless
of file type
Slide 47
Example: Text Message 4/18/2014Smartphone Based Human
Behavioral Analysis47
Slide 48
Strings and Grep Commands 4/18/2014Smartphone Based Human
Behavioral Analysis48 Strings - Captures groups of letters
separated by spaces in a file Grep - Searches for an expression
matching the users input expression
Slide 49
Hidden Markov Model Andy McWilliams
Slide 50
Hidden Markov Model Smartphone Based Human Behavioral
Analysis4/18/201450 Way of predicting the next possible output in a
sequence of events Equivalent to recognizing or interpreting that
particular sequence Slightly complicated because there can be
multiple ways to produce the same observed output
Slide 51
Hidden Markov Model 4/18/2014Smartphone Based Human Behavioral
Analysis51 Accomplished through MATLAB There will be different
matrices such as the transition matrix and emission matrix
Transition matrix probability of transitioning from one state to
the next Emission matrix probability that one state emits the next
state
Slide 52
Hidden Markov Model 4/18/2014Smartphone Based Human Behavioral
Analysis52 1. Model Evaluation Problem What is the probability of
the observation? Given an observed sequence and an HMM, how
probable is that sequence? Forward algorithm 2. Path Decoding
Problem What is the best hidden state sequence for the observation?
Given an observed sequence and an HMM, what is the most likely
state sequence that generated it? Viterbi algorithm 3. Model
Training Problem How to estimate the model parameters? Given an
observation, can we learn an HMM for it?
Slide 53
Example with Binary 4/18/2014 Smartphone Based Human Behavioral
Analysis 53
Slide 54
HMM MATLAB 4/18/2014Smartphone Based Human Behavioral
Analysis54 Answers received contain new transition and emission
matrices with probabilities There are three files in which we work
with to execute the code Multiple Assumptions must be made in order
to create the three files
Slide 55
Three Files of MATLAB 4/18/2014Smartphone Based Human
Behavioral Analysis55 1. Markovsource is a data file that contains
multiple fields of phone numbers, dates, etc. 2. The transfer file
contains matlab code to convert the data into the necessary values
3. The markovTEmatrix file contains the transition and emission
matrices These three files are used to receive the state
probabilities for each matrix
Slide 56
Assumptions 4/18/2014Smartphone Based Human Behavioral
Analysis56 There are three kinds of fields within the data file
such as phone numbers with 10 digits, dates, and other non-
numerical alphabetic fields Within the matrices, the first ten rows
represent the date fields probabilities The next ten rows represent
the phone number fields There is an extra 1 or 2 rows that are used
for the non- numerical alphabetic fields
Slide 57
Transition and Emission Matrices 4/18/2014Smartphone Based
Human Behavioral Analysis57
Slide 58
Multiple MATLAB Functions 4/18/2014Smartphone Based Human
Behavioral Analysis58 Hmmdecode - used to calculate the state
probabilities of a sequence of emissions Hmmviterbi used to
calculate the most probable state path for a hidden Markov model
Hmmtrain used to calculate likelihood estimates for the transition
and emission matrices from a certain sequence Hmmestimate can be
used if the sequence is already known, but it has something
missing
Slide 59
State Probabilities 4/18/2014Smartphone Based Human Behavioral
Analysis59
Slide 60
Administration Andy McWilliams
Slide 61
Validation Plan 4/18/2014Smartphone Based Human Behavioral
Analysis61 Sensor Data Extraction & Analysis Retrieve real-time
data from all chosen sensors Graph and analyze each set of data by
performing tests Raw Memory Extraction & Analysis Test each raw
memory extraction option Retrieve binary image of databases Machine
Learning Algorithms Use algorithms to recognize patterns in users
behavior Use HMM algorithm to retrieve deleted data
Slide 62
Budget 4/18/201462Smartphone Based Human Behavioral Analysis
Also received $400 from Dr. Hus budget to purchase Samsung Galaxy
S4 Mini
Slide 63
Schedule Smartphone Based Human Behavioral
Analysis4/18/201463
Slide 64
Impacts Healthcare, Activity Monitoring, Forensics National
Science Foundation Dr. Hu will use project findings to help teach
CPS at rural schools in a virtual environment 4/18/201464Smartphone
Based Human Behavioral Analysis