Embed Size (px)
Citation preview
The Unintended Use of Mobile Phone Features:Teaching Mobile Phones new tricks
Dr Eiman KanjoMobile Sensing and DataScience, Senior Lecturer
Computing Department
Nottingham Trent University
@eimankanjo
https://www.linkedin.com/in/eimankanjo/
CodeMobile April 2017
Background• Senior Lecturer at Nottingham Trent University
• Have been Developing for smart phones since 2004 (started with Symbian)
• I was one of the first researchers to utilise mobile phones as research tools.
• Moved between various research organisations including Nottingham University, Cambridge University.
• Started teaching mobile Development 2010 (Android)
• My PhD was in Computer Games at Abertay University, Dundee (2005).
Overview
• Mobile phone development for academic research
• Built-in and external Mobile Phones features and capabilities
• Mobile Sensing Research
• Utilising Mobile Phone Notifications and Mobile to model behaviour
• Programming mobile Phones to sense mood
• Utilising the latest development in Beacons to enable context and physical computing
• (Data Annotation Tools) the need for hardware and software tools to enable data collection
• Next steps for Mobile Phones in research
• Mobile Development at NTU
Mobile Phones for Academic Research
Social Science
Psychology
Health
Mental Health
GIS and Spatial Analysis
Data Collection Tools
Animal and wildlife Tracking
Smart Farms
Disaster ManagementMarketing & Media Studies
CodeMobile 2017 [email protected]
Retail and Marketing
Examples of mobile Phone Adoption for Research
Studies that have used mobile phones for mood disorder research (Torous, John et al ,2015)
1065 studies. Ten studies on major depressive disorder and 4 on bipolar disorder were
included. Nine out of 10 studies on depression related smartphone applications featured activedata collection and all 4 studies on bipolar disorder featured passive data collection.
Torous, John et al. Current research and trends in the use of smartphone applications for mood disorders Internet Interventions , Volume 2 , Issue 2 , 169 - 173
CodeMobile 2017 [email protected]
Mobile Phone Sensing
We take advantage of Mobile Phones sensing Capabilities
Data Collection
Monitoring Measurement Intervention DisseminationVisualisation
CodeMobile 2017 [email protected]
Mobile Sensing
Notifications
Audio
Camera
Motion
Location
Wireless Signals
Screen Activities
Wireless connectivity
Messages
Calls
Social Media
Web browsing
Messages
Web browsing
Light Sensor
Heart Rate
Sources of Information on Mobile Phones
CodeMobile 2017 [email protected]
Apps
MobSens• Mobile phones can provide coverage where static
sensors are hard to deploy and maintain, and largenumbers of cell phones already exist around theworld, providing the physical sensinginfrastructure.
• Availability of more powerful operating systemsand the transfer of standardized programminglanguages on ever-smaller computing platformshave spurred the recent development of softwareapplications for mobile computers
• Such systems can benefit from localcommunities as the driving element for environ-mental sensing. This approach, sometimesreferred to as “citizen science,” uses mobilesensor technology to help individuals personallycollect, share, compare, and participate inresearch work.
CodeMobile 2017 [email protected]
http://pages.experts-exchange.com/processing-power-compared/
MobSensMobSens software components installed on the phones often performthe following operations: sensing, filtering, processing, and loggingsensor data; rendering screen displays, including graphs, maps, and userinterfaces; and uploading data streams to back-end servers in real time.
Usually, each sensor data entry is combined with various informationsuch as the last valid GPS location plus additional information such Time ,Date (or UTC date), time, the phone’s International Mobile EquipmentIdentity (IMEI), user name, journey ID..etc.
CodeMobile 2017 [email protected]
Comparison of graphs showing the discrepancy between the two noise measurements generated by the N95 phone microphone NoiseSpy and the Norsonic meter
NoiseSpy: Turning Phone Microphone into Noise Sensor
CodeMobile 2017 [email protected]
NoiseSpy: Turning Phone Microphone into Noise Sensor
Screenshots of NoiseSPY(on Symbian)
Noise data collected by cyclists and overlaid on
Maps along with a time graph
Eiman Kanjo: NoiseSPY: A Real-Time Mobile Phone Platform for Urban Noise Monitoring and Mapping. MONET 15(4): 562-574 (2010)
Sound measurements a trial in Cambridge, the coloured traces represent the loudness in dBA
A cycling courier user collecting data using NoiseSpy system
Eiman Kanjo: NoiseSPY: A Real-Time Mobile Phone Platform for Urban Noise Monitoring and Mapping. MONET 15(4): 562-574 (2010)
CodeMobile 2017 [email protected]
PollutionSpy
CodeMobile 2017 [email protected]
Eiman Kanjo, Jean Bacon, David Roberts, Peter Landshoff: MobSens: Making Smart Phones Smarter. IEEE Pervasive Computing 8(4): 50-57 (2009)
Fresh: Cell-ID based Mobile Forum for CommunityEnvironmental Awareness
• Fresh is a Mobile interface that utilizes GPRS networking based on cell-IDs from mobile phones. This mobile utility (forum) encourages users to interact at a specific geographical area to allow people to discuss issues related to their local environment. .
• (e.g.using GsmCellLocation)• Get CellID()
• GetLac() gsm location area code..
CodeMobile 2017 [email protected]
First Location based Forum based on CellIDs
ViralNet for Crowd Managemennt
• Utilizes forwarding Protocol which adapts to the messages sizes, types, frequencies and initiators.
• Connection to furthest node rather than the nearest.
• Based Mainly on Mobile phones communications over Bluetooth
• Extends the Bluetooth range to hundreds of meters depending on the crowed size.
Mobile Affective Sensing System
Eiman Kanjo, Alan Chamberlain, “Emotions in Context: Examining Pervasive Affective Sensing", Personal and Ubiquitous Computing Journal, 2015.
CodeMobile 2017 [email protected]
mFeel
Kanjo, E., "mFeel: An affective mobile system", IEEE Pervasive Computing, vol. 11, issue 3, pp. 43-45, 2012.
CodeMobile 2017 [email protected]
Shopmobia
We leverage mobile phone processing power and physiological sensor in order to perceive and record customer emotions. Data are aggregated around different zones using NFC tagging technology.
Understanding Brain Activities and responses to Outdoor Places
Luluah Albarrak and Eiman Kanjo, "NeuroPlace: Making sense of a place", 4th International Conference on Augmented Human in Cooperation with ACM, Stuttgart Germany, ACM, March, 2013.
Eiman Kanjo, Alan Chamberlain,” ShopMobia: Emotion based Shop Rating System”, Affective Computing and Intelligent Interaction Conference, Geneva, Switzerland, IEEE, September, 2013.
CodeMobile 2017 [email protected]
Physical Computing: Beacons
• Bluetooth beacons are hardware transmitters - a class of Bluetooth low energy (LE) devices that broadcast their identifier to nearby portable electronic devices. The technology enables smartphones, tablets and other devices to perform actions when in close proximity to a beacon.
• iBeacon introduced by Apple in 2013 to enable retail/location based payment.
• Then few versions of Beacons have followed.
• Eddystone is a Google's standard for Bluetooth beacons (released by Google in July 2015).
http://www.travelonart.com/new-media/che-cosa-sono-i-beacons/
CodeMobile 2017 [email protected]
Eddystone
• Eddystone-UID functions in a very similar way to Apple'siBeacon, however, it supports additional data exchange.
• The beacon information available includes battery voltage,beacon temperature, number of packets sent since laststartup, and beacon uptime.
• Many Eddystone beacons have sensors (such as motionsensor, temperature, humidity), internal memory, longlifebattery, some have NFC chips in them.
Eddystone• There are two basic ways for apps to interact with beacons:
Monitoring and Ranging. • Monitoring allows detecting if there are beacons in range, • while Ranging is used to estimate proximity from beacons.
• Additionally, you can use the Physical Web/Eddystone standard tobroadcast URLs instead of beacon IDs.
• Android natively supports this as Nearby Notifications and on iOSit's integrated with Chrome.
• It's important because, on Android, Chrome is the default browserwhich makes it possible to reach millions of users without buildinga dedicated app.
BeaconsPhysical Computing/IoT enablers
Beacons
• Long Range Location Beacons: For indoor location, IoT prototyping or enterprise-grade deployments choose Location Beacons - the most robust beacons on the market. They support multiple packets, mesh networking, and have built-in sensors features: 5 sensors, NFC, mesh, GPIO, 1Mb EEPROM.
• Stickers are much smaller, but that also comes with shorter battery life (up to a year) and range. The idea behind stickers is to create nearables: smart objects broadcasting data about their location, motion, and environment.
https://estimote.com
CodeMobile 2017 [email protected]
Current Projects based on Beacons
• For Fitness and to track Physical Movement
• Location based games (Pokemon Go Style)
•Monitoring Customer Behaviour in Retail Environment
• For Smart City applications and infrastructure
CodeMobile 2017 [email protected]
Annotation on the go for Data Collection
1. Screen Buttons (Image Buttons)
2. Screen Swipe:(left and right).
3. Device Gesture: based on the phone accelerometer
motion.
4. Volume Buttons: logs volume levels.
5. Speech Labelling: For this, we utilised Android
Speech API for voice recognition to count words.
NFC as Annotation
Enablers (Proximity)
NFC on-Body: Two tags are positioned on the left and right
shoulder.
NFC tags on-Wall: scans NFC tags on the wall
Activity MonitoringGPS/Accelerometer data can be used to classify a user’s movement:
Running
Walking
Stationary
Combining motion classification with GPS tracking can recognize the user’s mode of transportation:
Subway, bike, bus, car, walk…
The Nike+iPod Sports Kit is an activity tracker device, developed by Nike, Inc., which measures and records the distance and pace of a walk or run.
Large number of Researchers are currently working on various acticty recognition algorithms which involves machine Learning (DeepLearning) on the go
Next Steps in the research world (1)
• Artificial Intelligence and Deep Learning to enable real-time training and prediction.
• E.g. On Mobile GPU, FPGAs(e.g. Lattice chip on Apple 7), Mobile TensorFLow…
Custom Sensors
Mobile phones still lack many sensing capabilities such as :
Blood pressure, heart rate, EEG
Barometer, temperature, humidity
Air quality, pollution, Carbon Monoxide
Specialized sensors can be embedded into peripherals:
Earphones
Dockable accessories / cases
Next Steps in the research world (2)
Next Steps in the research world (3)
• Think of it as a mini robot that can feel and monitor your behaviour.
• Or a form-changing device for gaming, entertainment or to deliver intervention(e.g. manage anxiety and stress).
Not just applications but new type of hardware, embedded devices and style are needede.g. Tactile biofeedback gadget
Actuators
Vibration
Heat/cooling
Electrical stimulation
Aroma generator
Mobile DevelopmentNottingham Trent University• Few Mobile Development modules are taught at NTU (Android &iOS)
• Undergraduate and Postgraduate Levels
• Mobile Development is popular among Final year projects
• New MSc Mobile Computing Course is planned
• Always on the look for Guest Lecturers
• Dedicated labs with a wide range of equipment including mobile phones for testing (one phone per student)
• This year our students have developed a wide range of applications and made use of a many native and external libraries.• E.g. most of our students have made use of Firebase to store and access their data.
• Many of the developed applications are professionally designed and tested.
• We encourage our students to choose ideas that are lied to real-business.
• Companies involvement is encouraged at early stages.
• If interested in delivering a guest lecture, engaging with students projects, or proposing a project idea, then please get in touch:
Mobile DevelopmentNottingham Trent University
