A Non-Intrusive Multi-Sensor System for Characterizing Driver Behavior

Joao G. P. Rodrigues†, Fausto Vieira†, Tiago T. V. Vinhoza†, Joao Barros† and Joao P. Silva Cunha‡

Abstract— Understanding driver behavior is critical towardsensuring superior levels of safety and environmental sustain-ability in intelligent transportation systems. Existing solutionsfor vital sign extraction are generally intrusive in that theyaffect the comfort of the driver and may consequently leadto biased observations. Moreover, low-complexity devices suchas GPS receivers and the multitude of sensors present inthe vehicle are yet to be exploited to the full extent oftheir capabilities. We present a real-life system that combineswearable non-intrusive heart wave monitors with a wirelessenabled computing platform capable of gathering and process-ing the data streams of multiple in-vehicle sources. Observedvariables include electrocardiogram, vehicle location, speed,acceleration, fuel consumption, and pedal position, amongothers. Preliminary results show that the proposed system iswell suited not only for characterizing driver behavior butalso for identifying and mapping potentially dangerous roadsegments and intersections.

I. INTRODUCTION

Improvements in road safety and transportation efficiencyby means of intelligent transportation systems (ITS) are ar-guably dependent on our ability to characterize the behaviorof different classes of drivers and their response to variousevents, technologies and travel conditions. Fortunately, thevehicles of today are equipped with a large number ofsensors that can be leveraged to extract behavioral patterns.Typical instances, for example [1], integrate on-board sensorswith human-computer interfaces and estimate driver stresslevels by means of pressure sensors in the steering wheeland on the gearshift knob. Although capable of detectingaggressive braking, acceleration, steering, or inefficient fuelconsumption, among other patterns, the system forces driversto hold the steering wheel in a particular position, whichmodifies their natural behavior and affects the outcome ofthe measurement process.

Another limitation of typical systems aimed at character-izing driver behavior is their lack of connectivity. Typically,such devices are operated as stand-alone on-board units thatstore the gathered data for future offline processing. Withthe advent of vehicular sensor networks [2], which consistof cars, buses and other vehicles capable of collecting sensormeasurements and transmitting the data over mobile ad-hocnetworks, it is now possible to grant remote access to in-vehicle sensing systems and observe the collected data in

†Instituto de Telecomunicacoes, Departamento de Engenharia Elec-trotecnica e Computadores, Faculdade de Engenharia da Universidade doPorto, Porto, Portugal‡Dep. de Electronica, Telecomunicacoes e Informatica and IEETA, Uni-

versidade de Aveiro, Aveiro, PortugalThis work was supported in part by FCT and the international partnership

program MIT|Portugal under the MISC project MIT-PT/TS-ITS/0059/2008.

real-time. The application described in [3] gathers informa-tion from Global Positioning System (GPS) and On-BoardDiagnostics (OBD) devices from several cars, which is thensent to a central data collection point over wireless channelswith some delay tolerance. Although known to be largelydependent on the mobility patterns [4], wireless interfacesfor inter-vehicle and vehicle-to-infrastructure communicationcan be deemed as robust enough for real-time monitoring ofvehicle and driver behavior.

Ultimately, we would like to correlate positioning, timeand road map information with the stress levels and emo-tional response of drivers. From the analysis of the dailyannotations of a large number of drivers one can infer thatnear accident incidents are highly correlated with feelings ofanger [5]. From [6], it becomes clear that some locations aremore likely to induce stress than others, induced either fromfeelings of risk or from extra driving difficulty [7]. It hasalso been suggested that some cardiac waves are altered bydriving events [8]. Thus it is reasonable to assume that overallroad safety can be increased, if such feelings are detected intime using biomedical signal processing (see e.g. [6] or [9])in combination with positioning information and in-vehiclesensing.

Seeking to overcome the aforementioned drawbacks ofexisting driver monitoring systems and simultaneously lever-age the benefits of vehicular sensor networks, we set out todevelop a solution that satisfies the following criteria:

• Non-intrusive: Driver behavior should not be affectedby the devices that are used to acquire the necessarybiomedical signals.

• Comprehensive: The system should exploit the datacollected from a large number of heterogeneous sensorsand correlate the different readings.

• Remotely Accessible: The collected measurementsshould be transmitted to a remote location for analysisand processing.

• Real-time Enabled: The information must reach theremote location within certain deadlines to enable real-time decision-making.

• User-friendly: Visualization tools should make it easyfor a human user to interpret the acquired temporal,spatial and sensing information.

Our main contribution is a system architecture that meetsthese criteria by combining wearable heart wave monitoringtechnology, on-board sensing units and wireless networkingcapabilities. The proposed system, which was implementedin a real vehicle, aggregates data from a GPS receiver and anOBD system. This information is correlated with the signals

2010 13th International IEEEAnnual Conference on Intelligent Transportation SystemsMadeira Island, Portugal, September 19-22, 2010

WA4.2

978-1-4244-7658-9/10/$26.00 ©2010 IEEE 1620

obtained from a special kind of garment that features anembedded ElectroCardioGram (ECG) biosignal monitor. Thecollected sensor data includes time, location, speed, fuel effi-ciency, pedal position, temperature and drivers’ ECG. The in-vehicle unit is capable of transmitting the data via wifi accesspoints or third generation mobile communication systems.By allowing non-intrusive, continuous monitoring of driverbehavior, the proposed system enables easier prototypingand testing of on-board safety systems. As in [10], otherpotential benefits include the possibility of improving driver’sskills towards higher fuel efficiency, stronger breaking safetyand more effective defensive driving. Some advanced meth-ods of characterizing drivers’ behavior, such as time tocollision and time to line crossing, are not exploited inthis paper because they require information from additionalsensors not usually present in the vehicles, such as lateralvideo-cameras, steering angle information, frontal radar ordistance sensors.

In Section II, we introduce the system architecture, includ-ing the sensing capabilities, and address the main implemen-tation issues. Preliminary results and analysis are presentedin Section III. In section IV we conclude the paper bydiscussing a number of potential services and applicationsfor the proposed testbed.

II. SYSTEM ARCHITECTURE

The basic design of the proposed system, which can beintegrated in any vehicle, is represented in Fig. 1. It isdivided into three main physical blocks: the sensors, the in-vehicle units and the central server. Data acquisition fromthe sensors is performed by the in-vehicle units. These unitsare also responsible for aggregating, scheduling the data and

Fig. 1. The main components of the proposed system architecture.

transmitting the more important data to the central serverthrough the network. The rest of the data processing isperformed remotely at the central server. Data visualizationis offered at the central server, or from any other place forexample via the Internet. In the following section, we explainthe functionalities and operation modes of the differentsystem components.

A. Wearable Technologies

Driver behavior in urban transportation systems is tra-ditionally measured by means of travel logs and inquiriesor polls [5], [6]. However, wearable technologies are nowavailable that make it possible to monitor the heart wavein a non-intrusive fashion. For estimating the human stressor instantaneous emotional responses we use the so-calledVitalJacket R© [11], [12], which consists of a smart t-shirtdesigned for bio-monitoring (see Fig. 2). Tests show that it iscapable of continuously monitoring the ECG wave, returningrelevant information about the fatigue and stress levels of theuser.

Fig. 2. Smart t-shirt used to register ECG and monitor heart rate. Electrodesare only felt during the initial minutes, after which the garment becomescompletely inconspicuous. [11], [12]

The textile fabric of the garment connects three lightelectrodes (placed on the chest of the driver) with a smallelectronic device (66 x 38 x 16 mm) located in a frontpocket. The device, which weights 50 g, broadcasts throughbluetooth not only the wearer’s heart rate but also thecomplete heart wave. The device has been designed to have abattery autonomy of up to 48 hours of continuous operation(including wireless communication).

B. In-vehicle Sensors

The sensors perform the data gathering operations of thesystem. The first prototype consists of a GPS, an OBD andan ECG recording device, which are connected via USBor bluetooth to a netbook with a wireless interface. Oursystem is able to gather information from a large variety ofin-vehicle sensors, through the standard OBD protocol. Forcommunication with these sensors we use a commercially

1621

available OBD-tool capable of gathering data from more than50 different sensors. The tool connects to the computer viaUSB and allows the user to query data from any sensor,processing up to ten queries per second. Some of the datagathered from the OBD connection are: in-vehicle air tem-perature (◦C), outside air temperature (◦C), engine rotationsper minute (RPM), vehicle speed (m/s), instantaneous fueleconomy (`/100km) and accelerator pedal position (%). Thedata returned by these sensors is typically conveyed in 16 bitcodewords.

The GPS device is connected via USB or bluetooth andis compatible with National Marine Electronics Association(NMEA) specifications. It returns information about latitude,longitude, altitude, speed, road inclination and timestampwith 32 bits of precision. The connection to the GPS iscontrolled by an intermediate daemon called gpsd [13],which can be queried at a socket connection for the GPSrelated data.

C. Data Processing

The data from the sensors is gathered, stored and pro-cessed by the in-vehicle units. It may also be transmitted inreal-time for further processing at the central server.

SynchronizationThe data-mining module has to ensure the chronological

consistency of the data collected by different sensors. Thegpsd daemon is used to synchronize the system clockbetween units with an estimated precision of only a fewmilliseconds. When the daemon is used to retrieve locationinformation, the data can have a latency of up to one second,and new data is usually available with a clock of 1 Hz.At normal vehicle speeds 1 second of error can representmore than 10 meters of location error, so the GPS timeand clock is used to synchronize the data from the othersensors. The OBD protocol is based on query-response andthe ECG information is continuously saved in the bluetoothdriver buffer without time information. We use the systemtime at the moment of the query or buffer-read operation tosynchronize the information from the OBD and VitalJacketrespectively.

StorageAfter the synchronization process, the gathered data is

stored in a local relational database. For the first experiments,we used SQLite [14], since it stores the data in a singlefile, simplifying data transfer from the netbook (prototype)to another computer. The database structure is organized asfollows: a new table is created at every trip, and added onenew row for every second (sampling rate of 1 Hz), with afixed number of columns, one for each type of information.From Table I we can see that each new line will haveapproximately 2000 bits of data, resulting in about 1 MBof data per hour of driving. The database structure maybe changed to integrate new types of sensor data, such asreadings from accelerometers, number of detected bluetoothdevices and wifi hotspots.

CommunicationsThe unit classifies the data according to its real-time

relevance and schedules the transmission to the centralserver. The current scheduler algorithm is based on prioritiesfor each sensor, but it can be easily changed to allow forother techniques, such as queries from the central server. Asshowed in Table I, the bandwidth needed to transmit all theinformation gathered is around 2 kbps. This requirement canbe lowered to around 328 bps by classifying the heart waveas not real-time relevant. The transmission is encrypted inorder to ensure the privacy and integrity of the collectedinformation. The transmitted data is marked as sent, but it iskept in the local database until a reliable communication canbe established and the data acknowledged. At user-definedintervals, all the data is sent to the server or central entity,for example through a high-speed wireless connection neara stop. The databases are then merged in a complementaryfashion.

TABLE IBANDWIDTH USED BY EACH TYPE OF SENSOR

System Sensors Size (bits) Bandwidth (bps)Unit Time 32 32GPS Lat, Lon, Alt, Speed, Incl 32 160VJ Heart rate and wave 8 + 1600 8 or 1608

OBD 10 different sensors ∼16 160Total bandwidth 360 or 1960 bps

The central server is also responsible for the data fusion,analysis and visualization methods. The Keyhole MarkupLanguage (KML) Generator implemented in this entity isvital for the visualization technique described in the nextsubsection. Data fusion is carried out at the server offeringnew insights by means of statistical analysis, as shall bepresented in section III.

D. Data Visualization

Another goal of our system is to provide a user-friendlyinterface to visualize and analyze the gathered data. GoogleEarth [15] was chosen to serve as the visualization platformsince it provides a straightforward way to overlay spatialdata and correlate different types of information. The GoogleEarth GUI uses KML files, based on eXtensible Markup Lan-guage (XML), to import placemarks and other informationthat is easily visualized in four dimensions. We present thedata as small one second-distance line-segments where thespatial information is the actual line location in the map. Thecolor and height (altitude) are used to represent two differentvariables, such as heart rate and speed. The time toolbar (top-left corner of Fig. 4) allows a time interval window to beselected, in order to replay the data evolution over time.

In Fig. 3 we can observe the registered heart rate of a tripover a highway bridge. The driver is heading south and hisheart rate increases when he approaches the highway exitnear the end of the bridge. Fig. 4 was taken from a closeupof the same bridge, where two trips are visible. We can seethat in both datasets the drivers’ heart rate increased as heapproached the highway exit.

1622

Fig. 3. Overview of a trip over Freixo Bridge. Hot colors represent highervalues of heart rate while light colors represent lower values. [15]

Fig. 4. Closeup of the same bridge with two trips. Brightness representsheart rate, height represents speed. [15]

Using the presented method of data visualization it is easyto correlate high volumes of data. The user can select thevariables to be shown as well as the color, the height and thetime range he wishes to see. For example, visualizing datafrom two months in a one year-long dataset is as easy asmoving two sliders on the time toolbar to set the beginningand the ending date. That toolbar also has a play buttonthat allows the user to see a temporal animation of thevalues, such as showing the average fuel consumption andthe outside air temperature on the streets along the variousmonths of the year.

In the central server the database is very large and withmany daily-recorded trips. This visualization method withone line per trip is only fit to analyze up to a few trips inthe same road. For analyzing many spatially-concurrent tripsthe lines have to represent the time-average of the observedvalues over the trips made during the prescribed time interval.

For example, the server can generate a KML file where thecolor of the lines represent the average speed of the trips inthat road, during a chosen interval of time. This data fusionmethod is harder to perform, since the KML Generator must,for every road, search in the large database for the records ofthe same road, organize them by timestamp, and compute theaverage with a chosen time window. This way it is possibleto preserve the Google Earth temporal-animation capability.However, it becomes hard to include or exclude more trips,since the process must be restarted.

III. EXPERIMENTS AND RESULTS

Our preliminary experiments show that the proposed sys-tem is well suited not only for characterizing driver behaviorbut also for identifying and mapping potentially dangerousroad segments and intersections.

The proposed testbed is not designed to detect driver’sindividual stress factors, such as favorite music playing onthe radio, a goal against the supported soccer team, or apassenger who just did something unpredictable. However,these factors can be analyzed with further processing ofthe data in parallel with some control questionnaires ofpsychological nature.

Random errors in the data sets caused by occasional andtime-limited sensor faults, driver stress induced by othersources unrelated to driving or any other odd events becomeprogressively less significant as the database grows larger (asper the law of large numbers).

We deem the testbed data visualization to be matureenough to detect a road or cross-road that causes emotionalresponses in drivers in a non-random fashion (as seen inFigs. 3 and 4), and to rank the best roads for example basedon fuel consumption, average speed, number of stops orinduced stress levels.

As a practical example, after one hour of measurementsand simple data fusion, we could already detect a strongrelationship between acceleration and stress. Fig. 5 shows agraph that illustrates this relationship. To produce the graph,for each second of the trip the heart rate variation (y-axis)was calculated as the increase in heart rate in relation to theaverage of the last 20 seconds. The vehicle acceleration (x-axis) was calculated as the increase in speed on the last 5seconds. The graph shows the histogram of the heart ratevariation across the different values of acceleration. Or inother words, it plots the different values of accelerationversus the average of the heart rate variation corresponding tothat acceleration value. Curve-fitting indicates that those twomeasures can be related approximately through a quadraticpolynomial. As common sense would suggest, breaks andaccelerations are stress inducing, - or, alternatively, stressfulevents may cause a driver to break or accelerate - while, onaverage, constant speed is more relaxing. With larger datasetsit is possible to draw some conclusions regarding this relationto see if it holds for every type of road (streets or highways)and for a larger universe of drivers.

Other results related to outside or in-vehicle temperatures,air pollution and route durations can also be analyzed.

1623

Fig. 5. Average heart rate increase caused by different vehicle accelerations

IV. APPLICATIONS AND CONCLUSIONS

We presented a prototype for a driver monitoring systemthat combines wearable non-intrusive heart wave monitorswith a multitude of in-vehicle sensors. Preliminary resultsshowed that a testbed with these characteristics can be veryuseful for ITS in that it can assist in the characterization ofdriver behavior as well as in the identification and mappingof hazardous locations.

In addition, the system allows for easier prototyping andtesting of on-board security systems. Notice that large-scale data gathering and data mining is also possible, ifwe consider not one but a large number of vehicles thatgather data sets on a daily basis, which can be analyzed fordifferent purposes. Examples include real-time informationabout the driver’s condition, the state of the roads, triptimes, transportation delays, driving styles, congestion, fuelconsumptions and air pollution. Decision makers can exploitthe wealth of collected data for urban planning, risk assess-ment, adaptive transportation scheduling or human resourcesmanagement. More specifically, real-time data about driverfatigue can be used by public transportation authorities toimprove job assignment and driver rotation schedules, thusincreasing the safety of the transportation system. Routeoptimization based on fuel consumption or congestion is alsoan example of a strategic decision that could be taken basedon the collected data.

In the future we plan on improving our prototype toaccommodate all the functionalities explained in this paperand make it programmable over the air to avoid mainte-nance overheads. We also plan on integrating more types ofsensors and devices, such as smart phones or PDAs. Thesedevices incorporate many different sensing capabilities, likeGPS, accelerometers, audio and video. The audio and videorecording of the driver is extremely useful in characterizingdriver behavior because it can allow the analysis of humanresponses before and after a detected event. Such events canbe detected through abnormal responses from other sensors,such as quick breaking or high bio-response. If the video

camera is pointed at the outside then further information canbe collected about the event. This allows for the estimation ofreaction time and awareness level of the driver by correlatingthe recorded events with the responses detected by the on-board systems and ECG device.

Traffic black spots like junctions, lane merge sections,road works and others can be identified by statisticallyabnormal high emotional responses (e.g. transient heart ratevariations) or traffic-related (e.g. sudden breaks) coupledwith the positioning information. This information can beused to improve road signals, to increase drivers’ awarenessof dangerous places or to detect and prevent imminentaccidents. Other extensions that can be easily incorporated inour testbed include human interfaces for alerting the driverof his physical condition, dangerous events or inter-vehiclewarnings transmitted over VANETs [10] [2].

ACKNOWLEDGMENTS

The authors would like to thank Biodevices for grantingus some VitalJackets and for the help with its integration.

REFERENCES

[1] J. F. Pompei, S. Taly, S. J. Buckley, and J. Kemp, “An automobile-integrated system for assessing and reacting to driver cognitive load,”in Proceedings of Convergence, Detroit, Michigan, USA, Oct 2002,pp. 411–416.

[2] M. Nekovee, “Sensor networks on the road: the promises and chal-lenges of vehicular adhoc networks and vehicular grids,” Edinburgh,UK, May 2005.

[3] B. Hull, V. Bychkovsky, Y. Zhang, K. Chen, M. Goraczko, A. Miu,E. Shih, H. Balakrishnan, and S. Madden, “Cartel: a distributedmobile sensor computing system,” in SenSys ’06: Proceedings of the4th international conference on Embedded networked sensor systems.New York, NY, USA: ACM, 2006, pp. 125 – 138.

[4] U. Lee and M. Gerla, “A survey of urban vehicular sensing platforms,”Computer Networks, vol. 54, no. 4, pp. 527 – 544, 2010.

[5] G. Underwood, P. Chapman, S. Wright, and D. Crundall, “Angerwhile driving,” Transportation Research Part F: Traffic Psychologyand Behaviour, vol. 2, no. 1, pp. 55 – 68, 1999.

[6] J. Healey and R. Picard, “Detecting stress during real-world drivingtasks using physiological sensors,” IEEE Transactions on IntelligentTransportation Systems, vol. 6, no. 2, pp. 156 – 166, Jun 2005.

[7] R. Fuller, “Towards a general theory of driver behaviour,” AccidentAnalysis & Prevention, vol. 37, no. 3, pp. 461 – 472, 2005.

[8] E. Simonson, C. Baker, N. Burns, C. Keiper, O. H. Schmitt, andS. Stackhouse, “Cardiovascular stress (electrocardiographic changes)produced by driving an automobile,” American Heart Journal, vol. 75,no. 1, pp. 125 – 135, 1968.

[9] G. Rigas, C. Katsis, P. Bougia, and D. Fotiadis, “A reasoning-based framework for car driver’s stress prediction,” in Proceedings of16th Mediterranean Conference on Control and Automation, Ajaccio,Corsica, France, Jun 2008, pp. 627 – 632.

[10] P. Papadimitratos, A. De La Fortelle, K. Evenssen, R. Brignolo, andS. Cosenza, “Vehicular communication systems: enabling technolo-gies, applications, and future outlook on intelligent transportation,”IEEE Comm. Mag., vol. 47, no. 11, pp. 84 – 95, Nov 2009.

[11] Biodevices. (2010, Mar) Vitaljacket R©, a wearable ambulatoryecg medical device. Aveiro, Portugal. [Online]. Available: http://www.vitaljacket.com

[12] J. P. S. Cunha, B. Cunha, A. S. Pereira, W. Xavier, N. Ferreira, andL. A. Meireles, “Vital jacket: A wearable wireless vital signs monitorfor patients’ mobility in cardiology and sports,” in Proceedings of the4th International ICST Conference on Pervasive Computing Technolo-gies for Healthcare, Munchen, Germany, Mar 2010.

[13] Berlios. (2010, Mar) GPSD. [Online]. Available: http://gpsd.berlios.de[14] SQLite. (2010, Mar). [Online]. Available: http://www.sqlite.org[15] Google. (2010, Mar) Google Earth. [Online]. Available: http:

//earth.google.com

1624