Darwin Phones: the Evolution of Sensing and Inference onMobile Phones

Emiliano Miluzzo†, Cory T. Cornelius†, Ashwin Ramaswamy†, Tanzeem Choudhury†,

Zhigang Liu§, Andrew T. Campbell†

†Computer Science, Dartmouth College, Hanover, NH, USA

§Nokia Research Center, 955 Page Mill Road, Palo Alto, CA, USA

ABSTRACT

We present Darwin, an enabling technology for mobile phonesensing that combines collaborative sensing and classifica-tion techniques to reason about human behavior and con-text on mobile phones. Darwin advances mobile phone sens-ing through the deployment of efficient but sophisticatedmachine learning techniques specifically designed to run di-rectly on sensor-enabled mobile phones (i.e., smartphones).Darwin tackles three key sensing and inference challengesthat are barriers to mass-scale adoption of mobile phonesensing applications: (i) the human-burden of training clas-sifiers, (ii) the ability to perform reliably in different envi-ronments (e.g., indoor, outdoor) and (iii) the ability to scaleto a large number of phones without jeopardizing the“phoneexperience” (e.g., usability and battery lifetime). Darwin isa collaborative reasoning framework built on three concepts:classifier/model evolution, model pooling, and collaborativeinference. To the best of our knowledge Darwin is the firstsystem that applies distributed machine learning techniquesand collaborative inference concepts to mobile phones. Weimplement the Darwin system on the Nokia N97 and Ap-ple iPhone. While Darwin represents a general frameworkapplicable to a wide variety of emerging mobile sensing ap-plications, we implement a speaker recognition applicationand an augmented reality application to evaluate the ben-efits of Darwin. We show experimental results from eightindividuals carrying Nokia N97s and demonstrate that Dar-win improves the reliability and scalability of the proof-of-concept speaker recognition application without additionalburden to users.

Categories and Subject Descriptors: C.3 [Special-Purposeand Application-Based Systems] Real-time and embeddedsystems

General Terms: Algorithms, Design, Experimentation,Human Factors, Measurement, Performance

Keywords: Mobile Sensing Systems, Distributed MachineLearning, Collaborative Inference, Mobile Phones

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.MobiSys’10, June 15–18, 2010, San Francisco, California, USA.Copyright 2010 ACM 978-1-60558-985-5/10/06 ...$10.00.

1. INTRODUCTION

The continuing need to communicate has always pushedpeople to invent better and more efficient ways to conveymessages, propagate ideas, and share personal informationwith friends and family. Social-networking, for example, isthe fastest growing phenomenon of the Internet era wherepeople communicate and share content with friends, family,and acquaintances. Recently, researchers started investigat-ing new ways to augment existing channels of communica-tion and improve information exchange between individu-als using the computational and sensing resources offeredby sensor-enabled mobile phones (aka smartphones). Thesephones already utilize sensor data to filter relevant infor-mation (e.g., location-based services) or provide better userexperiences (e.g., using accelerometer data to drive mobilephone sensing applications). However, information aboutuser’s behavior (e.g., having a conversation) and personalcontext (e.g., hanging out with friends) is often providedmanually by the user. This naturally leads to the followingthoughts: what if the available sensors are further exploitedto automatically infer various aspects of a person’s life inways that have not been done before? What if the character-ization of the person’s microcosmos could be seen as a newform of communication? We believe that as sensor-enabledmobile phones become commonplace, they can be used ata personal-scale to enrich and support communication andcollaboration, to measure and improve task performance,and to aide in the assessment of health and wellness.

There is a growing research effort in using mobile phonesto infer information about people’s behavior and their envi-ronment [46, 21, 20, 30, 29, 10, 27]. These systems typicallyrely on pre-trained models or classifiers, where the trainingdata from events of interest are acquired in advance. It is of-ten exceedingly hard to obtain a representative training setfrom which to build reliable classifiers (e.g., samples of anindividual’s voice in all possible environments). As a resultclassifiers tend to perform poorly. Furthermore, current ap-proaches do not take advantage of increased sensing densityoffered, for example, by the cloud of mobile phones aroundus. This cloud represents an ensemble of in situ resourcesthat can cooperate to boost sensing performance, make sens-ing more robust, and more effectively and scalably achievea common sensing task.

With the rising interest in mobile phone sensing applica-tions we believe there is a need to provide mechanisms thatmaximize the robustness, accuracy, and scalability of theseapplications. In this paper, we present Darwin, a novel col-

5

envir 2

envir 1

phone C

Evolution Pooling Collaborative Inference

phone A

phone B

model C,A,B

model A,B,C

model B,A,Cenvir 3

model A

it’s event Y!We all agree

(b) (c)(a)

phone A

BackendServer

Figure 1: Darwin steps: (a) evolution, (b) pool-ing and (c) collaborative inference. They representDarwin’s novel evolve-pool-collaborate model imple-mented on mobile phones.

laborative reasoning system that is self-extending and uti-lizes co-located mobile devices to achieve better accuracyand scalability, at lower cost to the user. As shown in Fig-ure 1, Darwin combines three different computational stepsto achieve its goal:

Classifier evolution is an automated approach to up-dating models over time such that the classifiers are robustto the variability in sensing conditions common to mobilephones (e.g., phone in the pocket, in pocket bag, out of thepocket), and settings (e.g., noisy and loud environments).A fully supervised learning method, where labeled examplesfrom different context are provided to the system, would notbe practical in this case since the phone owner would con-tinually have to provide labels any time an event is detectedthat is determined to be in a different setting and context.This simply does not scale and would be unacceptable tousers. While self-evolving classification models techniqueshave been investigated in the past [54], we show the actualdeployment of such techniques on a real phone based system.

Model pooling is a novel technique which is designedto answer the following question: can we reuse models thathave already been built and possibly evolved on other phones?This would increase the system scalability because therewould be no need to retrain classifiers for events which al-ready have classifiers trained to recognize them. With pool-ing, mobile phones exchange classification models wheneverthe model is available from another phone, thus, allowingmobile phones to quickly expand their classification capa-bilities; that is, if a given mobile phone does not have amodel for a person or an event, there is no need for it to cre-ate a new classifier as it can readily obtain and use anotherphone’s model. Note, that models can be exchanged in-situbetween co-located phones or from servers over the network.In either case the basic pooling process remains the same.

Collaborative inference combines the classification re-sults from multiple phones to achieve better, more robustinference with higher confidence in the sensing result. Afterpooling, co-located mobile phones all have the same clas-sifiers. At this point they can run the same inference algo-rithms in parallel and a final inference result can be obtainedby combining the output from the different phones. This al-lows the system to be more robust to degradation in sensingquality experienced by some of the phones (e.g., a personcarrying the phone in the pocket) and take advantage of im-

proved sensing quality offered by others (e.g., the phone isout of the pocket near the event to be sensed).

We show the performance of Darwin by exploiting the au-dio modality of mobile phones, in particular, we show thebenefit of applying Darwin to a speaker recognition applica-tion using audio sampled by the onboard microphone. Weshow the performance of the speaker recognition algorithmon the Nokia N97 [6] and Apple iPhone [1] in different set-tings and context when Darwin and the speaker recognitionapplication are used by eight people.

The reason we select speaker recognition is not because weintend to design a new speaker recognition algorithm (thereis a considerable amount of literature on this topic [44, 43,26, 18, 51]), but to show how Darwin improves a mobilesensing application inference quality.

Darwin is founded on an opportunistic sensing paradigm[12], where the user is not an active participant in the sens-ing activity (i.e., actively taking a sensor reading). In thiscase, sensing happens automatically and continuously whenthe system determines that the sensing context is right forsensing. Darwin can be thought of as a sensing system run-ning in the “background mode” of the mobile phone withoutany user intervention in actual sensing. The key contribu-tion of our work is to show how Darwin can boost the in-ference accuracy of mobile sensing systems by applying dis-tributed computing and collaborative inference concepts tothese systems when devices come together opportunistically.We conjecture that Darwin applied to other mobile phonesensing applications and systems that use the microphone asan audio sensor [27, 10] would also see similar performanceimprovements because audio sensed data is sensitive to thecharacteristics of the environment (e.g., noise, other peoplespeaking, etc.) and sensor context of the phone (e.g., inor out of pocket for example). In the paper, we also showhow Darwin could be integrated in a mobile social network-ing application, a place discovery application, and a friendtagging application.

Today, mobile phone sensing applications mostly exploitthe GPS, accelerometer, digital compass, and microphonesensors for personal sensing. In the future, mobile phonesensing will be societal-scale supporting a broad set of social,health and environmental applications, such as, tracking pol-lution, population well-being, or the spread of disease. It isalso likely that more sophisticated sensors will be embeddedin phones, such as, pollution and air quality sensors [20] andgalvanic skin response (GSR) sensors. The sensing and infer-ence quality of these applications is affected by many factors.Importantly, phone sensing context, i.e., the position of thephone on a person’s body in relation to the sensed event, ischallenging for these emerging applications. A phone in thepocket or bag might perform poorly when sensing air qual-ity or audio events. Classification models are also limited bythe quality of the trained data and their inability to capturedifferent characteristics from the data in different environ-ments. Darwin’s novel evolve-pool-collaborate model isdesigned to provide a foundation for a broad family of ex-isting and emerging sensors and applications, as shown inFigure 2. To the best of our knowledge, Darwin representsthe first system implemented on a number of mobile phonesthat can evolve, pool, and enable cooperation providing ro-bust, efficient, and scalable sensing.

The structure of the paper is as follows. Section 2 presentsthe detailed design of the Darwin system, followed in Sec-

6

Figure 2: Examples of application domains Darwincan be applied to: social context (e.g., in conversa-tion, in a meeting) and ambient audio fingerprintingusing the microphone; pollution monitoring leverag-ing the phone’s pollution sensor; radio fingerprint-ing for localization with GPS, WiFi and cellular tri-angulation; and applications exploiting the phone’scamera.

tion 3, by a discussion of privacy and trust issues. Section 4presents the system implementation and performance evalu-ation of Darwin applied to a proof-of-concept speaker recog-nition application. Following this, we discuss a number ofother sensing applications built on Darwin and then discussthe related work in Section 5 and Section 6, respectively.Section 7 concludes with some final remarks.

2. DARWIN DESIGN

In this section, we present the detailed design of the Dar-win system including the use case speaker recognition appli-cation.

2.1 Design Considerations

The design of Darwin is governed by the limited com-putational resources on the phone to run computationallyintensive machine learning algorithms and mobility issues.In what follows, we discuss these motivating factors and thedesign decisions that address them.

The main goal of mobile phones is expanding way beyondjust making phone calls. Compared to early mobile phones,modern smartphones are also powerful programmable plat-forms with up to 600MHz processors and 1GB of applicationmemory [34]. While smartphones have increasing resourcesrunning continuous sensing applications they present a num-ber of important challenges. These range from the designof efficient duty cycling algorithms that can maintain ac-ceptable fidelity and time between charges to the need topush more intelligence to the phone in terms of classifica-tion algorithms without impacting the user experience (e.g.,freezing the phone, slowing the UI, blocking calls). Machinelearning algorithms that run on the phone to process sen-sor data should be implemented in an efficient lightweightmanner. Darwin is designed to reduce on-the-phone com-putation based on a split-level computation design [29], of-floading some of the work to backend servers (as discussedin Section 2) while trading off the cost for local computationand wireless communication with backend servers.

Users carrying mobile phones also presents a number ofchallenges for continuous sensing applications that have tooperate under real-world mobility conditions. The contextof the phone is challenging to sensing. Users carry phonesin many different ways. Therefore, when a phone senses anevent, its context (e.g., in/out of the pocket, in/out the bag,etc.) will impact the sensing and inference capability of the

phone. Another challenge that mobility creates is that thesame phone may sense the same type of event under differentconditions (e.g., the same person speaking in a quiet office ornoisy restaurant). This leads to poor inference. A group ofco-located phones running the same classification algorithmand sensing the same event in time and space could computedifferent inference results because of the context problemand slight environmental differences, as discussed above. Inessence, each phone has a different viewpoint of the sameevent. These real-world issues arise because sensing takesplace out in the wild – not in a controlled laboratory setting– and is governed by the uncontrolled mobility of users.

Darwin exploits mobility and addresses these challenges.It uses classifier evolution to make sure the classifier of anevent on a phone is robust across different environments –works indoors and outdoors for example. Extracting fea-tures from sensed events in order to train a classifier iscostly for a mobile phone in terms of computation and time.Darwin allow phones to pool classification models when co-located or from backend servers. Pooling radically reducesthe classification latency because a phone can immediatelystart to make inferences without the need to train classi-fiers. Different phones running the same classifier and sens-ing the same event are likely sensing the event differently,as discussed above. Darwin uses collaborative inference tocompensate for this difference, boosting the final inferenceresult. Darwin exploits mobility because it is designed to beopportunistic in its use of classifier evolution, pooling, andcollaborative inference.

2.2 Darwin Operations

In what follows, we present a high level description of howDarwin operates: (1) each mobile phone builds a model1 ofthe event to be sensed through a seeding phase. Over time,the original model is used to recruit new data and evolvethe original model (see Figure 1(a)). The intuition behindthis step is that, by incrementally recruiting new samples,the model will gather data in different environments andbe more robust to environmental variations. The phonecomputes the feature vector2 locally on the phone itself andsends the features to a backend server for training. This isbecause the feature vector computation is quicker and moreenergy efficient than the training phase of a machine learningalgorithm such as a Gaussian Mixture Model (GMM), whichis the technique we use (it takes approximately 2 hours totrain a GMM with 15 seconds of audio from experimentson the Apple iPhone and N97). (2) When multiple mo-bile phones are co-located they exchange their models sothat each phone has its own model as well as the co-locatedphones’ models. Model pooling, as shown in Figure 1(b),allows phones to share their knowledge to perform a largerclassification task (i.e., in the case of speaker recognition,going from recognizing the owner of the phone to recogniz-ing all the people around in conversation). After models are

1A classification model is represented by a mathematicalexpression with parameters. For example, in the case of aGaussian classification model, the model is identified by aGaussian function with mean and standard deviation as theparameters. Refer to Section 2.5 for further details.2A feature vector is a vector of numerical elements represent-ing an event. For example, in the case of activity recognitionapplications that use the accelerometer, two feature vectorelements often used are the mean and standard deviation ofthe accelerometer readings.

7

pooled from neighboring mobile phones, each phone runs theclassification algorithm independently. However, each phonemight have a different view of the same event – i.e., differ-ent phone sensing context. For example, one of the phonesmight be inside the user’s pocket whereas another one mightbe outside, or one of the phones could be closer to the sens-ing event than others. (3) Collaborative inference exploitsthis diversity of different phone sensing context viewpointsto increase the overall fidelity of classification accuracy, asillustrated in Figure 1(c).

2.3 Speaker Recognition Use Case

We choose speaker recognition as our proof-of-concept ap-plication because the audio modality is generally sensitive toenvironment and phone sensing context and we believe thefindings from this application will generalize to other classi-fication problems such as in [27, 10] or pollution for example[20] for which the phone sensing context is important. Thespeaker recognition application attempts to determine theidentity of a speaker by analyzing the audio stream comingfrom a microphone. The recognition process includes thefollowing steps:

Silence Suppression and Voicing. The system firsteliminates audio chunks that contain silence or low ampli-tude audio and then runs a voicing routine to remove thechunks that do not contain human voice. By focusing onlyon chunks that are likely to contain human speech we reducethe false-alarm rate of the classification system. The silencesuppression filter works on 32 ms of audio at a time and dis-cards portion of the audio whose root mean square (RMS)value falls below a threshold T . The threshold T is deter-mined experimentally under various conditions, for example,recording voice using the mobile phone in quiet indoor envi-ronments, on a sidewalk of a busy road, and in a restaurant.The voicing is performed by training a GMM using severalhours of non-voice audio captured in various conditions (e.g.,quiet environments, noisy from car traffic, etc.) and discard-ing the audio chunks whose likelihood falls with a +/- 5%from the non-voice likelihood. This threshold is also deter-mined experimentally and found to be accurate for manydifferent scenarios. More advanced techniques could be con-sidered in future work such as the voicing scheme proposedin [11].

Feature Vector. The feature vector consists of Mel Fre-quency Cepstral Coefficients (MFCCs) which are proven tobe effective in speech audio processing [17][16][42]. We usecoefficients 2 to 20 and skip the first coefficient because itmodels the DC component of the audio.

Speaker Model Computation. Each speaker is mod-eled with a mixture of 20 gaussians (hence, a 20-componentGMM). The reason we use GMM is because GMM algo-rithms are widely used in the speaker recognition literature[40][39]. An initial speaker model is built by collecting ashort training sample – 15 seconds in our current implemen-tation. When a user installs the application on their phonethey are asked to provide a sample of voice by reading loudtext displayed on the phone screen. In Darwin, the initialmodel evolves to capture the characteristics of the differentenvironments where the person happens to be located.

Each speaker i’s model Mi corresponds to a GMMi, ∀ i, i= 1..N (where N is the number of speakers), and GMMi isthe model trained and evolved by Pi for speaker i3. A GMMi

3We associate one speaker to a single phone. Whenever the

Figure 3: (a) Training and (b) Evolution steps.

is characterized by the tuple Mi = <µi ,Σi ,wi>, where µi

is the P × Q multi-dimensional array containing the meanvalues of the gaussian distribution, P is the number of com-ponents and Q is the number of dimensions of the GMMmodel. Σi is the Q × Q × P multi-dimensional covariancematrix, whereas wi is the 1 × P array of weights for eachgaussian component. For a detailed discussion of a GMMfor speaker identification see [40]. The Darwin implemen-tation uses Q = 19 (i.e., the number of MFCCs coefficientemployed) and P = 20. We fix the number of componentsto 20 because the larger the number of components, where acomponent represents a single gaussian within the mixture,the more accurate the model is. At the same time as thenumber of components increases, the computing cost for themodel increases. For our implementation, we experimentallyverify that 20 components provide the best tradeoff betweencomputation cost and classification accuracy.

Speaker Inference. In Darwin, speaker inference op-erates in a collaborative and distributed manner. The in-ference is collaborative because all the mobile phones con-tribute to the final results by communicating the confidencelevel associated with certain audio with other mobile phonesin the neighborhood using the short-range radio, such as,Bluetooth or WiFi. The inference is distributed in the sensethat the final inference result is derived locally on each in-dividual mobile phone without relying on any particular ex-ternal entity. We show how collaborative inference booststhe performance of the speaker recognition application.

2.4 Classifier Evolution

2.4.1 Initial Training

The initial training phase is intended to be short so thatthe application can be used immediately by people withoutrequiring a prolonged initialization phase. Clearly, a shorttraining data set implies a less accurate classification model.For this reason off-the-shelf speaker recognition applications[47][18][23] use large number of samples typically several tensof seconds of speakers’ voices in order to achieve accept-able recognition accuracy. In our system, the initial speakermodel is just the starting point and the model is used torecruit new training data and evolve the model on-the-gowithout additional user intervention. For applications otherthan speaker recognition, the initial model of the event tobe sensed is provided by the system during the deploymentphase. The evolution algorithm is designed to be applied todifferent sensing modalities other than audio, i.e., air qual-ity sensing, etc. The initial training phase consists of takinga sample or seed of the sensed data and using half of thedata to build the model. The remaining half for building abaseline likelihood (BL) as shown in Figure 3(a).

speaker is using n phones we would have n different GMMmodels, one per phone for the same speaker.

8

During the initial training phase, a person is asked to talkinto her phone until a voicing audio stream of 30 seconds iscollected by the mobile phone. The first 15 seconds of thetraining set are used for training purposes and the remain-ing 15 seconds to set a baseline for the classifier evolutiontechnique, as discussed in Section 4. The raw audio streamreceived from the microphone is first passed through the si-lence suppression and voicing filter; then, the training andbaseline audio are both fed into the MFCCs extractor. TheMFCCstrain computed from the training audio form the fea-ture vector that is used to build a GMM of the speaker. Thebaseline audio is used to extract MFCCsbase and to deter-mine the BL that is used to recruit new audio samples, asdiscussed in Section 2.4.2. The baseline is computed onlyfor the model that the phone trains, which, in the case ofspeaker recognition, is the model of the phone owner voice.

2.4.2 Evolution

An important challenge in training classifiers is to obtainsufficient amount of labeled data for supervised training.Data labeling is tedious and expensive and real-time label-ing is seldom practical since people are averse to being in-terrupted in order to label data. In essence, labeling datadoes not scale.

Therefore, machine learning algorithms running on mobilephones can not fully rely on supervised learning strategies.In contrast, a fully unsupervised learning process withoutany human intervention can often latch onto idiosyncrasiesof the data and is not guaranteed to model the classes theuser might care about. Darwin takes a different approachby using a classifier evolution algorithm on mobile phoneswhich uses a simple semi-supervised learning strategy [54].We show that such an approach can be effective in boostingthe performance of the speaker recognition system. Dar-win’s classifier evolution revolves around the fact that mo-bile phones create different classification models for differ-ent environments if existing classification models do not fitthose environments. When the speaker recognition appli-cation bootstraps, the speaker’s voice is sampled in a quietenvironment to build the initial model for the owner of thephone (15 seconds of voicing audio). During runtime, if thelikelihood of the incoming audio stream is much lower thanany of the baseline likelihoods corresponding to the differ-ent models on the phone, then a new classification modelis evolved. The classifier evolution algorithm comprises thefollowing steps: (i) obtain the likelihood for a 96 ms in-coming audio chunk; (ii) compare the maximum likelihoodestimator (MLE) result with BL for models already existingon the phone; (iii) recruit the new audio as part of the train-ing data, if necessary; (iv) re-train the classification model;and finally (v) compute the new BL, which is the likelihoodof the original baseline plus the newly recruited data. Theclassifier evolution algorithm is detailed in Algorithm 1.

Comparison with BL. If L is the number of classifica-tion models for different environments, the Likelihoodnew iscompared with BLi, ∀ i, i=1..L. At the beginning, the phonecontains only BL1, which is derived from the initial trainingdata set. If Likelihoodnew falls between a +/- 3% intervalaround one of the BLi, then that BLi is used as the refer-ence BL for the environment where the speaker is situated.We determine the 3% threshold experimentally in differentsettings and this value is the one that best generalizes todifferent environments and with the best performance. Be-

Algorithm 1 Pseudo-code of classifier evolution al-gorithm running on node i.

NumEnvType ← number of environments for which amodel is defined.

while there is a new incoming audio chunk do

Compute Likelihoodnew

Compare Likelihoodnew with BLi, ∀ i, i=1..L

if ∃ j ∈ {1..NumEnvType} s.t. Likelihoodnew is within+/-3% BLj then

{The phone has a baseline for the environment}BLcomp ← BLj

environmentType ← j

else

{The phone does not have a baseline for the environ-ment}BLcomp ← BL1

environmentType ← 1

end if

LikelihoodDifference ← | Likelihoodnew - BLcomp |Add LikelihoodDifference to vecDiff

meanDiff ← mean(vecDiff)

stdDevDiff ← stddev(vecDiff)

if ((stdDevDiff ≤ (previousStdDev − 5%) OR(stdDevDiff ≥ (previousStdDev + 5%))) AND(Likelihoodnew ≥ BLcomp - mean(vecDiff)) then

Recruit new training data for environmentType

Calculate new model for environmentType

Add data to baseline audio of environmentTypemodel

Calculate new BLj

end if

end while

cause the threshold is derived for a set of environments wherepeople spend most of their time (e.g, quiet indoors, outdooralong a sidewalk of a busy street, and in a noisy restaurant)we are confident that the threshold would extend to othersimilar environments.

After having determined which model to use, the absolutevalue of the difference between Likelihoodnew and BLi isrecorded. The vector vecDiff holds these differences foreach new audio chunk. The mean and standard deviation ofthe elements in vecDiff are then calculated.

Data Recruiting and Model Re-Training. New datais recruited for re-training if the following two conditionshold: i) the standard deviation of the elements in vecD-iff, when the new difference is added, oscillates outside a+/-5% boundary around the standard deviation calculatedbefore the new difference is added to vecDiff ; and ii) theLikelihoodnew is greater or equal than (BLi - mean{vecDiff}).The first condition ensures we are recruiting only voice datathat differs from the current model and the second normal-izes the likelihood with respect to the mean. As new datais added to the training set, the inference model i is re-computed and a new BLi is calculated by adding the audiochunk to the baseline audio of model i. The recruiting stops

9

when the likelihood stabilizes inside the +/- 5% boundarybecause convergence is reached.

As a product of the classifier evolution algorithm, differentclassification models are automatically created for differentenvironments and locally stored on the phone. We provethe effectiveness of our algorithm by running the speakerrecognition application and Darwin system in three differ-ent environments, that reflect common places people findthemselves. In our experiments, three models are automat-ically created. In our speaker recognition application eachmodel uses 63KB of storage space and approximately 500KBof storage for the training data for each model, which in-cludes the initial training set plus the fraction of data re-cruited as the model evolves. The overall storage require-ment for a single speaker with three different environmentsis ∼1.8MB. This amount of storage is reasonable for mod-ern mobile phones because they have several gigabytes ofpersistent memory and up to 1GB or more of applicationmemory. Under more limited memory constraints the num-ber of classification models for the same speaker and for theother speakers could be set to a maximum and the mod-els arranged according to a circular buffer policy. As thenumber of models in the buffer exceeds the buffer capacity,the oldest models could be removed to accommodate newmodels.

2.5 Model Pooling

Model pooling is based on the simple premise of sharingclassification models that have already been built and opti-mized by other mobile phones. It is a simple but effectiveway to increase classification timeliness by minimizing theinference latency. Model pooling boosts classification scala-bility, accuracy, and speed by providing a mechanism for fastexploitation of existing models by mobile phones rather thanbuilding classifiers from scratch themselves. Model poolingboosts classifiers scalability because models are not requiredto be trained for multiple events, but just for those events themobile phone is most likely to be exposed to. For example, inthe speaker recognition case the percentage of time that thephone owner’s voice is captured, because of conversationswith nearby people or phone calls, is greater than the per-centage of time any other speaker is captured. In this case, itwould be possible to accurately re-train and refine over time,using the evolution algorithm, only the phone owner’s voicemodel rather than everybody else’s voice model. Mobilephones can pool models – voice models in the case of speakerrecognition – from other co-located phones for events thatthey do not have. These models are readily available, usable,and require no extra training steps. Model pooling does notnecessarily occur between co-located phones. Models can bepooled from backend servers too. Assume a particular phonebuilds a sophisticated audio inference or pollution model ofan area [22, 10, 20]. Phones can geo-tag models and uploadthem to backend servers. From this point on other phonesmoving into these geo-tagged areas do not have to wait togenerate their own models. They simply download modelsfrom a server if available and start making inferences.

In order to formalize model pooling, let P be a mobilephone, Mp be the model derived and optimized for a certainevent by P , and Mi be the classification models individuallyderived and optimized by N other mobile phones Pi wherei = 1 .. N . If K of these mobile phones are co-located withP , following the pooling phase P would have its own model

Mp and models Mi, where i ∈ {1 .. K}. Each model Mi

is stored locally by P in order to be used again without theneed to pool them by phone Pi, unless the model has beenevolved by Pi. In that case, node Pi would announce to itsneighbors that a new model has been computed and is readyto be pooled. As a result of the model pooling algorithm,all phones receive the classification models of all the otherphones at the end of the model pooling phase.

After node P has pooled a classification model Mi =<µi ,Σi ,wi> from node Pi, node P will replace Pi’s clas-sification model Mi only if Pi announces a new classifica-tion model Mnew

i . Node Pi determines that it is time toannounce the new classification model Mnew

i when the evo-lution of the model is complete.

2.6 Collaborative Inference

Collaborative inference is designed to boost the classifi-cation performance by taking into account the classificationresults of multiple phones“observing”the same event insteadof relying on only individual classification results. The ideabehind collaborative inference is as follows: given that anevent could be sensed by multiple, distributed but co-locatedmobile phones, Darwin leverages the classification results ofindividual phones to achieve better accuracy. The hypoth-esis is that we can take advantage of the multiple sensingresources around an event and exploit their spatial distribu-tion and context diversity. In the case of speaker recognition,when a person is talking with a group of people, the phonesof the other people would pick up the speaker’s voice withdifferent characteristics. Some phones could be in pockets,others on a clip belt or closer to a source of event than theother phones. In this case, if the application relies only oneach individual phone’s classification, the speaker classifica-tion accuracy would be poor when the phone sensing contextis not good (e.g., in pocket or near a noise source) in relationto the specific event. The same reasoning holds for pollutionmeasurements using mobile phones.

Following the pooling phase, each phone Pi contains theother phones classification models Mi, where i ∈ {1 .. K},and K is the number of neighbors. Each phone Pi runsthe inference algorithm in parallel using the common set ofmodels. The fact that the event is classified by multiplephones using common classification models and that thesephones may be exposed to different contexts (some of whichare better than others) can lead to higher accuracy whenthe results are combined. The collaborative inference phasebreaks down into three distinct steps: i) local inference oper-ated by each individual phone; ii) propagation of the resultof the local inference to the neighboring phones; and iii) fi-nal inference based on the neighboring mobile phones localinference results.

2.6.1 Local Inference

During the local inference phase each node individuallyoperates inference on the sensed event using its own infer-ence model of the event and the inference models pooledfrom other phones. The goal is to locally derive the con-fidence level of the inferred event in order to be able tocommunicate this confidence level to neighboring phones.By having the confidence level of each phone sampling theevent, all the phones are able to run the collaborative in-ference step to compute the final inference result. In or-der to operate collaborative inference, the phones must be

10

time-synchronized because they need to perform inferenceon the same event at the same time. We rely on the factthat mobile phones support time synchronization throughthe cellular infrastructure. We measure a time synchroniza-tion error between four iPhones synchronized through thecellular network of 500 ms. If the error is larger than 500 mswe use a loose synchronization approach. One of the phones(randomly picked) sends a broadcast message which, whenreceived by all the neighbors at the same time, triggers thesampling phase. After having received this broadcast mes-sage phones are synchronized.

Following the audio signal sampling and filtering, the streamis divided in 96 ms long chunks4. MFCCs are extracted fromeach chunk. At this point, a maximum likelihood estimationalgorithm is run in order to verify which of the models bestfits the audio chunk. To avoid having the maximum likeli-hood estimator running through too many pooled models,which could potentially be costly in terms of computationfor a mobile phone, the estimator only uses the models ofthe phones detected in the neighborhood. Neighbor detec-tion is performed using short-range radio technology, suchas, Bluetooth and WiFi.

In order to increase the classification accuracy we feedthe maximum likelihood estimation result into the window-ing block. After the windowing step the local classificationresult is derived. If the maximum likelihood estimator re-turns that the i−th audio chunk belongs to event E, theevent E is deemed to be true if and only if the following twoconditions hold: (i) event E, i.e., a certain speaker speaking,is detected at least once in a window comprising the previ-ous two audio chunks; and (ii) the classification accuracyconfidence is at least 50% larger than the confidence for anyother events. The reason for the first condition is to guaran-tee that a voice sample is included – since audio chunks are96 ms long, if a speaker is detected at least twice in a windowof time of duration 96 ms × 3, then we have better confi-dence that we are really capturing that speaker. A largerwindow size would increase the classification latency and af-ter experimentation we determine that a window size of 3best suits our system, which requires near-real time classifi-cation results. The reason for the second condition, havingdetermined the 50% threshold experimentally, is to dampenthe impact of false positives. The pseudo-code of the localinference algorithms is shown in Algorithm 2. The locally in-ferred speaker ID is associated with the classification modelthat best fits the audio chunk following the windowing pol-icy. Because a phone computes the inference confidence foreach of the K models as reported in Algorithm 2, the resultof the local inference takes the form of the following vec-tor LIj

1..s = {confidenceSpeaker1, confidenceSpeaker2, .., confidenceSpeakerk}, where the subscript notation 1..sindicates that the vector contains the inference results forspeakers 1 to s and the superscript index j indicates that thevector is generated by node j. Consider for example, thatthree nodes (N1, N2, and N3) are co-located and runningthe speaker recognition application. If S1, S2, and S3 arethe speakers associated with nodes N1, N2, and N3 respec-tively, and assuming S1 is actually speaking, then an outputfor the LI vectors could, for example, be: LI1

1,2,3 = {0.65,

4We use 96 ms for each audio chunks to make it a multipleof the MFCC binning size of 32 ms. This multiple makesthe duration of the audio chunk small enough to maximizethe likelihood of capturing just one speaker at a time.

0.15, 0.2}, LI21,2,3 = {0.4, 0.5, 0.1}, and LI3

1,2,3 = {0.7, 0.2,0.1}. The reason of N2 expressing lower confidence aboutspeaker 1 could be caused by the fact that mobile phone N2

may not have a good sensing context.

Algorithm 2 Pseudo-code of the local inference al-gorithm running on node i using K pooled models.The number K is determined by the number of de-tected neighbors. The locally inferred speaker IDcorresponds to the index of the model best fitted bythe audio chunk following the windowing policy.

while there is a new incoming audio chunk do

{Go through all the classification models:}for j = 1 to K do

arrayLE[j] ← likelihood estimation for audio chunkend for

Retrieve the max likelihood from arrayLE and the as-sociated index indexMA.Retrieve the second larger confidence from arrayLE andthe associated indexSMA.

{Windowing policy:}if (arrayLE[indexMA] − arrayLE[indexSMA] ≥0.5) AND ((indexMA == previousMAX) OR(indexMA == previousPreviousMAX)) then

locallyInferredSpeaker ← indexMApreviousPreviousMAX ← previousMAXpreviousMAX ← indexMA

end if

end while

2.6.2 Local Inference Propagation

Following the local inference step, each mobile phone ibroadcasts the result of its local inference (i.e., the vectorLIi) in order to give neighboring phones the ability to moveto the final step, i.e., the final inference. A time stamp isassociated with vectors LIi in order to align local inferenceresults in time. A weight is also associated with vectors LIi.The weight is an indication of the quality of the local infer-ence. For example, assuming the mobile phone determinesits context (e.g., in or out of the pocket) and is running anaudio or pollution sensing application [20], then the valueof the weight is small or large when the phone is in or outof the pocket, respectively. Therefore, a LIi vector with asmall weight indicates that the mobile phone i is not in asuitable sensing context and has less influence on the finalinference phase.

The phones broadcast their local inference results at therate the local inference is computed. The local inference iscomputed right after an audio sample is collected. Conse-quently, if the phone polls the microphone every A number ofseconds, the local inference is computed and broadcasted ev-ery A seconds as well. For example, a local inference broad-cast rate of 1Hz implies receiving the local inference results,hence, computing the final inference, every second. Clearly,the higher the broadcast rate the faster the final inference(see Section 2.6.3). We do not propose any specific localinference broadcast rate, since this depends on the applica-tion requirements. Rather, we show the cost of transmitting

11

a single confidence level from which the total cost for eachspecific transmission rate is derived (see Section 4.4).

2.6.3 Final Inference

The goal of the final inference phase, which follows the lo-cal inference propagation, is to compensate event misclassifi-cation errors of each individual mobile phone achieving bet-ter classification confidence and accuracy of sensed events.Mobility and context affect the local inference result, some-times such that an event could be misclassified. For thefinal inference, Darwin combines the local inference resultsderived by individual mobile phones that are spatially dis-tributed in the area of the target event. The technique usedto perform the final inference is to find the inferred speaker(IF) that maximizes the joint probability as in Equation 1.

IF = arg maxj,j∈{1..S}

{Prob(s1j),Prob(s2j), ..,Prob(sKj )} (1)

where K is the number of co-located phones that havebeen contributing with their local inference, S is the num-ber of speakers, and Prob(si

j) is the probability that speakerj is detected by phone i. This operation is facilitated by thefact that the property of independence between the differ-ent LIj

1..S vectors, ∀j, j = 1..K, is verified since each node(i.e., phone) Ni, i = 1..K, performs independent event sens-ing. Given the property of independence, we can re-writeEquation 1 as:

IF = arg maxj,j∈{1..S}

{KY

i=1

LIi1,

KY

i=1

LIi2, ..,

KY

i=1

LIiS} (2)

Since each node Ni has the vectors LIk1..S (where k ∈

{1,..,K}) after the local inference propagation phase, eachnode Ni can compute Equation 2 to produce the final in-ference. In order to assure that Equation 2 is calculatedusing the local inference results of the same event, Equation2 is computed only over the local inference results that arealigned in time. For each computation of Equation 2 onlythe local inference results which differ in time for at mostδ ms are considered. In our implementation, we set δ =150 ms, which we determine experimentally to be the bestvalue. The larger δ, the bigger is the likelihood that thelocal inference results refer to different events. The smallerδ, the higher is the likelihood to capture just one event, butthe closer we get to the phones time synchronization error.Because time stamps are associated with LI vectors (seeSection 2.6.2) it is possible for the phone to determine if anevent is sampled at the same time.

In order to provide a more robust system against falsepositives, a windowing approach is adopted where a speakeris deemed to be speaking only if they are detected for at leastone more time in the past two iterations of Equations 2. Thispolicy, similar to the windowing policy introduced for thelocal inference in Section 2.6.1, provides experimentally thebest tradeoff between classification delay and classificationaccuracy.

3. PRIVACY AND TRUST

Security, privacy, and trust raise considerable challengesin the area of mobile phone sensing. While we do not presentsolutions to these challenges, those solutions are critical to

the success of the research discussed in this paper. Dar-win incorporates a number of design decisions that are stepstowards considering these challenges. First, the raw sen-sor data never leaves the mobile phone nor is it stored onthe phone - we only store models and features computedfrom the raw sensor data. However, features and modelsthemselves are sensitive data that needs to be consideredappropriately and therefore protected. In the case of thespeaker recognition application, the content of a conversa-tion is never disclosed, nor is any raw audio data ever com-municated between phones. The data exchanged betweenphones consists of classification confidence values and eventmodels. Next, mobile phone users always have the ability toopt in or out of Darwin, hence, no model pooling and col-laborative inference would take place unless the users makesuch a determination.

To meet privacy, security, and trust requirements Darwinphones should: i) run on trusted devices; ii) subscribe toa trusted system; and iii) run a trusted application thatis either pre-installed on the phone or downloaded from atrusted third party (e.g., Apple App Store, Nokia Ovi Store,or Android Market). Any phone-to-phone interaction (e.g.,pooling and collaborative inference) should be regulated byoff-the-shelf authentication and authorization mechanismsthat prevent the injection of malicious code or intentionallydistorted inference results from adversaries.

4. SYSTEM PERFORMANCE

In what follows, we first discuss the implementation ofDarwin on the Nokia N97 and Apple iPhone and then presentthe detailed evaluation results of the Darwin system sup-porting the speaker recognition application.

4.1 Phone Implementation

Darwin is implemented on the Nokia N97 using C++, KissFFT [4] for fast fourier transform (FFT) calculations, andQT [7], which is a wrapper around C++ for the graphicaluser interface. On the Apple iPhone we use C++ and theFFTW fast fourier transform library [3]. The necessary algo-rithms, i.e., GMM training, the probability density function,and MFCC extraction are ported to the N97 and iPhonefrom existing Matlab code that we verified to work correctly.We plan to make this toolkit available in the future as anopen source project. The availability of this toolkit on aphone is a considerable resource for building more powerfulclassifiers. The backend server responsible to run the modeltraining and re-training for evolution is implemented on aUnix machine using C and standard socket programming forcommunications. A UDP multicast client is implementedto allow local inference results propagation whereas an ad-hoc lightweight reliable transport protocol has been built tosend feature vectors to the backend for model training, tosend trained models to the phones, and to exchange modelsduring the pooling phase. Because we target heterogeneousscenarios, an audio sampling rate of 8KHz is used in orderto run Darwin on the iPhone 3G, which can drive the micro-phone up to 48KHz sampling, the iPhone 2G and the NokiaN97, which only support 8KHz audio sampling rate.

4.2 Experimental Results

We evaluate the Darwin system using a mixture of fiveN97 and iPhones used by eight people over a period of twoweeks generating several hours of recorded audio containing

12

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0 50 100 150 200 250 300 350

Acc

urac

y

Time (seconds)

Speaker 1Speaker 2Speaker 3

Figure 4: Accuracy, without evolution, for threespeakers when walking along a busy road withoutclassifier evolution and having trained the classifica-tion models for indoors only.

0

1

2

3

4

5

6

7

8

0 50 100 150 200 250 300 350

Num

ber o

f Rec

ruits

Time (seconds)

Figure 5: Evolution sessions count over time in theindoor scenario for speaker 8.

speech portions. In order to evaluate the system againstground truth data, the audio data is manually labeled byextracting the voicing chunks of each of the eight speakers.The audio is recorded in different locations under differingconditions such as a quiet indoor environment, walking ona sidewalk along a busy and noisy street, and in a noisyrestaurant. This provides a good basis to validate Darwinunder very different operating conditions. We only presenta subset of the results from our experiment due to spacelimitations.

4.2.1 The Need for Classifier Evolution

We conduct a simple experiment to show the need for clas-sifier evolution on mobile phones. Three people walk alonga sidewalk of a busy road and engage in conversation. Thespeaker recognition application without the Darwin compo-nents runs on each of the phones carried by the people; thatis, no classifier evolution, model pooling, and collaborativeinference algorithms are enabled for the experiment. Thevoice classifier for each person is trained in a quiet indoorenvironment. We quantify the performance of a classifica-tion model trained indoors when operating in a noisy out-door setting. Figure 4 shows the classification accuracy [53]for mobile phones 1, 2, and 3 for speaker 1, 2, and 3, respec-tively. It is evident from the plot that the accuracy is quitelow because the maximum speaker classification accuracy forspeaker 1 and 2 is 63% and 61%, respectively, and only 52%for speaker 3. The poor performance is because the classifi-cation model trained indoor performs poorly outdoors. Thishighlights the challenge presented when sensing in differentenvironments. Building audio filters capable of separating

0

20

40

60

80

100

120

0 50 100 150 200 250 300 350 400

Siz

e (K

B)

Time (seconds)

Speaker 1Speaker 2Speaker 4

Figure 6: Size of the data set recruited during theevolution phase in the restaurant scenario.

0 1 2 3 4 5 6 7 8 9

10

0 200 400 600 800 1000 1200 1400

Tim

e (s

ec)

Training data size (KB)

Figure 7: Server training computation time as afunction of the training data size. The server hasa 2.4GHz cpu and 4GB of RAM.

voice from other types of noise is challenging and would notlikely scale given the large pool of possible sounds that maybe encountered by a mobile phone on the street or in anyother environments. This result motivates the need for de-signing efficient classifier evolution techniques that operatetransparently to the person carrying the phone in order toevolve classification models according to the scenarios wherepeople find themselves.

Let us take a look now at the evolution algorithm per-formance. Figure 5 shows the distribution for the durationof the experiment of the number of audio chunks recruitedby the classifier evolution algorithm for speaker 8 in the in-door case. Similar results are observed for other speakersnot shown in the results. As expected, a larger number ofchunks are recruited during the first phase, when the ap-plication is run after the initial training, than towards theend, when the model has already been refined and little orno model evolution is required.

Figure 8 shows the accuracy improvement as the amountof data sent from the phone to the backend for re-traininggrows. This result refers to a different outdoor environmentthan the one in Figure 4. This second outdoor scenario isnoisier than the first one, causing the initial accuracy be-fore evolution to be lower than the one reported in Figure 4.Clearly, the larger the training set capturing the character-istics of the new environment the better the performance.Figure 6 shows the amount of audio data recruited by theevolution algorithm for the duration of the restaurant ex-periment. It is interesting to see that after a few minutesof conversations the models of the three speakers diminishtheir training data recruitment and model evolution eventu-ally stops (as happening for speaker 1 model). This confirms

13

0

0.1

0.2

0.3

0.4

0.5

0.6

0 100 200 300 400 500 600 700 800 0

6992

13984

20976

27968

34960

41952

48944

Acc

urac

y

Byt

es s

ent t

o se

rver

Time (sec)

AccuracyBytes sent to server

Figure 8: Classifier accuracy and the amount ofneeded training data in an outdoor scenario.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 300 350

Acc

urac

y/P

reci

sion

/Rec

all

Time (sec)

accuracyprecision

recall

Figure 9: Mean recall, precision, and accuracy in in-door quiet environment with collaborative inferencefor the eight speakers.

the behavior observed for Darwin in the quiet indoor envi-ronment. The only difference is in the amount of recruiteddata between the restaurant and quiet scenario. In fact,in the former case the amount of data recruited ranges be-tween 78KB and 100KB, whereas in the indoor case it isbetween 16KB and 70KB. This is in line with the fact thatless data is needed to evolve an indoor model than a“restau-rant” model (i.e., a very different environment from a quietplace) since initial training is performed in a quiet place.Figure 7 reports the server training algorithm running timeas the training data set grows. The measurement refers tothe outdoor case and levels off when the accuracy (shownin Figure 8) reaches the maximum. Right before the endof evolution the server takes about 10 seconds to train theGMM model using a 1.3MB data set.

4.2.2 Experimental Scenario One: Quiet Indoor En-

vironment

We first show the performance of the speaker recognitionapplication analyzing data collected from 5 different phonesconcurrently running the Darwin system in a meeting set-ting in an office environment where 8 people are involvedin a conversation. The phones are located at different dis-tances from people in the meeting, some on the table andsome in people’s pockets. The aim of the experiment is tostudy the impact of different phone sensing context showinghow a low classification accuracy due to adverse context iscompensated by Darwin. In Figure 11 we show the classifi-cation precision [53] for speaker 8 calculated by speaker 8’sphone using the basic speaker recognition application whenspeaker 8 is talking. Figure 11 shows that the precision of

0.5 0.55 0.6

0.65 0.7

0.75 0.8

0.85 0.9

0.95 1

1 2 3 4 5 6 7

Nor

mal

ized

diff

eren

ce

Pair of mobile phones

basicthreshold

windowing

Figure 10: Normalized true positive-false positivedifference between speaker 8 and all the other speak-ers when speaker 8 is speaking. The closer the nor-malized difference to 1, the larger is the true posi-tives compared to false positives.

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300 350P

reci

sion

Time (seconds)

Speaker 1Speaker 2Speaker 3Speaker 4Speaker 5Speaker 6Speaker 7Speaker 8

Figure 11: Precision for speaker 8 basic inferencewhen speaker 8 is speaking in an indoor quiet set-ting.

the basic speaker recognition application is below 50%. Thisresult indicates that speaker recognition using an individualmobile phone is challenging. The reason is that the phonecould have poor sensing context for example in the pocket(as for part of the conversation of speaker 8) or affected byother factors such as noise mixed with voice.

Figure 10 shows the benefit of applying the 50% thresh-olding technique and windowing policy (described in Sec-tion 2.6) to the basic speaker recognition classification al-gorithm. The 50% thresholding technique makes sure thatthere is a 50% difference between the confidence of the in-ferred speaker and every other speaker, whereas the win-dowing policy reduces the effect of false positives. Figure10 shows the difference between the true positive and falsepositives counts normalized by the true positive count forspeaker 8 speaking. The closer the normalized differenceto 1, the larger is the number of true positives versus thenumber of false positives. Figure 10 shows the combinedpositive effect of the 50% thresholding and windowing tech-niques, which makes the normalized difference larger com-pared to the basic technique. This is a first step toward thefinal inference, which is part of the collaborative inferencephase of Darwin; however, it is not enough to achieve higherclassification accuracy. In fact, when the Darwin system isactivated a further performance boost is registered. Figure9 shows results for the mean recall [53], precision, and accu-racy results for the eight speakers classification when Darwinis running. It is evident from the plot that Darwin boosts

14

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50 60 70 80 90

Cla

ssifi

catio

n D

iffer

ence

Time (seconds)

Diff 1Diff 2Diff 3Diff 4Diff 5Diff 6Diff 7

Figure 12: Classification difference betweenspeaker 8 and the other speakers without Dar-win.

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50 60 70 80 90

Cla

ssifi

catio

n D

iffer

ence

Time (seconds)

Diff 1Diff 2Diff 3Diff 4Diff 5Diff 6Diff 7

Figure 13: Classification difference betweenspeaker 8 and the other speakers with Darwin.

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300 350 400

Pre

cisi

on

Time (seconds)

Speaker 1Speaker 2Speaker 3Speaker 4Speaker 5Speaker 6Speaker 7Speaker 8

(a) Precision for speaker 4 calcu-lated by the speaker’s mobile phonewithout collaborative inference.

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300 350 400

Pre

cisi

on

Time (seconds)

Speaker 1Speaker 2Speaker 3Speaker 4Speaker 5Speaker 6Speaker 7Speaker 8

(b) Precision for speaker 4 calcu-lated by mobile phone 1 withoutcollaborative inference.

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300 350 400

Pre

cisi

on

Time (seconds)

Speaker 1Speaker 2Speaker 3Speaker 4Speaker 5Speaker 6Speaker 7Speaker 8

(c) Precision for speaker 4 calcu-lated by mobile phone 2 withoutcollaborative inference.

Figure 14: Precision for speaker 4 on different phones in a noisy restaurant environment without collaborativeinference.

the classification performance of the speakers identification.The boost is due to collaborative inference, which leveragesthe other co-located mobile phones in order to achieve betterclassification accuracy.

The other interesting result from Figure 9 is the positiveimpact of Darwin’s classifier evolution. The accuracy, pre-cision, and recall increase over time at the beginning of theexperiment as new audio is recruited and the classificationmodels re-trained taking into account the new audio. Theperformance for accuracy, precision, and recall levels off atthe point where the classifier evolution algorithms does notbenefit from more data, as discussed in Section 2.4.

The benefit of model evolution, model pooling, and col-laborative inference can be also seen in the results shown inFigures 12 and 13. If we indicate with TP and FP, respec-tively, the true and false positives when speaker 8 is speak-ing, the y-axis of the plots reports the quantity (TP-FP)/TPover time. A FP count is maintained for each speaker, thusin Figures 12 and 13 eight curves are shown. In one caseDarwin is disabled (Figure 12), in the other case Darwinis enabled (Figure 13). When (TP-FP)/TP is close to 1 itmeans that the number of true positives dominates the num-ber of false positives. In contrast, if (TP-FP)/TP is close to0 we have that the number of false positives approximatesthe number of true positives. We can see that the differencebetween speaker 8’s true positives and any other speakers’false positives is larger when Darwin in running (as shownin Figure 13) than when it is not (see Figure 12). This isanother indication of how Darwin improves the classificationresult for a given speaker.

4.2.3 Experimental Scenario Two: Noisy Indoor

Restaurant

In the next experimental scenario, we evaluate Darwinwhen the speaker recognition application is running on fivephones while five people are having a meal in a noisy restau-rant. This contrasts the first scenario of a quiet indoor set-ting. Three out of five people are engaged in conversation.Two of the five phones are placed on the table, the otherphones are in the pants pockets for the entire duration ofthe experiment. In Figure 14, we show the classification pre-cision of speaker 4 from three of the mobile phones locatedaround the table people are sitting at; note, we observe simi-lar trends for the other phones. Figure 14(a) is the precisioncomputed by speaker 4’s phone, which is the closest phoneto speaker 4 (it is carried by speaker 4) for when speaker 4 istalking. The reason we select speaker 4 for the evaluation isthat speaker 4 is the closest person to a nearby table whereanother group of people is having a loud conversation. Herewe show the benefit of the Darwin system for the phone ofa speaker who is positioned in a non optimal context, i.e.,close to a noise source.

Figures 14(b) and 14(c) refer to the precision calculatedby two other phones, which we call phone 1 and 2, locatedat the opposite side of the table where speaker 4 is sitting.Figures 14(b) and 14(c) show on average higher classifica-tion precision on phones 1 and 2 when speaker 4 is talk-ing than when the classification is performed by speaker 4’sphone reported in Figure 14(a). This is because phones 1and 2 are more distant from the source of noise and con-sequently they are able to capture higher quality speaker

15

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 50 100 150 200 250 300 350 400

Rec

all

Time (seconds)

Speaker 1Speaker 2Speaker 4

(a) Recall.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

0 50 100 150 200 250 300 350 400

Pre

cisi

on

Time (seconds)

Speaker 1Speaker 2Speaker 4

(b) Precision.

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300 350 400

Acc

urac

y

Time (seconds)

Speaker 1Speaker 2Speaker 4

(c) Accuracy.

Figure 15: Recall, precision, and accuracy in a noisy restaurant with Darwin for three speakers.

4’s voice audio than speaker 4’s phone itself. The importantpoint is that Darwin boosts the classification performance ofspeaker 4 by leveraging other surrounding phones experienc-ing better sensing contexts; this is shown in Figures 15(a),15(b), and 15(c) which report, respectively, the recall, pre-cision, and accuracy of the three speakers in the restaurantexperiment, including speaker 4, over time. From Figure15(c) it can be seen that speaker 4’s accuracy is low (∼65%) at the beginning of the experiment but starts increas-ing as the experiment proceeds. The classifier evolves andthe surrounding mobile phones participate in collaborativeinference. At the end of the experiment, speaker 4’s classi-fication accuracy reaches ∼80%. The fact that speaker 4’srecall and precision present low values for the duration of theexperiment (as shown in Figures 15(a), 15(b), respectively)confirms that speaker 4 voice is impacted most of the timeby loud conversation from the next-table.

4.2.4 Experimental Scenario Three: Walking Out-

doors Along a Sidewalk in a Town

The final scenario we study is an outdoor environmentwhere five people walk along a sidewalk and three of themare talking. This contrasts the first two scenarios. In thisexperiment, five people carry phones either clipped to theirbelt or inside their pockets. As in the restaurant scenario,the amount of audio data recruited by Darwin to evolve theclassifier is larger than the indoor evolution case and rangesbetween 90KB and 110KB of data. The performance boostusing the Darwin system can be observed in Figure 16 wherethe speaker recognition classification accuracy increases to80-90%. The greatest improvement is observed by speaker1 whose phone is clipped to their belt. This mobile phoneis exposed to environmental conditions such as wind andpassing cars making the audio data noisy and voice hard topick up. However, we note that some of the other phonesexperience better sensing context and by relying on thesephones Darwin boosts the final classification accuracy forspeaker 1.

4.3 Impact of the Number of Mobile Phones

In this experiment, we study how the Darwin system’sperformance changes as the number of mobile phone partic-ipating in model pooling and collaborative inference varies.This experiment is also conducted in the same noisy restau-rant discussed in scenario two, which represents a challeng-ing sensing environment. The experiment consists of threepeople speaking and five phones carried by five people posi-tioned around the table. Some phones are placed on the ta-ble and others remain in speaker’s pockets. The experimen-

tal scenario starts with only two of the five phones runningthe Darwin system. More nodes are subsequently added upto a maximum of five – all phones run the Darwin systemand are randomly positioned around the table. The classifi-cation accuracy for each of the three speakers as a functionof the number of phones running Darwin is shown in Fig-ure 17. As expected, the larger the number of co-locatedmobile phones running Darwin, the better the inference ac-curacy. The performance gain using collaborative inferencegrows with the number of phones according to the algorithmdiscussed in Section 2.6 and in particular to Equation 2.While two phones do not have sufficient “spatial” diversityto experience gain from the Darwin system, the classifica-tion accuracy is boosted when the three remaining phonesare added.

Speaker 3 experiences low accuracy due to the proximityof their phone to another group of people involved in a loudconversation. This is perceived as noise by the speaker iden-tification classifier and negatively impacts speaker 3’s voiceclassification more than speaker 1 and 2. Speaker 2 experi-ences low accuracy with three phones running Darwin dueto speaker 2’s voice characteristics. Speaker 2’s classificationmodel poorly classifies speaker 2’s voice when operating indi-vidually. This could be due to the fact that the initial train-ing audio is not recorded correctly or that the 20-component19-dimensional GMM for speaker 2 does not properly modelspeaker 2’s voice. In this case, a larger number of nodesis needed to perform accurate speaker 2 classification. TheDarwin system compensates not only errors due to differentsensing context but also for poor event classification model-ing. This is possible because multiple phones co-operate togenerate more accurate inference results. The confirmationthat speaker 2’s model is not accurate comes from the factthat speaker 2’s recognition with 3 phones performs poorlyin comparison with speaker 1 and 3 in a quiet setting (seeFigure 18), which is where the classifier should perform thebest given the initial indoor training stage.

We also determine that the reason for better inferenceaccuracy with 3 phones in the restaurant experiment forspeaker 3 is that the other two phones are closer to speaker3 than they are in the quiet indoor case. This offers betteraudio signal quality for the collaborative inference step.

In summary, the Darwin system boosts the classificationaccuracy when the sensing environment or context adverselyimpacts quality of inference, when the individual classifica-tion model yields poor accuracy given a person’s voice char-acteristics (as in the case of speaker 2 for our experiments),and when sensors or microphones have different hardwarecharacteristics [39].

16

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 50 100 150 200 250

Rec

all

Time (seconds)

Speaker 1Speaker 2Speaker 3

(a) Recall.

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250

Pre

cisi

on

Time (seconds)

Speaker 1Speaker 2Speaker 3

(b) Precision.

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

0 50 100 150 200 250

Acc

urac

y

Time (seconds)

Speaker 1Speaker 2Speaker 3

(c) Accuracy.

Figure 16: Recall, precision, and accuracy for three speakers walking on a sidewalk along a busy road.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

2 3 4 5

Acc

urac

y

Number of mobile phones

Speaker 1Speaker 2Speaker 3

Figure 17: Accuracyin a noisy restaurantwhen an increasingnumber of phonesparticipate to Darwin.

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2 3 4 5

Acc

urac

y

Number of mobile phones

Speaker 1Speaker 2Speaker 3

Figure 18: Accuracyin a quiet indoor set-ting when an increasingnumber of phones par-ticipate to Darwin.

4.4 Time and Energy Measurements

When proposing a complex but powerful classification ar-chitecture such as Darwin the natural question is: how doesthis system impact the performance of everyday mobile phones?While we have completed a first implementation of Darwinon the Nokia N97 and Apple iPhone we recognize that thereare challenges and future work to be done. In what follows,we present some time and energy measurements for Darwinrunning on the N97 (similar performance is observed for theiPhone). We believe that smart duty cycling is also a futurepart of our work which would improve the energy results pre-sented in this section. Averaged baseline measurement aretaken before each measurement in order to have a baselinereading, which we subtract from each measurement. Themeasurements are performed using the Nokia Energy Pro-filer tool [5] and repeated five times. The mean values arereported. The running time of each Darwin component for1 second of audio sampling is reported in Table 1. Themost computationally intensive routines are the local infer-ence (which involves the probability density function calcu-lation for eight speakers) and receiving the model from theserver. Figure 19 shows the power, CPU load, and memorymeasurements on the N97 when running the Darwin com-ponents. It can be seen that the local inference step takesthe largest amount of power (see Figure 19(a)) and CPUload (see Figure 19(b)). Substantial memory usage is mea-sured for MFCC extraction compared to the other compo-nents (see segment (C) of Figure 19(c)). This suggests thatthe MFCC extractor implementation requires optimization.Receiving a new model from the server and broadcasting itto neighbors during pooling also causes more power drainthan the other routines. However, evolution and pooling areoperations that occur rarely (i.e., evolution only once for a

Table 1: Average running time for processing 1 secaudio clip, sending, and transmitting data.

Routine Running Time (s)Silence suppression 0.003

Voicing 0.565MFCC extraction 1.4Local inference 3.67

TX MFCC to server 0.862RX model from server 4.7TX model to neighbors 1.91

TX local inference 0.127RX local inference 0.09

certain environment and pooling only once to send a modelto neighbors), consequently, pooling and evolution do notheavily account for resource usage and power consumption.Routines that instead occur periodically, such as audio sam-pling, voicing, MFCC extraction, etc., require less powereach time they are active.

Finally, Figure 20 shows the measured battery lifetimeand the inference responsiveness (defined as the inverse ofinference delay) as a function of the audio sampling inter-val and collecting 1 second of audio each time the micro-phone is polled. We obtain the shortest battery lifetime(∼27 hours) for a periodic sampling interval of 10 seconds(this sampling interval guarantees the highest inference re-sponsiveness). However, if smart duty-cycling techniquesare adopted [48], the phone could operate in a low sensingduty-cycle mode, e.g., with a sampling rate of 60 seconds,when Darwin is not running. As an event is detected, suchas voice in case of speaker recognition, Darwin could becomeactive in a high duty-cycle mode, e.g., using a 10 second sen-sor polling rate, for the duration of the event. As the eventdisappears the phone could go back to low duty-cycle modeand Darwin would stop working. This would guarantee highapplication responsiveness while maintaining several hoursof battery duration. More detailed analysis of resource con-sumption and the development of low-energy duty cyclingfor Darwin are important future work. We believe howeverthat new duty cycling techniques discussed in the literaturefor mobile phones [48, 32] could boost the phone’s batterylifetime of Darwin phones.

5. DEMO APPLICATIONS

Darwin can be used by other emerging mobile phone sens-ing applications in addition to the speaker recognition appli-cation discussed in this paper. In what follows, we discusshow a number of demo applications that use different sensing

17

(a) Power.

(b) CPU load.

(c) Memory.

Figure 19: Power, CPU load, and memory usageof the N97 when running Darwin components. Thecomponents have been made run sequentially andthe operations have been segmented as reported inthe following labels: (A) One sec audio sampling;(B) Silence suppression; (C) MFCC extraction; (D)Voicing; (E) Local inference; (F) MFCC transmis-sion to the server; (G) Model reception from theserver; (H) Model transmission to neighbors; (I)Local inference broadcast; (L) Local inference re-ception.

25 50 75

100 125 150 175 200 225 250 275 300 325

0 20 40 60 80 100 120

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2

Bat

tery

life

time

(hou

rs)

Res

pons

iven

ess

(1/s

ec)

Sampling time (sec)

Battery lifetimeResponsiveness

Figure 20: Battery lifetime Vs inference responsive-ness.

Figure 21: Virtual Square, an augmented reality ap-plication on the N97 with the support of Darwin.

modalities can be supported by Darwin. We discuss threedifferent demo applications.

5.1 Virtual Square Application

Darwin could support applications in the social spheresetting, where speaker identification, for example, could beused to augment the context of a person and their bud-dies and make the buddy proximity detection more accurate.Similarly, it could be used for reliable detection of meetings.A pure Bluetooth or WiFi based proximity detection systemmight not work accurately considering the large amount ofdevices a Bluetooth or WiFi scan could potentially returnin the area where the person is. We have developed VirtualSquare, an application that exploits augmented reality topresent a person’s sensing status information [29] includingwhom the person is in proximity/chatting with at a cer-tain moment and location. Our Virtual Square prototype isbuilt for the Nokia N97 writing a combination of QT code,for the body of the program, and Symbian routines to accesslow level functionalities for the collection of magnetometerreadings from the onboard magnetometer. GPS and mag-netometer sensors are used to geo-tag the person’s sensorcontext which is stored on a server and is retrievable bythe buddies who subscribe to the service. Users have fullcontrol of the application privacy settings and can opt-outfrom disclosing who they are chatting with at any time. Ascreenshot of the application is reported in Figure 21. It isclear how Virtual Square, by simply pointing the phone aswhen taking a picture and moving it around, is a powerfulmeans to see “through walls” and gather information aboutpeople and places in the very intuitive way of an augmentedreality interface. The old concept of square as a place wherepeople gather to chat and meet now becomes virtual, beingenabled by the smartphones sensors which allow to charac-terize people’s microcosmos.

5.2 Place Discovery Application

Darwin could support applications using different sensingmodalities; for example, place discovery applications basedon radio frequency (RF) activity from WiFi access points[22]. Darwin can be integrated with such an application inthe following way:

- Initial training and evolution: the classification modelin this case is the RF signature profile that characterizes acertain place. The dynamics of the radio characteristics andthe fact that access points are being added or removed incertain areas make the RF profile time varying. An RFprofile could be initially built by a mobile phone and then

18

evolved as the mobile phone visits the same place multipletimes in the future.

- Pooling: when a mobile phone visits an area for whichit does not have an RF profile it has two options: eitherbuild a profile, which requires time and introduces delay ininference, or, pool a profile for the area from a nearby mobilephone or backend server. Building an RF profile could takemore time than the duration of the stay in the area, whichmeans that the place might not be discovered. By pooling,the profile is immediately ready to be used in the inferencephase.

- Collaborative inference: if multiple mobile phonesare co-located in the area they can co-operate to performa more accurate place inference. Given that sensed RF ac-tivity could be slightly different from phone to phone, col-laborative inference could be used to determine what is themost likely discovered place by, for example, selecting theplace that is reported with highest probability by each ofthe mobile phones.

5.3 Friend Tagging Application

The idea of this application is to exploit face recognitionto tag friends on pictures. Namely, the application auto-matically associates a name to a person in the picture if theperson is recognized. Darwin could improve the applicationin the following manner:

- Initial training and evolution: the initial trainingstarts on each user’s mobile phone. The mobile phone de-rives a model for the person’s face through a training pic-ture. Following this initial training seed, the face model fora person can evolve over time. For example, a person’s faceis often captured by the phone’s camera (e.g., when usingvideo conferencing) allowing the model to be refined underdifferent conditions (varying light conditions, on the move,etc).

- Pooling: when friends get together their phones pooleach other’s face models. Person A’s phone does not haveto derive a face model for person B. It pools it directly fromperson B ’s phone.

- Collaborative inference: face detection can now berun in parallel to tag friends when taking group pictures.Imagine a group of co-located friends taking pictures of eachother. Each picture could have different people and thelighting and angle of each shot could vary considerably. Co-located phones individually run their face classification al-gorithm and then exchange information to refine the finalinference; for example, tagging the people that the local in-ferences returned with highest confidence.

6. RELATED WORK

Work on applications and systems for sensing enabled mo-bile phones is growing in importance [31, 9, 35, 8, 19, 21, 13,27, 10, 30, 29]. Most of the work in the literature, however,propose local sensing operations running on individual de-vices and do not exploit in-field mobile phones interactions.An exception to this is the work in [19], which considers con-text driven sampling and calibration techniques for mobilesensor networks.

Sensor node co-operation is studied mainly in the contextof static sensor networks where fusion [24, 38, 49] and ag-gregation [52, 45, 33] techniques are applied. The benefitof sensor nodes cooperation in the context of object track-ing using distributed Kalman Filters is discussed in [36, 37].

In [24] the authors propose distributed energy efficient roleassignment and [38] discusses signal processing techniquesto reduce the amount of sensor data needed to detect anevent, while [49] proposes the adoption of distributed aver-age consensus in order to compensate sensing errors. In theCASA project [2] researchers adopt techniques for collabo-rative and adaptive sensing of the atmosphere using radartechnologies. All these projects present techniques for staticand not mobile sensor networks. To the best of our knowl-edge, there is little or no work addressing the issue of howto leverage context-sensitive mobile sensing devices such asmobile phones as proposed by Darwin. There is work on thecontext of using embedded sensors such as the Intel MSP[14] to infer people’s activity. However, no interactions be-tween these devices are taken into account in order to realizeco-operative strategies such as those discussed in this paper.

Recently, techniques that leverage heterogeneous sensingdevices in order to exploit external sensing modalities as afurther input for classification algorithms or boosting appli-cation fidelity in mobile sensing scenarios are proposed in[25, 15]. Our work goes beyond the idea of borrowing sensorreadings from other sensors since we propose collaborativeinference techniques that combine with classifier evolutionand model pooling.

Semi-supervised machine learning techniques are investi-gated for word sense disambiguation [50], to identify subjec-tive nouns [41], or to classify emotional and non emotionaldialogues [28]. However, no work studies semi-supervisedlearning techniques in the context of mobile sensing appli-cations or frameworks.

Audio analysis for speaker identification is a well exploredarea in the literature [17, 42, 16, 44, 43, 26, 18, 40, 23].Although we do not propose new speaker recognition tech-niques, we show how to build a lightweight speaker identifi-cation application capable of running on mobile phones.

7. CONCLUSION

In this paper we presented the design, implementation,and evaluation of the Darwin system that combines classifierevolution, model pooling, and collaborative inference for mo-bile sensing applications on phones. The classifier evolutionmethod presented in this paper is an automated approachto updating models over time such that the classifiers arerobust to the variability in sensing conditions and settingscommon to mobile phones. Mobile phones exchange classifi-cation models whenever the model is available from anotherphone, thus, allowing mobile phones to quickly expand theirclassification capabilities. Collaborative inference combinesthe classification results from multiple phones to achievebetter inference accuracy and confidence. We implementedDarwin on the Nokia N97 and Apple iPhone in support ofa proof-of-concept speaker recognition application. We alsoshowed the integration of Darwin with some demo appli-cations. Our results indicate that the performance boostoffered by Darwin is capable of off-setting problems withsensing context and conditions and presents a frameworkfor scaling classification on mobile devices. Future workwill consider duty cycling techniques for better energy con-servation and studying simplified classification techniques,for example, building more computationally light GMMs formobile phones without impacting performance. We believethe development of such a classification toolkit for mobile

19

phones will enable new research on phones for human cen-tered applications.

8. ACKNOWLEDGMENTS

This work is supported in part by Nokia, Intel Corp.,Microsoft Research, NSF NCS-0631289, and the Institutefor Security Technology Studies (ISTS) at Dartmouth Col-lege. ISTS support is provided by the U.S. Department ofHomeland Security under award 2006-CS-001-000001, andby award 60NANB6D6130 from the U.S. Department ofCommerce. The views and conclusions contained in thisdocument are those of the authors and should not be inter-preted as necessarily representing the official policies, eitherexpressed or implied, of any funding body. A special thanksgoes to our shepherd Deborah Estrin.

9. REFERENCES[1] Apple iPhone. http://www.apple.com/iphone.[2] The CASA project. http://www.casa.umass.edu/.[3] FFTW. http://www.fftw.org.[4] Kiss FFT. http://sourceforge.net/projects/kissfft.[5] Nokia Energy Profiler. http://tinyurl.com/ycostaq.[6] Nokia Series. http://europe.nokia.com/nseries/.[7] Qt platform. http://qt.nokia.com/.[8] Sensorplanet.

http://research.nokia.com/research/projects/sensorplanet/index.html.[9] T. Abdelzaher, et al. Mobiscopes for Human Spaces. In IEEE

Pervasive Computing, volume 6, page 20. IEEE, 2007.[10] M. Azizyan, et al. SurroundSense: Mobile Phone Localization

via Ambience Fingerprinting. In MobiCom’09, 2009.[11] S. Basu. A linked-HMM Model for Robust Voicing and Speech

Detection. In ICASSP’03, 2003.[12] A. Campbell, S. Eisenman, N. Lane, E. Miluzzo, and

R. Peterson. People-Centric Urban Sensing. In WICON’06,2006.

[13] A. Campbell, et al. The Rise of People-Centric Sensing. InIEEE Internet Computing Special Issue on Mesh Networks,July-August 2008.

[14] T. Choudhury, et al. The Mobile Sensing Platform: anEmbedded System for Capturing and Recognizing HumanActivities. IEEE Pervasive Magazine, Spec. Issue on

Activity-Based Computing, 2008.[15] S. B. Eisenman. People-Centric Mobile Sensing Networks.

Ph.D. Dissertation, October 2008.[16] H. Ezzaidi and J. Rouat. Pitch and MFCC Dependent GMM

Models for Speaker Identification systems. In Electrical and

Computer Engineering, 2004. Canadian Conference on,volume 1, 2004.

[17] Z. Fang, Z. Guoliang, and S. Zhanjiang. Comparison ofDifferent Implementations of MFCC. J. Comput. Sci.

Technol., 16(6), 2001.[18] G. Friedland and O. Vinyals. Live Speaker Identification in

Conversations. In ACM Multimedia’08, 2008.[19] R. Honicky, et al. Increasing the Precision of Mobile Sensing

Systems Through Super-Sampling. UrbanSense08, page 31,2008.

[20] R. Honicky, et al. N-SMARTS: Networked Suite of MobileAtmospheric Real-Time Sensors. In Networked Systems for

Developing Regions’08, 2008.[21] A. Kansal, M. Goraczko, and F. Zhao. Building a Sensor

Network of Mobile Phones. In IPSN’07.[22] D. Kim, J. Hightower, R. Govindan, and D. Estrin.

Discovering Semantically Meaningful Places from PervasiveRF-Beacons. In UbiComp ‘09, pages 21–30. ACM, 2009.

[23] T. Kinnunen, T. Kilpelainen, and P. Franti. Comparison ofClustering Algorithms in Speaker Identification. In SPC 2000.

[24] R. Kumar, et al. DFuse: a Framework for Distributed DataFusion. In SenSys’03, 2003.

[25] N. Lane, et al. Cooperative Techniques SupportingSensor-Based People-Centric Inferencing. In Pervasive’08.

[26] I. Lapidot, H. Guterman, and A. Cohen. Unsupervised SpeakerRecognition Based on Competition Between Self-OrganizingMaps. IEEE Transactions on Neural Networks,13(4):877–887, 2002.

[27] H. Lu, et al. SoundSense: Scalable Sound Sensing forPeople-Centric Applications on Mobile Phones. MobiSys’09,2009.

[28] B. Maeireizo, D. Litman, and R. Hwa. Co-Training forPredicting Emotions with Spoken Dialogue Data. In ACL’04,2004.

[29] E. Miluzzo, et al. Sensing Meets Mobile Social Networks: theDesign, Implementation and Evaluation of the CenceMeApplication. In SenSys’08, 2008.

[30] P. Mohan, et al. Nericell: Rich Monitoring of Road and TrafficConditions Using Mobile Smartphones. In SenSys’08, 2008.

[31] M. Mun, et al. PEIR, the Personal Environmental ImpactReport, as a Platform for Participatory Sensing SystemsResearch. In MobiSys’09, 2009.

[32] M. Musolesi, et al. Supporting Energy-Efficient UploadingStrategies for Continuous Sensing Applications on MobilePhones. In Pervasive’10, 2010.

[33] S. Nath, et al. Synopsis Diffusion for Robust Aggregation inSensor Networks. In SenSys’04, 2004.

[34] Nokia. N900. http://maemo.nokia.com/n900.[35] NRC. Sensing the World with Mobile Devices. Technical

Report, 2008.[36] R. Olfati-Saber. Distributed Kalman Filter with Embedded

Consensus Filters. In CDC-ECC’05, pages 8179–8184, 2005.[37] R. Olfati-Saber. Distributed Kalman Filtering and Sensor

Fusion in Sensor Networks. Lecture notes in control and

information sciences, 331:157, 2006.[38] S. Patil, et al. Serial Data Fusion Using Space-Filling Curves

in Wireless Sensor Networks. In IEEE SECON’04, 2004.[39] D. Reynolds, et al. Speaker Verification Using Adapted

Gaussian Mixture Models. Digital Signal Processing,10(1-3):19–41, 2000.

[40] D. Reynolds and R. Rose. Robust Text-Independent SpeakerIdentification Using Gaussian Mixture Speaker Models. IEEE

transactions on Speech and Audio Processing, 1995.[41] E. Riloff, J. Wiebe, and T. Wilson. Learning Subjective Nouns

Using Extraction Pattern Bootstrapping. In CoNLL’03, 2003.[42] E. Shriberg. Higher-Level Features in Speaker Recognition.

Lecture Notes in Computer Science, 4343:241, 2007.[43] E. Shriberg, et al. Modeling Prosodic Feature Sequences for

Speaker Recognition. Speech Communication.[44] E. Shriberg and E. Stolcke. The Case for Automatic

Higher-Level Features in Forensic Speaker Recognition. InInterspeech’08, 2008.

[45] N. Shrivastava, et al. Medians and Beyond: new AggregationTechniques for Sensor Networks. In SenSys’04, pages 239–249.ACM New York, NY, USA, 2004.

[46] D. Siewiorek, et al. Sensay: a Context-Aware Mobile Phone. InISWC’03, 2003.

[47] C. Tadj, et al. GMM Based Speaker Identification UsingTraining Time Dependent Number of Mixtures. In ICASSP’98,volume 2, 1998.

[48] Y. Wang, et al. A Framework of Energy Efficient MobileSensing for Automatic User State Recognition. In MobiSys’09,2009.

[49] L. Xiao, S. Boyd, and S. Lall. A Scheme for Robust DistributedSensor Fusion Based on Average Consensus. In IPSN’05, 2005.

[50] D. Yarowsky. Unsupervised Word Sense DisambiguationRivaling Supervised Methods. In ACL’95.

[51] S. Zhang, S. Zhang, and B. Xu. A Robust UnsupervisedSpeaker Clustering of Speech Utterances. In IEEE

NLP-KE’05.[52] J. Zhao, et al. Computing Aggregates for Monitoring Wireless

Sensor Networks. In SNPA’03, 2003.[53] M. Zhu. Recall, Precision and Average Precision. In Technical

Report, University of Waterloo, 2004.[54] X. Zhu. Semi-Supervised Learning Literature Survey. In

Computer Sciences TR, University of Wisconsin-Madison,2008.

20