Toothbrushing Monitoring using Wrist Watch

Hua HuangStony Brook University

hua.huang@stonybrook.edu

Shan LinStony Brook University

shan.x.lin@stonybrook.edu

ABSTRACTDaily toothbrushing is essential for maintaining oral health.However, there is very limited technology to monitor theeffectiveness of toothbrushing at home. In this paper, asystem is built to monitor the brushing quality on all 16tooth surfaces using a manual toothbrush and an off-the-shelf wrist watch. The toothbrush is modified by attachingsmall magnets to the handle, so that its orientation and mo-tion can be captured by the magnetic sensor in the wristwatch. The toothbrushing gestures are recognized based oninertial sensing data from the wrist watch. As the acousticsignal collected from the watch is correlated with the mo-tion of toothbrushing stroke, acoustic sensing algorithm isdesigned to assist in recognition. User-specific toothbrush-ing order is also utilized to improve the surface recognition.In extensive experiments with 12 users over 3 weeks, oursystem successfully recognized toothbrushing gestures withan average precision of 85.6%.

CCS Concepts•Applied computing → Life and medical sciences;•Human-centered computing → Ubiquitous and mobilecomputing systems and tools;

KeywordsToothbrushing Monitoring, Activity Recognition, Wearables

1. INTRODUCTIONDental caries and gum diseases are among the most preva-

lent chronic diseases in both children and adults [13]. Mostof these dental diseases are the results of bacteria depositedon the surface of the teeth. Without proper brushing, suchbacteria accumulate on the tooth surfaces in a complex calledplaque, destroying the outermost layer of the tooth (enamel)and initiating gingival inflammation, resulting in dental de-cay and gum diseases [6]. Proper toothbrushing is crucial to

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.

SenSys ’16, November 14-16, 2016, Stanford, CA, USAc© 2016 ACM. ISBN 978-1-4503-4263-6/16/11. . . $15.00

DOI: http://dx.doi.org/10.1145/2994551.2994563

plaque reduction and dental pathology prevention. Ameri-can Dental Association recommends brushing teeth for twominutes and twice a day using the Bass technique [29, 16].Unfortunately, there is limited technology for assessmentand reassurance of home toothbrushing compliance. Thereare electric toothbrushes which are designed to make tooth-brushing easier and simpler, but they still rely on users tomove the toothbrush on the correct tooth surfaces. Also,they do not detect whether the user has brushed with theproper technique, complete coverage and sufficient durationfor all tooth surfaces. Therefore, it is desirable to developtechnologies for toothbrushing monitoring.

Wearable devices have been employed in recognition ofvarious activities, such as hand-washing [14], smoking [27],and exercising [11]. These research studies show valuableresults that demonstrate the potential of activity recogni-tion using wearables in health applications. Different fromthese applications, toothbrushing requires the usage of ahand tool: the toothbrush. Both the motions of the tooth-brush and the user gestures to operate the toothbrush needto be recognized.

The toothbrushing activity consists of a series of gesturesfor maneuvering a toothbrush to scrub teeth. There aremany dynamic interactions between the toothbrush and theuser’s hand during toothbrushing, such as grabbing, point-ing, rotating, and flipping of the toothbrush. As the mo-tions of toothbrush and user gestures are tightly coupledwith each other, it is very challenging to model and moni-tor these activities. In this paper, we present a system fortoothbrushing monitoring. This system consists of a man-ual toothbrush and a wrist watch. The manual toothbrushis modified by attaching small magnets (7 cents) on the han-dle. Then we develop a magnetic sensing algorithm on thewrist watch to capture the orientation of the toothbrush andits relative movement to the hand.

To recognize toothbrushing gestures, we construct tooth-brushing gesture models using inertial sensing data. Ourmodel is based on underlying basic motions (wrist flexion/extension, forearm rotation, elbow flexion/extension, andshoulder flexion/extension), which allows us to characterizeeach gesture quantitatively by its motion direction, speed,amplitude, and frequency. Besides using inertial sensingdata, acoustic data collected from the microphone in thewrist watch is used to recognize toothbrushing gestures aswell as toothbrushing stroke frequencies. We also developa hidden Markov model to learn the order of toothbrushingsurfaces, which varies from person to person. By integrat-ing these gesture recognition techniques with the toothbrush

Figure 1: Modified Toothbrush Prototypes

orientation detection, our system is able to recognize thetooth surface being brushed at each moment, and estimatethe toothbrushing effectiveness by counting the number ofstrokes.

We deploy our system with a cloud-based framework, eachwrist watch in our system is connected with the backendsystem on the cloud. 12 users used our system to conductdaily toothbrushing for up to 3 weeks. Experiments havebeen designed to evaluate the recognition accuracy of differ-ent gestures for brushing 16 tooth surfaces, and frequencyof strokes. Experimental results demonstrate that our sys-tem achieves high recognition accuracy in every toothbrush-ing session. We also present a case study with 12 users todemonstrate the plague reduction under proper toothbrush-ing using our system.

We build a real-time toothbrushing monitoring system us-ing wrist watch, which recognizes the brushing quality on all16 tooth surfaces.• To capture dynamic interactions between the tooth-

brush and the useraAZs hand, we attach small magnetson off-the-shelf toothbrushes and design novel mag-netic sensing algorithms to detect toothbrush orienta-tion and motion.• We identify coupled motions of a useraAZs wrist, hand,

and arm during toothbrushing with the Bass tech-nique, and design models based on inertial sensing datato characterize toothbrushing gestures.• We employ a set of features including order of brushing

to improve the recognition performance. We also useacoustic profile to estimate the brushing quality.• We build a toothbrushing monitoring system proto-

type using wrist watch. In a 3 week experiments with12 users, our system successfully recognized the brush-ing gestures on 16 tooth surfaces with an average pre-cision of 85.6%.

2. SYSTEM OVERVIEW

2.1 System Design and ConsiderationOur system includes the following four components:• Modified toothbrush. We modified off-the-shelf man-

ual toothbrushes by attaching an array of magnets onthe handle of the brush. A picture of such tooth-brushes is shown on the Figure 1.• Wrist watch. We use off-the-shelf wrist watches. Four

types of sensors in the wrist watch are used: accelerom-eter, gyroscope, magnetic sensor, and acoustic sen-sor. Most signal processing and machine learning algo-rithms are executed on the smartwatch to ensure real-time processing and to reduce wireless communicationtraffic. The main idea of the recognition algorithm isdemonstrated in Figure 2.

Figure 2: Gesture Recognition

• Smartphone or tablet. A smartphone or tablet is usedas the user interface. It also relays data from the wristwatch to the cloud. The real-time brushing surfacerecognition result is displayed on the phone or tablet.• Cloud. The cloud stores a database of labeled train-

ing data and features, and uses these data to computea personalized machine learning profile for each user.The cloud also stores the public profile and toothbrush-ing progress report of the user, so that the user’s den-tist can monitor the dental hygiene results remotely.

Our system has two phases: the training phase and therunning phase. In the training phase, a user is instructedto conduct toothbrushing using the Bass technique with oursystem for one or multiple times. The system records train-ing data as the baseline. In the running phase, the userconducts daily toothbrushing with our system. Our systemrecognizes the tooth surface being brushed and detects in-correct toothbrushing techniques in real-time. When thetoothbrushing is finished, the system provides a brushingquality report on the number of brushing strokes for eachsurface of the teeth.

2.2 The Bass Toothbrushing TechniqueThere are 16 major surfaces of the teeth that we brush,

as shown in Figure 3. The labial and buccal surfaces aresides of the teeth that are adjacent to the inside of the lipsor the cheek. In this paper, we refer them as outer surfaces.The occlusal surfaces of the teeth are the surfaces that comein contact with those in the opposite jaws during biting orchewing. We refer them as chewing surfaces. The palataland lingual surfaces of the teeth are those next to one’stongue. We refer them as the inner surfaces of the teeth.

For manual brushing, systematic reviews have shown thatthe Bass technique has consistently demonstrated good re-sults. It allows the toothbrush bristles to reach the inter-proximal areas and achieves more effective control in plaquelevels [29, 16]. Several basic gestures of the Bass techniqueare shown at the left lower corner of Figure 2. The generaldescriptions of the Bass technique are: a) to brush frontouter, back outer and back inner surfaces, place the tooth-brush at a 45-degree angle to the gums. Brush these areaswith vertical movement of the toothbrush bristles. Eachbrushing stroke begins from the gum line to the tip of theteeth. b) To brush chewing surfaces, move the bristles of thetoothbrush along the chewing surface of the back teeth in aback and forth motion. c) To brush front inner surfaces, tiltthe brush vertically and make up-and-down strokes.

On the other hand, incorrect brushing, such as the hori-zontal brushing on the inner or outer surfaces, as shown atthe right lower corner of Figure 2, can cause two to threetimes more wear to the enamel than the Bass technique[7, 17, 26].

Figure 3: 16 Tooth Surfaces

Sufficient toothbrushing time is also crucial to good tooth-brushing results. Many previous oral hygiene studies recom-mend two minutes of toothbrushing time [15]. However, suf-ficient overall duration does not guarantee sufficient coveragefor all surfaces of the teeth. In practice, people often dis-tribute brushing time unevenly to different surfaces, whichmay cause over-brushing or under-brushing on some surfacesof the teeth. By monitoring the toothbrushing gestures, oursystem is able to recognize the toothbrushing duration andthe number of brushing strokes for each surface, as well asthe overall toothbrushing time.

2.3 The Coordinate SystemsWe analyze the human arm motions for each toothbrush-

ing gesture quantitatively using a simplified human armmodel, as shown in Figure 4. The arm is divided into theupper arm, the forearm, and the hand. The elbow jointconnects the upper arm and the forearm; the wrist jointconnects the forearm and the hand. The toothbrush is heldby the user in his or her hand. Here we use gesture to referto the movements of the user’s arm.

To describe the toothbrush gesture accurately, it is neces-sary to define the relevant coordinate systems, as shown inFigure 4. The user’s coordinate system is denoted by Xu,Yu, Zu. The Xu−Yu plane is horizontal to the floor, and theYu axis points to the facing direction of the user. The sensordata obtained from the watch uses watch coordinates: Xw,Yw, Zw. When the watch is worn on the right wrist, the Xw

axis is parallel to the forearm. The Yw axis is perpendicularto the forearm and is parallel to the palm, and the Zw axisis perpendicular to the forearm and points toward the backof the forearm. The motion of the toothbrush is describedin the toothbrush coordinate system: Xt, Yt, Zt. Xt axispoints to the orientation of the toothbrush bristles, and Zt

axis points to the top of the toothbrush. These coordinatesystems are used in the following sections.

3. TOOTHBRUSH BRISTLE ORIENTATIONRECOGNITION

The detection of the motions of the toothbrush in a user’shand is essential for toothbrushing gesture recognition. Specif-ically, toothbrush motions are critical for gesture recognitionin the following scenarios:• Scenario 1: The toothbrush bristle orientation, i.e.,

Yw

Xw

Zw

Yu

Xu

ZuZt Yt

Xt

Figure 4: The Arm Model and Coordinate Systems

the orientation of Xt, as shown in Figure 4. Certaingestures are very similar to each other, such as the oneswhen brushing the chewing surfaces of the upper andlower teeth. The only difference is the orientations ofthe toothbrush bristles.• Scenario 2: Toothbrush rotations around the Zt axis.

Certain users use his or her fingers to rotate the tooth-brush to scrub teeth.

In our study, we found that these two types of toothbrushmotions made almost no effect on wrist-worn accelerometersand gyroscopes. Therefore, it is necessary to apply alterna-tive approaches to detect these motions.

Our solution is to customize off-the-shelf toothbrushes sothat they become sensible to the smartwatches. We proposeto install small magnets on the toothbrushes. When themagnets move along with the toothbrush, the surroundingmagnetic field is disturbed, which can be reliably capturedby the magnetic sensor in the watch. This enables our sys-tem to recognize the toothbrush motions.

Figure 1 shows a couple prototype toothbrushes. Themagnets are marked by white boxes in the figure. We cansee that small magnets are attached to the handle of thegray toothbrush, whereas some magnets are put inside thehandle of the other purple toothbrush. We tested and foundthat when the magnet’s north pole direction is perpendicularwith the Zt direction of the toothbrush, the sensing resultswere more effective in distinguishing toothbrush bristle ori-entations. Such small magnets cost only 7 cents each. Ourvolunteers did not experience any inconvenience with themodified toothbrushes. In the future, we can develop moreuser-friendly designs, such as a magnetic snap or case thatcan be easily installed on the handle of a toothbrush.

We denote the magnetic moment of the magnet on thetoothbrush as ~m, the location of the magnet relative to thewatch as ~r, and the magnetic permeability of air as µ0. Thenthe magnetic flux density ~B = (Bx, By, Bz) at the wristwatch is represented by the following equation:

~B =µ0

4π

3~r(~r · ~m)− ~m |~r|2

|~r|5. (1)

From the above equation, we can see that when the userchanges his or her hold of the toothbrush, which affects ~r,or rotates the toothbrush with fingers, which affects ~m, themagnetic field ~B will change as well. This enables us to infertoothbrush motions from the magnetic field.

3.1 Toothbrush Bristle OrientationIn an experiment, a user held a toothbrush to let the

toothbrush bristle face the following 4 surfaces: the Left

0 10 20 30 40time (s)

−100

−50

0

50

100

150

200mag

netic

field streng

th (u

T)

LLO LLC LLI LUC

By Bz

(a) Toothbrush Bristle Orientation

4 6 8 10 12 14 16 18 20time (s)

−50

0

50

100

150

200

mag

netic

field streng

th (u

T)

forearmrotation

Wristmotion

elbow/shouldermotion

fingerrotation

Bx By Bz

(b) Magnetic Sensing Data under Tooth-brushing Gestures

−100 −50 0 50 100Bx (uT)

−150

−100

−50

0

50

100

B z (u

T)

RLC

RUC

RLC

RUC

northeastsouthwest

(c) Impact of Geo-magnetic Field on BrushMotion Detection

Figure 5: Magnetic Sensing Results

Lower Outer (LLO) surface , the Left Lower Chewing (LLC)surface, the Left Lower Inner (LLI) surface, and the LeftUpper Chewing (LUC) surface, while magnetic sensing datawas collected from the wrist watch.

The magnetic sensing result is shown in Figure 5a. Fromthis figure, we can see that the magnetic field strength By

and Bz are dramatically different when the toothbrush bris-tles have different orientations. This allows us to recognizethe approximate orientation of the toothbrush bristles dur-ing the toothbrushing of each surface. When the toothbrushbristles orientation stays fixed in one direction, the magneticstrength readings are very stable. When the toothbrush bris-tles orientation changes by 90 degree, the magnetic strengthreadings change significantly to new levels. This observationsuggests us to design a threshold based detection algorithmto capture the toothbrushing surface transitions.

3.2 Toothbrush RotationWe conducted experiments to study the relations between

the magnetic sensor data and different toothbrushing mo-tions of the user. In this experiment, a user first performedthree basic gestures while holding the toothbrush: the fore-arm rotations, the wrist flexions/extensions and the elbow/shoulder motions. After that he rotated the toothbrush withhis fingers.

The corresponding magnetic field readings are shown inFigure 5b. From this figure, we can see that when the useris rotating the toothbrush with fingers (from 17s to 21s),the magnetic field readings along all the three axes oscil-late, with a frequency directly related to the rotation speed.When the user is conducting wrist or elbow/shoulder mo-tions (from 7s to 17s), the motions’ influences on the mag-netic field reading are small, and we don’t see any oscillation.This is because neither the relative location of the tooth-brush ~r, nor the relative direction of the magnetic moment~m to the watch changes significantly.

3.3 Robustness to Geomagnetic FieldThe geomagnetic field is the magnetic field that extends

from the Earth’s interior out into space. It can interferewith our magnetic sensing system and significantly degradethe sensing performance. In our observation, we find thatthe geomagnetic field has a varying magnitude dependingon locations, and its value can go up to more than 100 µT .To obtain robust sensing results under such interference, weuse multiple magnets with strong magnetization together.To test the influence of geomagnetic field, we conducted anexperiment: a user brushed his Right Lower Chewing (RLC)

surface and Right Upper Chewing (RUC) surface with atoothbrush with four small magnets attached, while facingfour cardinal directions: east, west, north, and south. Weplot the magnetic sensing data of Bx and Bz axes in theFigure 5c. Clearly, we can see two clusters representing twotoothbrush bristles orientations, and these two clusters areseparated with a long distance. This indicates that withstrong magnets on the toothbrush, we can obtain robustsensing results on the toothbrush bristle orientation, regard-less of which cardinal direction the user is facing. We canalso see that within a cluster, plotted points with differentcardinal directions do not overlap with each other, whichshows there is still noticeable impact from the geomagneticfield.

3.4 Feature DesignBased on these observations, we design signal processing

algorithms to extract features. Given the magnetic fieldreading in a time window, we firstly calculate average andvariance values ofBx, Bx, andBz. We also calculate the zerocrossing rate, which is defined as the difference between twoconsecutive times when the magnetic field strength valuecrosses the average value. In our experiments, when theuser is brushing using finger rotations, the zero crossing rateranges from 0.2 seconds to 1 seconds, with variation less than3.

4. TOOTHBRUSHING GESTURE RECOG-NITION

In this section, we study the motions of the user’s armduring the toothbrushing process, and develop algorithms torecognize the gestures for brushing different tooth surfaces.There are two major challenges in achieving this goal:• Recognition of similar toothbrushing gestures. Effec-

tive features need to be designed to capture the subtledifferences in gestures of brushing different tooth sur-faces.• Robust recognition with gesture variations. When brush-

ing the same surface, the users can have different armpostures, such as either holding the arm horizontallyor vertically. This is a commonly seen variation andcan significantly affect the sensing results.

We address the first challenge in Section 4.1 and 4.2 bydeveloping motion models for the arm, and design featuresbased on these models. We address the second challengein Section 4.3 by studying the user gesture variations anddesigning a transformation algorithm to offset the effect.

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4Time (s)

−10

−5

0

5

10

15Ac

cele

ratio

n (m

/s2 )

AccyAccz

(a) Acceleration Data of El-bow/Shoulder Motion

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4Time (s)

−2.0−1.5−1.0−0.50.00.51.01.5

Acce

leratio

n (m

/s2 )

(b) Band-pass FilteredAcceleration data of El-bow/Shoulder Motion

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4Time (s)

−7.5

−5.0

−2.5

0.0

2.5

5.0

7.5

Acce

leratio

n (m

/s2 )

(c) Acceleration data of WristMotion

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4Time (s)

−2

−1

0

1

2

3

Acce

leratio

n (m

/s2 )

(d) Band-pass Filtered Accel-eration data of Wrist Motion

Figure 6: Acceleration Data of Elbow and Wrist Motions

4.1 Toothbrushing Gesture ModelAlthough toothbrushing involves complex motions of the

human arm, there are four basic types of motions: wrist flex-ion/extension, forearm rotation (pronation/supination), el-bow flexion/extension, and shoulder movement. Each brush-ing gesture is a combination of these basic motions. Ourstrategy is to establish models for these motions, and designrecognition algorithms based on these models.

We note that both elbow and shoulder movements affectthe entire forearm, the wrist watch cannot differentiate thesetwo types of basic motions. So we combine them as a singlemotion type, the shoulder/elbow flexion/extension.

4.1.1 Forearm RotationThe forearm rotation motion is closely correlated with the

gyroscope reading gx along Xw axis. This is because thewatch is usually tightly attached to the user’s wrist, androtates at the same rate as the forearm. Therefore, we cal-culate the moving variance of gx, denoted by σg

x, as a featureto indicate the existence of this motion. Our experimentalresults show that this feature is effective. For example, formany users, the gesture of brushing of left lower outer sur-face of teeth involves much forearm rotation, and the valueof σg

x remains large during this brushing gesture.

4.1.2 Elbow/Shoulder Flexion/ExtensionWhen a user brushes the two front inner surfaces and the

four chewing surfaces, the elbow/shoulder flexion/extensionis used primarily. In Figure 6a, we plot the accelerome-ter data collected when the user is brushing the right lowerchewing surface. In this figure, the blue dashed line and thered thick line represent the acceleration of the watch alongthe Yw axis (Accy) and the Zw axis (Accz), respectively. Wecan see that the values of Accy and Accz fluctuate periodi-cally according to each brushing stroke. The same patterncan be found when the user is brushing all the other sur-faces involving the same basic motion type. This pattern is

0.0 2.5 5.0 7.5 10.0 12.5 15.0Frequency (Hz)

0

10

20

30

40

50

Powe

r/Frequ

ency

(dB/Hz) Elbow Motion

Wrist Motion

Figure 7: Power Spectrum of Elbow and Wrist Motion Ac-celeration Data

generated by the regular back-forth horizontal motions. Tocapture this type of motion, we analyze the power spectrumof the signal. In Figure 7, we plot the log-scale power spectrafor the acceleration data of the shoulder/elbow motion (theblue line) and the wrist motion (the red dashed line). Fromthis figure, we can see that for the elbow motions, there isa large peak located at around 4.5 Hz, which correspondsthe brushing stroke frequency. Based on this observation,we extract the peak value of the power spectrum max(Pacc)and the percentage of energy concentrated around the peakfrequency (percY ) as features to recognize this type of mo-tion.

4.1.3 Wrist Flexion/ExtensionWe discover that a characterizing pattern of such motions

is the negative correlation between the acceleration valuesof Accy and Accz. This is illustrated in Figure 6c, whereAccy (the blue dashed line) and Accz (the red line) are neg-atively correlated. This is significantly different from theelbow brushing motions when Accy and Accz are positivelycorrelated (shown in Figure 6a). This observation motivatesus to measure the correlation of Accy and Accz as a featureto recognize the wrist motions.

We also find that in some cases, the user’s toothbrushinggestures include both the wrist and the elbow motions. Inthese cases, both types of motion influence the correlationbetween Accy and Accz, and may entirely cover up the wristmotion pattern. This affects the detection accuracy for thesystem, so it is crucial to separate the influence of these twotypes of motion.

We notice that for the wrist motions, two peaks are gener-ated in the power spectrum, as shown in the red dashed linein Figure 7. On the other hand, only one peak is generatedfor the power spectrum of elbow motions (the blue line).This indicates that these two types of motions may createinfluence on wrist acceleration data at different frequencies.Therefore, we conduct a band-pass filter centering at abouttwo times of regular brushing frequency on the accelerationvalues, and the results are shown in Figure 6d. We can seethat this filter successfully suppresses the influence of theelbow motion on the correlation between Accy and Accz,and the feature for wrist motions is revealed. On the otherhand, in Figure 6b, the correlation between Accy and Accz ispositive due to the absence of wrist motions. Therefore, wewill apply a band-pass filter to the acceleration value beforecalculating the correlations.

The feature to recognize wrist motions is defined as fol-lows. After the acceleration values of Accy and Accz are

0 1 2 3 4 5

dxy

1

2

3

4

5

6

dyz

FUO

FLI

LLC

Figure 8: PCA Direction Feature

collected, we apply a band-pass filter with center frequencyabout two times of brushing frequency, and get the filteredvalue Acc′y and Acc′z. Then the correlation Cy,z betweenAcc′y and Accz is calculated. If Cy,z is smaller than a nega-tive threshold, then wrist motion is detected.

4.2 Other Features

4.2.1 PCA-based motion directionWhen brushing different surfaces, the toothbrush is mov-

ing along different directions. For example, when a userbrushes his or her chewing surfaces, the toothbrush typi-cally moves along a direction perpendicular to the directionof gravity. On the other hand, when the outer surfaces arebeing brushed, the toothbrush generally moves along a di-rection parallel to the direction of gravity. Therefore, themoving direction of the toothbrush also provides importantinformation for the toothbrushing gesture recognition.

We design the Principle Component Analysis (PCA) basedmotion direction feature to capture this information. PCA[18][5] is a classical algorithm that finds out a set of linearlyuncorrelated variables called principal components. Theprinciple components are found in such a way that the firstprincipal component has the largest possible variance, andeach succeeding component, in turn, has the highest vari-ance possible under the constraint that it is orthogonal toall the previous ones. In our scenario, the first principalcomponent represents the dominant motion direction of thewatch.

Specifically, we calculate the PCA direction feature as fol-lows. For each time window, this feature is extracted fromthe accelerometer sensor data [Accx, Accy, Accz]. For eachpair of dimensions i and j, i, j ∈ {x, y, z}, i 6= j, we feedthe sensor data X = [Acci, Accj ] to the PCA algorithm andcompute the eigenvectors W of the covariance matrix XTX.The 2×2 matrix W contains the information about the prin-ciple component. We record the slope dij = W [2, 1]/W [1, 1]to represent the motion direction in the ij plane.

We plot the PCA direction features for brushing Front Up-per Outer (FUO), Front Lower Inner (FLI) and Left LowerChewing (LLC) surfaces in Figure 8. We can clearly see theplotted points naturally form three clusters, which allowsus to differentiate these different gestures. This shows thatthe PCA direction feature can distinguish not only horizon-tal (LLC) motions from vertical motions (FUO), but alsohorizontal motions with different directions (LLC and FLI).

4.2.2 Statistical Features

(a) (b)

Figure 9: Toothbrushing with Different Elbow Posi-tions:(a)High Elbow Position, (b)Low Elbow Position

We also extract the statistical features from the sensordata. Given the sensor data s = [s1, s2, ...sn] collected in atime window, we compute their average value s, variance σs

and skewness γs. In other words, we calculate the first, sec-ond and third order moments of the sensor data as the statis-tical features. The average value reflects the overall state ofthe gesture. For example, the average value of gravity datais closely related to the user’s elbow position. The variancevalue is related to the intensity of the motions. For example,the variance of the acceleration shows how intensely the useris brushing the teeth.

The skewness is used to distinguish the subtle differencesof the motions for brushing the upper or lower teeth. Skew-ness has been used to measure how asymmetric a randomvariable is about its mean. For example, when a user brushesthe front upper outer surface, he generally exerts larger forcewhen moving the toothbrush from the gum line downwardto the end of the upper teeth. After that, with a smallerforce, he raises the brush upward to prepare for next brush-ing stroke. When brushing the front lower outer surface, theprocess is the opposite: the toothbrush moves upward witha larger force and downward with a smaller force. The dif-ference in the acceleration of the downward-upward motionscan be captured by the skewness of the sensor data.

4.3 Dynamic Gestures VariationsWe now consider the variation of the toothbrushing mo-

tions for the toothbrushing gestures for the same surface.We focus on the change of elbow positions during the brush-ing. This is illustrated in Figure 9. In this figure, the user(a) is brushing with his elbow raised up, whereas, the user(b) is brushing with his elbow put down. This type of el-bow position variation is common in real life. When theuser changes his or her elbow positions, the sensing resultsfrom the smartwatch change significantly, even though theuser is still brushing the same tooth surface with the sametechnique. This type of variation can significantly affect thetoothbrushing surface recognition accuracy.

This type of variation is due to the change of watch co-ordinates Xw − Yw − Zw. When the elbow position is high,the Zw is facing vertically upward. However, when the el-bow position is low, Zw rotates to the horizontal right-handside. On the other hand, since the user is brushing the samesurface, the motions of the toothbrush remain the same. Asa result, the sensor data collected are significantly differentdue to the rotation of the coordinate system.

The basic idea is to conduct a coordinate rotation trans-formation to offset this type of variation. We compute therotation matrix R to transform the sensor data from thesmartwatch’s coordinates Xw − Yw −Zw to the user coordi-nates Xu − Yu − Zu.

To compute the rotation matrix R, we have the following

Time (s)0 0.5 1 1.5 2

So

un

d W

ave

Am

plit

ud

e

-0.03

-0.02

-0.01

0

0.01

0.02

0.03

0.04

0.05

0.06

Original Sound Wave

Moving RMS

Peak Locations

Figure 10: Brushing Stroke Estimation

two observations. Firstly, gravity sensed by the watch isparallel to the Zu. This is because, by definition, Zu isperpendicular to the floor, and parallel to gravity. Secondly,when the user is brushing, the Yw direction of the watch isapproximately parallel to the Yu direction of the user, sincewhen the user is brushing teeth, his/her hand is held up inthe front. Based on these two observations, we can computethe rotation matrix R by solving the following equation set:

[0, 0, g] = R ∗ ~gravity

[0, 1, 0] = I ∗R ∗ ~Yw(2)

Here ~gravity represents the gravity sensed by the watch,which can be obtained by the Android API. g representsthe magnitude of gravity. We take the value of 9.8m/s2. Bymultiplying the raw sensor data with the rotation matrix R,we can get the sensor data represented in the user’s coordi-nate. After this transformation, the sensor data are readyto be used for the feature extraction algorithms.

5. COUNTING TOOTHBRUSHINGSTROKES USING ACOUSTIC SENSING

Each time a toothbrushing gesture is identified, our sys-tem estimates the number of brushing strokes conductedduring that time interval. In this study, we use the acousticsensors to achieve this goal. The acoustic sensors have beenused in toothbrushing surface recognition [20]. One advan-tage of the acoustic sensors is that they are affected by noisesdifferent from those affecting the motion sensors. Therefore,by including the acoustic sensing in the toothbrushing mon-itoring, we can potentially improve the system robustness tonoises.

Sounds are generated during each toothbrushing stroke.To count the toothbrushing strokes, our system use a RootMean Square (RMS) based approach to track the intensityof the sound wave, and mark each significant peak as anindicator of a toothbrushing stroke. Given any sound waveseries qt collected for the current time interval [tstart, tend],we compute the moving RMS value during this time window,denoted by qrms

t . After the RMS value of the sound waveis computed, we then find all the local maximas of qrms

t

that have values larger than the average RMS value of theentire window. Then we combine peaks that are within 0.1seconds of each other. We count all the remaining soundwave peaks as an estimation of toothbrushing strokes. This

Algorithm 1 Toothbrushing Monitoring Algorithm

1: Collect sensor data from accelerometer, gyroscope, grav-ity sensor, magnetic sensor and acoustic sensor

2: Preprocess sensor data3: Conduct coordinate rotate transform for acceleration

data based on the gesture variation model4: Extract features based on the toothbrush orientation

model and the toothbrushing gesture models.5: Detect incorrect toothbrushing techniques6: Recognize toothbrushing surface7: Estimate the number of toothbrushing strokes8: Display and backup data

algorithm is illustrated in Figure 10. We can see that in thisfigure, the sound wave is collected in 2 seconds. There are5 peaks of RMS value detected, which correspond to the 5toothbrushing strokes during this time interval.

6. TOOTHBRUSHING MONITORING AL-GORITHM

Our gesture recognition strategy is summarized in Al-gorithm 1. During the toothbrushing process, the smartwatch continuously collects sensor data from the accelerom-eter, gyroscope, gravity sensor, magnetic sensor and acous-tic sensor. These data are buffered in windows of a fixedtime length. Then the acceleration data are processed usinggesture variation offset algorithm described in Section 4.3.Based on the toothbrush orientation model and the arm mo-tion models, the system extracts the toothbrush orientationfeatures, the forearm rotation features, the elbow/shoulderflexion/extension features, wrist flexion/extension features,PCA-based motion direction features and the statistical fea-tures.

After all the features are extracted, the system proceedsto detect incorrect toothbrushing techniques. The incor-rect toothbrushing technique detection algorithm serves twogoals. Firstly, it filters out unseen gestures, so that the sur-face classification algorithm can have better performance.Secondly, it warns users about incorrect toothbrushing tech-niques from time to time.

The detection of incorrect toothbrushing technique is chal-lenging because there are many different incorrect ways ofbrushing. Therefore, it is impractical to collect training datafor all these techniques. Our strategy is to regard this prob-lem as an anomaly detection problem. Based on our modelsof toothbrush orientations and toothbrushing gestures, oursystem will identify whether or not the sensor data is normal(Bass toothbrushing technique), or is too different from thetraining data (non-Bass toothbrushing technique).

We select the one-class Support Vector Machine (SVM)classifier to detect incorrect toothbrushing techniques. Thenice property of SVM is that by using the kernel trick, thealgorithm can compute a highly non-linear separating planefor classification. The one-class SVM separates all the datapoints from the origin in the feature space and maximizesthe distance from this hyperplane to the origin [32]. Thisresults in a binary function that returns +1 if the data isin the region of the training data points, and -1 elsewhere.Each time an incorrect toothbrushing technique is detected,a message will be displayed to notify the user.

If the input data is recognized to be the correct (Bass)

FUO FLO FU

I FLI LLO LLC LLI LUOLUC LU

IRLO RL

C RLIRUORUC RU

I

Next Surface

FUOFLOFUIFLILLOLLCLLI

LUOLUCLUIRLORLCRLI

RUORUCRUI

Curre

nt Surface

100

100

100

100

4.5 95.

90. 4.5 4.5

86. 13.

95. 4.5

4.5 9.0 86.

95. 4.5

13. 86.

95. 4.5

9.0 72. 18.

72. 27.

13. 4.5 81.

Figure 11: Transition Probability for Toothbrushing Surface

toothbrushing technique, the system will determine the tooth-brushing surface at the next stage. In general, any multi-class classifier can be used. We evaluated several classifiersand selected the Naive Bayes classifier. Specifically, giventhe training label (the ground truth toothbrushing surface)zi and the training feature set fi, the conditional probabil-ity p(fi|zi), the class prior probability p(zi) and the featureprobability distribution p(fi) are computed. In the testingphase, given an input feature set finquiry, the predicted classprobability is computed using the Bayes Rule:

p(z|finquiry) =p(finquiry|zi) ∗ p(zi)

p(finquiry). (3)

The label z′ that maximizes p(z′|finquiry) is returned bythe algorithm as the predicted toothbrushing gesture.

Finally, the toothbrushing stroke estimation algorithm isused. The estimated number of toothbrushing strokes andthe predicted labels are recorded and will be presented tothe user after the toothbrushing is finished.

6.1 Improving Recognition Accuracy withToothbrushing Order

People usually have set toothbrushing habits. One suchhabit is the order of brushing the 16 tooth surfaces. Withthe data we collected, we observed that users had certainpatterns in the order of toothbrushing. Similar observationsare also reported in literature [25]. In this subsection, wedevelop a hidden Markov model to learn the probability dis-tribution of the user-specific brushing order, which allowsus to estimate the current brushing surface based on previ-ously brushed surfaces. We can also use it to correct falserecognition results from motion models. Combined with ourgesture recognition algorithms, this improves the accuracyof our brushing gestures recognition solution.

6.1.1 ObservationsSome users have stable toothbrushing orders. As an exam-

ple, we analyzed twenty-two toothbrushing sessions for oneof the users, and computed the transition probability matrix

of the brushing order in Figure 11. We found that there wasa strong pattern in the toothbrushing order. From this ma-trix, we can see that the transition probabilities from FrontUpper Outer (FUO) to Front Lower Outer (FLO), from FLOto Left Lower Outer (LLO), from Front Upper Inner (FUI)to Front Lower Inner (FLI), and from FLI to Right LowerOuter (RLO), are 100%. This means that these orders werefollowed in all the twenty-two toothbrushing sessions. Be-sides, for all the surfaces, the transition probabilities for themost likely next surface are all greater than 70%. In sum,the toothbrushing order is highly predictable and was fol-lowed by most of the twenty-two toothbrushing sessions.

6.1.2 Integration with the Surface Recognition Algo-rithm

We design a simple algorithm that can be easily inte-grated with our system. The algorithm maintains a sur-face transition probability matrix Mt. The algorithm is in-voked only when the Bayes based surface classifier makes aprediction z(t)′ that is different from the previous predic-tion z(t − 1). If the classification confidence p(z(t)′|f) isgreater than an ambiguity threshold Tc, it means the pred-ication is accurate, and the new prediction value z(t)′ willbe used to update the matrix Mt. On the other hand, ifthe classification confidence p(z(t)′|f) is smaller than Tc,the algorithm will find out the most likely next surface z′′

given the previous surface z(t − 1) from the transition ma-trix Mt. Then the algorithm computes the weighted prob-ability c1 = (1 − α)p(z(t)′|f) + αMt(z(t − 1), z(t)′) andc2 = (1 − α)p(z′′|f) + αMt(z(t − 1), z′′). The algorithmwill return z(t)′ if c1 ≥ c2, and return z′′ otherwise.

7. EVALUATION

7.1 Experiment Setup

7.1.1 System ImplementationWe have implemented and tested the system on the Gear

Live smartwatch [4], Nexus 7 (2013) tablets [3] and the Ama-zon Web Service (AWS) [1] platform. The Gear Live smart-watch has an accelerometer, a gyroscope, a gravity sensor,a magnetic sensor, and a microphone. It has 512 Mb RAMand a Quad-core 1.3GHz CPU. It uses a Bluetooth radio tocommunicate with the tablet. We use the AWS S3 serviceto store the sensor data, and use the AWS EC2 computationplatform to process the data.

For the software part, we have implemented the featureextraction and classification modules on the Android Oper-ating System installed on the Gear Live smartwatch.

7.1.2 DatasetsWe deployed this system to 12 volunteers to conduct daily

toothbrushing. At the beginning of the experiment, wedemonstrated the Bass technique to the volunteers in per-son and let them watch an instruction video. Each volun-teer brushed his or her teeth for at least two minutes twicea day. The system asked users to label each of their brush-ing surfaces with a user interface on the tablet during eachtoothbrushing session. This baseline collection phase lastedfor up to 14 days. After this phase, the users conductedtoothbrushing normally wearing our wrist watch withouthaving to input gesture labels. The ground truth labels wererecorded by a second observer or by video recording. The

second phase of data collection lasted for up to 7 days. Byone set of training data, we mean the amount of sensor datacollected in one toothbrushing session.

To evaluate the incorrect toothbrushing detection algo-rithm, we recorded sensor data from four of these volun-teers. The volunteers brushed their teeth using the hori-zontal technique, one of the most commonly seen incorrecttoothbrushing technique [28]. They brushed each of their in-ner and outer surfaces (12 surfaces) using horizontal strokes.Each volunteer brushed for five times. The incorrect tooth-brushing detection algorithm was trained solely by correcttoothbrushing data and was tested using both the correctand incorrect toothbrushing data.

7.1.3 Evaluation MethodologyToothbrushing sensor data collected from each user are

divided into one-second segments. We use k-fold cross val-idation for performance evaluation. Each round of cross-validation involves partitioning a sample of data into com-plementary subsets, performing the analysis on one subset(called the training set), and validating the analysis on theother subset (called the validation set or testing set). Toobtain statistical confidence, ten rounds of cross-validationwere performed using different partitions, and the validationresults are averaged over the rounds.

To measure the toothbrushing surface recognition accu-racy, we use the precision, recall, and F1 score. Precision isdefined as the percentage of time the algorithm makes cor-rect predictions. The recall rate is defined as the percentageof time when the user is brushing any surface, and the sys-tem recognizes it correctly. F1 score is commonly used tointegrate both results. Specifically, precision, recall and F1

score are computed using the following equation:

precision = TP/(TP + FP )recall = TP/(TP + FN)F1 = 2 ∗ precision ∗ recall/(precesion+ recall)

(4)

In these equations, TP represents true positive, FP rep-resents false positive and FN represents false negative.

To evaluate the incorrect toothbrushing gesture recogni-tion accuracy, we use the Recall rate and the True Nega-tive Rate (TNR). TNR measures the proportion of incorrecttoothbrushing gestures that are detected by the algorithm.It is defined as TN/(TN + FP ).

7.2 Toothbrushing Surface Recognition

7.2.1 The Effectiveness of Different FeaturesWe present the results on effectiveness of different fea-

tures. Specifically, we evaluated the statistical features (sta)defined in Section 4.2.2, the gesture model features (model)defined in Section 4.1, the PCA direction features (pca) de-fined in Section 4.2.1, and the toothbrush bristle orientationfeatures (mag) defined in Section 3.4. Finally, we also testedthe effectiveness of the toothbrushing order based algorithmdescribed in Section 6.1. In each round of cross validation,three sets of sensor data were used to train the machinelearning algorithm, and the remaining sensor data were usedfor evaluation. The evaluation accuracy results are shownin Figure 12.

From this figure, we can see that when only the statisticalfeatures are used, the recognition precision is only 65.6%,

sta stamodel

sta,pcamodel

sta,pcamodelmag

all,order

0.5

0.6

0.7

0.8

0.9

PrecisionRecallF1

Figure 12: Surface Recognition Accuracy vs. Different Fea-ture Sets

meaning that the subtle differences of the motions are notcaptured by these features alone. When the gesture modelfeatures are included, the system can better distinguish theback-forth horizontal motions and the vertical rolling mo-tions. As a result, the recognition precision is improved to72.9%. The PCA features are designed to determine thedirections of the toothbrush motions, and can provide anadditional 3% of improvements to the recognition precision.

When the magnetic sensing based toothbrush orientationfeatures are used, the precision is boosted to 85.6%. Theimprovement is due to the features’ effectiveness in distin-guishing orientations of the toothbrushes, which are hard torecognize using motion sensors alone. For example, withoutthese features, the toothbrushing gestures of upper or lowerchewing surfaces, which have almost identical motions, arehard to distinguish.

In the implementation of the toothbrushing order basedalgorithm, we set the ambiguity threshold Tc to a high valueso that only approximately 5% of the testing data will berecognized as ambiguous. The results are shown in Figure 12in the column with label“all,order”. From this figure, we cansee that there is a 2.1% increase in the surface recognitionprecision. It shows that toothbrushing order information isuseful in the surface recognition algorithm. Nevertheless,from this experiment we observed that there were still twoways to improve the algorithm. Firstly, some false predic-tions occur continuously. In such cases, the algorithm wouldfalsely regard them as brushing the same surface so that nocorrection will occur. Secondly, we find that the posteriorprobability of the Naive Bayes classifier is not always aneffective indicator of prediction confidence. In some cases,the prediction confidence (i.e. the posterior probability) re-turned by the Bayes classifier is close to 1, even when theprediction results are incorrect [8]. To fully utilize the tooth-brushing order information, it’s important to have a moreaccurate prediction confidence estimation.

We note that in different rounds of cross validation, about3% of standard deviation in the recognition accuracy (pre-cision, recall and F1) existed, as we can see in the figure. Toimprove the confidence of the recognition accuracy results,we have conducted ten rounds of experiments and calculatedthe average.

7.2.2 Surface Recognition Confusion MatrixTo better understand the strengths and weaknesses of the

FUO FLO FU

I FLI LLO LLC LLI LUOLUC LU

IRLO RL

C RLIRUORUC RU

I

Predicted label

FUOFLOFUIFLILLOLLCLLI

LUOLUCLUIRLORLCRLI

RUORUCRUI

True

labe

l81. 6.2 1.2 4.4 1.2 2.5 3.118. 76. 2.1 1.6 0.5 0.5 0.5

3.2 85. 0.6 0.6 5.1 0.6 3.2 1.29.9 77. 0.7 0.7 3.0 2.2 5.3

0.6 1.3 96. 0.6 0.617. 0.6 76. 5.40.7 3.0 87. 9.0

1.4 0.7 1.4 0.7 1.4 93. 0.70.6 0.6 0.6 0.6 91. 0.6 0.6 4.7

0.7 6.4 85. 4.3 1.4 1.42.2 7.5 86. 3.7

0.7 1.4 2.2 1.4 94.2.9 1.4 95.

0.6 4.1 11. 81. 0.6 2.08.9 91.

2.0 0.6 0.6 96.

Figure 13: Toothbrushing Surface Recognition ConfusionMatrix

surface recognition algorithm, we study the confusion ma-trix M shown in Figure 13. In this matrix, each entry incell M(i, j) represents the probability for the algorithm topredict a gesture as label j, when its true label is i. Due tothe roundup error, each row of the matrix may not sum upto 100%.

The confusion matrix shows that our designs are effec-tive in many ways. Firstly, we can see that our systemeffectively distinguishes the upper and lower chewing sur-faces. For example, there are almost no confusions betweenRLC and RUC, or between LLC and LUC. This is becauseof the introduction of the magnetic sensing design, whichreliably distinguish upper or lower chewing surfaces. Be-sides, the Left Lower Outer (LLO) surface is reliably rec-ognized (96%), showing the effectiveness of the forearm ro-tation model. Furthermore, we can see that the six elbowmotion gestures (FUI, FLI, LLC, LUC, RLC, RUC) are al-most never confused with other gestures. This also showsthe effectiveness of the elbow motion model.

We can also see that there are difficulties in toothbrushingsurface recognition. There are a few toothbrushing gesturesthat are very similar to each other. We can see that thereare 18% of Front Lower Outer (FLO) data falsely classifiedas Front Upper Outer (FUO). Gestures of toothbrushingthese two surfaces are very similar, and the major differ-ence is the rolling direction of the toothbrush, and the skew-ness features(Section 4.2.2) are designed to distinguish them.Nevertheless, there are still cases not recognized. Anothermajor source of error comes from the toothbrushing direc-tion recognition. To name a few, 17% of Left Lower Chewing(LLC) data are falsely classified as Front Lower Inner (FLI),and 8.1% of Right Upper Chewing (RUC) data are classi-fied as Front Upper Inner (FUI). The motions of these threegestures are similar. They are all horizontal back-forth mo-tions primarily driven by elbow/shoulder flexion/extension.The key to distinguishing them is to recognize their motiondirections, and the PCA-based direction features are used.

1 2 3 4 5 6Number of Training Data Sets

0.60

0.65

0.70

0.75

0.80

0.85

PrecisionRecallF1

Figure 14: Surface Recognition Accuracy vs. Training DataSet Size

However, there are still cases when this feature cannot dis-tinguish effectively.

Even though there are recognition errors during some one-second segments, since the brushing of each surface lasts forseveral seconds, it is still possible for the system to recognizethe surface accurately. We will work on this direction in thefuture work.

7.2.3 The Influence of Training Data Set SizeIn this experiment, we studied the toothbrushing surface

recognition accuracy when different amount of training datawere used. For each user, we use from one to six sets oftoothbrushing data to train the machine learning model. Af-ter the model had been trained, we evaluated its recognitionaccuracy using all the remaining sensor data. The resultsare shown in Figure 14.

From this figure, we can see that when one set of tooth-brushing data is used, the surface recognition precision is69.2%. As more training data sets are used, the surfacerecognition accuracy improves. When the number of train-ing data sets is three, the recognition precision, recall andF1 score are 85.6%, 83.6%, and 84.5%, respectively. Whenmore than three sets of sensor data are used, the recognitionaccuracy remains stable, meaning that the marginal gain inthe system performance is small. This experiment showsthat to have high-quality toothbrushing monitoring, a userjust needs to brush his or her teeth carefully for three timesto allow the system to learn. After that, the system will beable to provide high-quality monitoring.

7.3 Incorrect Toothbrushing Technique Detec-tion

The evaluation results are shown in Figure 15. We cansee that when only one set of toothbrushing data is used intraining, the recall rate is 74%, and the TNR is 50%. Asthe number of training data sets increases to 4, the recallrate stabilizes at 90%, while the TNR gradually increases to66%. When at least four sets of training data are provided,the system can maintain a high recall rate of 90%. Since therecall rate is high, the users will be able to brush continu-ously without being disturbed by false alarms frequently, aslong as they are brushing using the Bass technique.

We can also see that the true negative rate ranges from50% to 66%, meaning that the algorithm can miss incor-

1 2 3 4 5 6 7Number of Training Data Sets

0.4

0.5

0.6

0.7

0.8

0.9Accuracy

RecallTNR

Figure 15: Incorrect Toothbrushing Technique Detection

Avg. (mAh) Std. Dev. (mAh)

Training 3.3 0.63Monitoring 3.9 0.60Google Fit 2.8 0.53

Table 1: Energy Consumption for Each Toothbrushing Ses-sion

rect brushing techniques. Although the system may missincorrect gestures from time to time, its detection accuracyis high enough to detect long-term incorrect toothbrushinghabits. This is because each time the user brushes in the in-correct technique for one second, the system on average hasa probability of 66% of detecting it. If the incorrect tooth-brushing technique is frequently repeated, the probability ofthe system identifying at least one instance of such incorrectbrushing goes up.

7.4 Impact on Battery LifeOne resource constraint of our system is in the smart-

watch because it has smaller battery capacity (310mAh).In this section, we conducted experiments to test the im-pact of our system on the watch battery life. Using threesets of the toothbrushing monitoring system, we repeatedthe toothbrushing monitoring process for five times each,and measured the energy consumptions using the batteryhistorian tool [2]. The results are shown in Table 1. Dur-ing the training phase, the system did not execute recogni-tion algorithms. Instead, the system recorded sensor dataand uploaded the data to the cloud after each toothbrushingsession. In the monitoring phase, besides recording and up-loading sensor data, the system also conducted recognitionalgorithms in real-time every second, and sent the recog-nition results to the tablet for display. We can see thatunder the highest workload, the smartwatch consumed 3.9mAh per toothbrushing session. Since each user brushed forabout 2 minutes for twice a day, normal daily toothbrushingwith our system consumes less than 5% of the smartwatchbattery capacity.

We also tested the energy consumption rate of anothercommonly used activity recognition app for smartwatch, theGoogle Fit. This app continuously recorded the value ofthe step sensor and display the results. We run this appon three smartwatches for five times, and each time lastedfor two minutes. We found that the smartwatches on av-

1 2 3 4 5 6 7 8 9

User ID

0

0.05

0.1

0.15

0.2

0.25

0.3

Perc

enta

ge E

rror

Figure 16: Stroke Counting Accuracy

erage consumed 2.8 mAh, with standard deviation of 0.53mAh. A possible reason for the lower energy consumptionwas that fewer computation tasks were executed than oursystem. Besides, we also tested the idle energy consumptionof the smartwatch. When a smartwatch was left unattendedto display time only, the battery life was about 110 hours.These results showed that in terms of energy consumption,our system had a similar performance to those in the mar-ket, and would not seriously affect the smartwatch batterylife.

7.5 Case StudyTo evaluate the effectiveness of our smart toothbrushing

system in improving dental hygiene, we have recruited 12volunteers and conducted a user study. We invited a dentistfrom a local hospital to check plaque level of every toothsurface for each volunteer. Each subject chewed a plaquedisclosing pill, which highlighted plaques on teeth. The den-tist recorded the plaque levels using the Turesky ModifiedQuigley-Hein Plaque Indexes (TQHPI) [12] before and afterthe subjects brushed their teeth. Each subject was giventhe instructions on the Bass technique and was using oursystem during toothbrushing.

This study consisted of two parts. In the first experiment,9 volunteers brushed each of their tooth surfaces with thirtybrushing strokes. We evaluated the effectiveness of our sys-tem in estimating toothbrushing effort. In the second exper-iment, 3 volunteers brushed teeth with different numbers ofstrokes for each surface. We assessed the relation betweentoothbrushing effort and the plaque removal quality.

We note that the result of this set of experiments canbe affected by a few factors, such as insufficient chewing ofplaque disclosing tablets, personal toothbrushing habit, andamount of toothpaste. Fortunately, the obtained result wasconsistent and therefore served the purpose of demonstrat-ing the potential value of our smart toothbrushing system.

7.5.1 Counting Toothbrushing StokesIn the first experiment, we tested the effectiveness of the

toothbrushing stroke counting algorithm described in Sec-tion 5. The detection error was defined as |predicted number−ground truth|/|ground truth|.

The results are shown in Figure 16. For the 9 subjects,our algorithm on average achieves an error rate of 10.3%.This demonstrates its effectiveness in evaluating the tooth-brushing effort of the users.

For the subject 4, we can see that the error rate is as highas 30%. This is because the subject 4 was brushing teeth

00.20.40.60.81

1.21.4

Subject  A  -­‐ 100 Subject  B  -­‐ 60 Subject  C  -­‐ 35Average  P

lagu

e  Re

duction  

using  T

QHP

I

Subject  with  Estimated  Brushing  Effort  (%)

Figure 17: Tooth Plague Reduction vs. Measured ToothBrushing Effort

with his mouth closed. As a result, the sound collected bythe device was small and had low signal to noise ratio. Tosolve this problem, we plan to design a hybrid toothbrushingstroke counting algorithm that combines both acoustic sens-ing and motion sensing, which has better noise robustness.

7.5.2 Plaque ReductionFrom the previous experiment, we showed that our system

was able to count the toothbrushing strokes with small error(10.3%). In the second experiment, we tested how the num-ber of toothbrushing strokes were related to the tooth plaqueremoval rate. 3 subjects were asked to perform toothbrush-ing with different numbers of brushing strokes (10, 20, and30) on each surface, while they used our system to estimatetheir brushing effort. Our system estimated the effort oftoothbrushing by counting the number of brushing strokes.30 strokes on each surface was considered 100% brushing ef-fort in this analysis, the tests of 10 and 20 brushes on eachsurface were normalized accordingly.

The experimental result of this test is shown in Figure 17.We can see that the plague reduction increases as the ef-fort of toothbrushing increases. Insufficient toothbrushingsubject B (60% effort) and subject C (35% effort) only re-move 62.7% and 16.8% plague, comparing to that of sub-ject A (100%). These results show that first insufficienttoothbrushing can be reliably detected by our system. Andsecond, insufficient toothbrushing directly affect the effec-tiveness of the plaque removal. Even short term insufficienttoothbrushing activities do harm to our teeth as plaque ac-cumulates quickly and introduces gum diseases.

8. STATE OF THE ARTVarious techniques have been employed to perform gesture

recognition, such as radio waves [30], video [24], and wear-able sensors [9, 19, 31]. Among different wearable sensingdevices, wrist watch has received a lot of attention. In pa-per [14, 22], the authors studied the hand hygiene quality as-sessment using wrist-worn devices. They achieved good per-formance in recognizing correct hand-washing gestures. Theauthors of [27] designed probabilistic models to successfullyrecognize smoking gestures with high accuracy. In [23], theauthors applied Dynamic Time Warping (DTW) to matchthe inquiry data to the model data in the training sets. Theyachieved good performance (over 98% of accuracy) in clas-sifying eight gestures. These works provided valuable in-sights and demonstrated the feasibility of complex gesturerecognition using wrist-worn devices. However, their pri-mary goal was to conduct gesture recognition. In our work,besides recognition of gestures, we also assessed the quality

of toothbrushing. Moreover, we developed a novel magneticsensing algorithm to capture the motion of a hand tool, thetoothbrush.

In order to monitor toothbrushing, several systems havebeen designed. In [10], authors designed a vision-basedmotion tracker for toothbrushing. In [20], the authors de-veloped a hidden Markov model to recognize acoustic pat-terns during toothbrushing. Different from these works, wemainly focused on motion recognition for both toothbrushand brushing gestures. Also, our approach recognized all16 tooth surfaces with high accuracy, whereas their solutiononly recognized four different tooth quads.

There have been many research works on electric tooth-brushes. In [33], the authors presented evidence that Os-cillating Rotating Pulsating toothbrushes achieved effectivehygiene result. The authors of [34] explored the use of pres-sure sensors on the electric toothbrushes. The authors of [21]used acceleration and magnetic sensors in the electric tooth-brush to detect the motion and orientation of the tooth-brush. Our magnetic sensing design was related to but dif-ferent from the designs proposed in these works. To sensethe motions of the toothbrush, previous approaches gener-ally installed various sensors on the toothbrush, while ourapproach was to install a magnet on the toothbrush to makeit sensible by the magnetic sensor on the smartwatch. Ourapproach significantly reduced the cost and the system com-plexity, and improved the system robustness to the change ofgeomagnetic field. Besides the novelty in design, our systemalso tracked the toothbrushing quality and detected incor-rect toothbrushing gestures, which were not addressed inprevious studies.

9. CONCLUSION AND FUTURE WORKAs the most important oral hygiene activities at home,

daily toothbrushing can effectively prevent tooth decay andgum diseases. To accurately monitor toothbrushing compli-ance and effectiveness, we developed a novel sensing systemusing an off-the-shelf toothbrush and a wrist watch. Exper-imental results with 12 users for up to 3 weeks showed thatour solution reliably recognized the gestures for all 16 brush-ing surfaces using the Bass technique with a high accuracy.

In the future, we plan to investigate the recognition ofdifferent incorrect toothbrushing, which is important to helpusers form good toothbrushing habits. We will apply thedeep learning based algorithms to capture the subtle changesin the data from the magnetic sensor and gravity sensor dueto the incorrect toothbrushing gestures. We are workingwith the dental professionals to conduct a rigorous clinicalstudy on the effectiveness of our system with real patients.We will conduct a control group study, where one group ofusers will brush teeth with the help of our system, whilethe others will brush normally. By comparing the plaqueremoval quality in both groups, we can show whether or notthe system can improve dental hygiene.

10. ACKNOWLEDGMENTSThis work is supported in part by NSF grants CNS 153608-

6, CNS 1463722, IIS 1460370 and CNS 1553273. We thankthe anonymous reviewers and our shepherd, Lama Nachman,for their insightful comments. We are grateful to GuoliangXing, John Stankovic, Jie Gao, Wenting Lin D.M.D. andMaria Ryan D.M.D. for valuable discussions.

References[1] Amazon web service. https://aws.amazon.com. Ac-

cessed: 2016-08-24.

[2] The battery historian. https://github.com/google/battery-historian. Accessed: 2016-09-11.

[3] Nexus 7(2013). https://www.asus.com/us/Tablets/Nexus 7 2013/. Accessed: 2016-08-24.

[4] Samsung gear live. http://www.samsung.com/us/support/gear/gear-live-support/. Accessed: 2016-08-24.

[5] H. Abdi and L. J. Williams. Principal component anal-ysis. Wiley Interdisciplinary Reviews: ComputationalStatistics, 2(4):433–459, 2010.

[6] A. D. Association. Tackling tooth decay. Journal ofAmerican Dental Association, 2013.

[7] D. Bartlett and P. Shah. A critical review of non-carious cervical (wear) lesions and the role of abfrac-tion, erosion, and abrasion. Journal of Dental Research,85(4):306–312, 2006.

[8] P. N. Bennett. Assessing the calibration of naive bayesposterior estimates. Technical report, DTIC Document,2000.

[9] S. Berman and H. Stern. Sensors for gesture recogni-tion systems. Systems, Man, and Cybernetics, Part C:Applications and Reviews, IEEE Transactions on, 2012.

[10] Y.-C. Chang, J.-L. Lo, C.-J. Huang, N.-Y. Hsu, H.-H.Chu, H.-Y. Wang, P.-Y. Chi, and Y.-L. Hsieh. Play-ful toothbrush: Ubicomp technology for teaching toothbrushing to kindergarten children. In Proceedings of theSIGCHI Conference on Human Factors in ComputingSystems, CHI ’08, 2008.

[11] H.-T. Cheng, F.-T. Sun, M. Griss, P. Davis, J. Li, andD. You. Nuactiv: Recognizing unseen new activitiesusing semantic attribute-based learning. In Proceedingof the 11th Annual International Conference on Mo-bile Systems, Applications, and Services, MobiSys ’13,pages 361–374, New York, NY, USA, 2013. ACM.

[12] M. Cugini, M. Thompson, and P. Warren. Correlationsbetween two plaque indices in assessment of toothbrusheffectiveness. J Contemp Dent Pract, 7(5):1–9, 2006.

[13] C. for Disease Control and Prevention. National centerfor health statistics. Health, United States, 2014.

[14] V. Galluzzi, T. Herman, and P. Polgreen. Hand hy-giene duration and technique recognition using wrist-worn sensors. In Proceedings of the 14th InternationalConference on Information Processing in Sensor Net-works, IPSN ’15, pages 106–117, New York, NY, USA,2015. ACM.

[15] C. Ganss, N. Schlueter, S. Preiss, and J. Klimek. Toothbrushing habits in uninstructed adults: frequency, tech-nique, duration and force. Clinical oral investigations,13(2):203–208, 2009.

[16] J. Gibson and A. B. Wade. Plaque removal by the bassand roll brushing techniques. Journal of periodontology,48(8):456–459, 1977.

[17] M. Hammadeh and J. Rees. The erosive susceptibility ofcervical versus occlusal enamel. The European journalof prosthodontics and restorative dentistry, 9(1):13–17,2001.

[18] I. Jolliffe. Principal Component Analysis. John Wiley& Sons, Ltd, 2014.

[19] B. Kellogg, V. Talla, and S. Gollakota. Bringing ges-ture recognition to all devices. In 11th USENIX Sym-posium on Networked Systems Design and Implementa-tion (NSDI 14), 2014.

[20] J. Korpela, R. Miyaji, T. Maekawa, K. Nozaki, andH. Tamagawa. Evaluating tooth brushing performancewith smartphone sound data. In Proceedings of the 2015ACM International Joint Conference on Pervasive andUbiquitous Computing, pages 109–120. ACM, 2015.

[21] Y. J. Lee, P. J. Lee, K. S. Kim, W. Park, K. D. Kim,D. Hwang, and J. W. Lee. Toothbrushing region detec-tion using three-axis accelerometer and magnetic sen-sor. IEEE Transactions on Biomedical Engineering,2012.

[22] Y. Li, X. Chen, X. Zhang, K. Wang, and Z. Wang.A sign-component-based framework for chinese signlanguage recognition using accelerometer and semgdata. Biomedical Engineering, IEEE Transactions on,59(10):2695–2704, Oct 2012.

[23] J. Liu, L. Zhong, J. Wickramasuriya, and V. Vasude-van. uwave: Accelerometer-based personalized gesturerecognition and its applications. Pervasive and MobileComputing, 5(6):657 – 675, 2009. PerCom 2009.

[24] R. Lockton and A. W. Fitzgibbon. Real-time gesturerecognition using deterministic boosting. In in Proceed-ings of the British Machine Vision Conference (BMVC

aAZ02, 2002.

[25] I. MacGregor and A. Rugg-Gunn. A survey of tooth-brushing sequence in children and young adults. Jour-nal of periodontal research, 14(3):225–230, 1979.

[26] R. S. Manly and F. Schickner. Factors influencing testson the abrasion of dentin by brushing with dentifrices.Journal of Dental Research, 23(1):59–72, 1944.

[27] A. Parate, M.-C. Chiu, C. Chadowitz, D. Ganesan, andE. Kalogerakis. Risq: Recognizing smoking gestureswith inertial sensors on a wristband. In Proceedingsof the 12th Annual International Conference on Mo-bile Systems, Applications, and Services, MobiSys ’14,pages 149–161, New York, NY, USA, 2014. ACM.

[28] B. T. Piotrowski, W. B. GILLETTE, and E. B. HAN-COCK. Examining the prevalence and characteristicsof abfractionlike cervical lesions in a population of usveterans. The Journal of the American Dental Associ-ation, 132(12):1694–1701, 2001.

[29] M. Poyato-Ferrera, J. Segura-Egea, and P. Bullon-Fernandez. Comparison of modified bass technique withnormal toothbrushing practices for efficacy in supragin-gival plaque removal. International journal of dentalhygiene, 1(2):110–114, 2003.

[30] Q. Pu, S. Gupta, S. Gollakota, and S. Patel. Whole-home gesture recognition using wireless signals. In Pro-ceedings of the 19th Annual International Conferenceon Mobile Computing & Networking, MobiCom’13, 2013.

[31] R. I. Ramos-Garcia and A. W. Hoover. A study oftemporal action sequencing during consumption of ameal. In Proceedings of the International Conference

on Bioinformatics, Computational Biology and Biomed-ical Informatics, BCB’13, pages 68:68–68:75, New York,NY, USA, 2013. ACM.

[32] B. Scholkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, J. C. Platt, et al. Support vector method fornovelty detection. In NIPS, volume 12, pages 582–588,1999.

[33] O.-B. E. Toothbrushes. The scientific evidence behindoscillating-rotating-pulsating brush technology. 2013.

[34] P. Walters, R. Bartizek, and A. Biesbrock. A studyto assess the use of the pressure sensor in the oral-btriumph with smartguide. 2008.