Bite detection and differentiation using templates of wrist motion

MS Defense Exam Soheila Eskandari

Committee members:Dr. Adam Hoover (chair)

Dr. John N. Gowdy Dr. Eric R. Muth

December 5th, 2013

Department of Electrical and Computer Engineering

2

Outline

◦Motivation and Background◦Methods◦Results◦Conclusions

3

Motivation

• One third of U.S. adults were overweight and another one third were obese in 2003-2004 (reported by NHANES)

• Cost associated with obesity was $117 billion in the US in 2000

Obesity treatments

Weight maintenance goal is to achieve: EI=EEThe problem is with the tools people use to measure EI

Mobile Health technologies• Mobile monitoring of the human electrocardiogram

(ECG)• Heart rate,• Breathing frequency,• Blood pressure variations,• Breathing amplitude.• Detection of different sleep phases

6

Wrist motion tracking Dong et al. [7,8] developed a wrist-worn device to

track wrist motion and measure the number of bites taken during a meal. Additional research showed that bites, automatically counted using this method, correlated with self-reported caloric intake at the meal level at 0.5.

Amft [1] developed a wrist-worn device with the primary objective of detecting drinking activities, the container used, and the fluid level.

Junker and Amft [1,2] presented a recognition system that used five inertial sensors located on the wrists, upper arms, and upper torso. Their research describes motion gestures based on the particular utensil used, establishing four gestures (cutlery, drink, spoon, hands).

7

Bite detection based on threshold method by wrist motion tracking

T1 and T2 : The roll velocities T3 : Time interval between the

first and second events of roll motion

T4 : Time interval between the end of one bite and beginning of the next bite

Tested on total of 276 subjects 22,383 bites True detection rate of 76% with

a positive predictive value of 87%

Adjusting the second timing threshold (T4): True detection rate of 82% and a positive predictive value of 82%

Threshold algorithm:

Let EVENT = 0Loop Let V_t = measured roll vel. at time t if V_t > T1 and EVENT = 0 EVENT = 1 Let s=t if V_t<T2 and T-s>T3 and EVENT = 1 Bite detected Let s=t EVENT = 2 if EVENT = 2 and T-s>T4 EVENT = 0

8

Template matching Determine similarity between templates and an

unknown signal Similarity by sum of the cross correlation coefficient:

and the value of absolute difference:

9

MethodsData collectionBite templatesBite differentiationBite detection

10

Data collectionData recorded in a cafeteria environment at

Clemson University (NIH grant 1R41DK091141-A1).

Cafeteria info: 800 guests, provides a wide range of foods and beverages, utensils, and containers.

Total data collected: 276 subjects (131 males and 145 females, ages from 18 to 75 years old, BMI from 17.4 to 46.2 , ethnically diverse)

11

Data collection tools:

Ground truth Total of 22,383 bites

Bite TemplatesDetermine the overall pattern and variability pattern of wrist motion of a biteCreated by :Using both the accelerometers and gyroscopes data• Averaging the motion data across all the bites in the

22,383 total ground truth bites• Over a six second window centered on the bite time• Templates of food and drink bites• Four different types of food bites: bites taken with a fork, bites taken with a spoon, food bites eaten using one hand food bites eaten using both hands

14

Bite differentiationRecognizing different types of bites using

template matching against the typical motion pattern

? ? ?

Algorithm:

• Minimum scoring template identifies the most closely matching bite

Bite detection

Detect the bites from other activities during a meal by template matching based on just roll motionSteps: Sum of absolute difference between a bite template and

the wrist motion data at every time step Detecting local minima Best template matched at the local minima position

Detected bite

Detected bite

Detected bite

Ground truth bites

Computer detected bites

18

ResultsBite templatesBite differentiationBite detection

19

Total bite templates

20

17,166 ground truth food bites3,185 bites drink bites

Food bites (17,166 bites) Drink bites (3,185 bites)

21

Food bites larger average motion in the Z and roll axesDrink bites larger average motion in the X and yaw axes

Food bites (17,166 bites) Drink bites (3,185 bites)

22

Drink bites longer (slower) motion than food bites in the yaw axis. Roll motion for drink bites is opposite to food bites, with negative roll preceding positive roll.

Food bites (17,166 bites) Drink bites (3,185 bites)

23Food bites (17,166 bites) Drink bites (3,185 bites)

Food bites opposite average motion with drink bites in roll axes

24Fork (8,764 bites)

Spoon (1,986 bites) Single hand (9,241 bites)

Both hand (2,441 bites)

Ax

Ay

Az

Yaw

Pitch

Roll

25

Bite differentiation

Ground truth

Computer detected

Food (Ax,Ay,Az,Yaw,Pitch,Roll)

Drink(Ax,Ay,Az,Yaw,Pitch,Roll)

Food 75%,72%,68%,72%,43%,64%

25%,28%,32%,27%,57%,36%

Drink 13%,10%,12%,40%,19%,5.6%

87%,90%,88%,60%,81%,94%

Accuracy 81%,81%,78%,66%,62%,79%Ground truth

Computer detected

Food

Drink

Food 70% 30%

Drink 5% 95%

Accuracy 83%

Bite differentiation of food and drink bites using all 6 motion axes.

Accelerometer and gyroscope motions confusion table for food & drink bites recognition.

26

Confusion matrices for the five types of bites according to utensil, for each axis Overall accuracy for recognizing for the 4 different types

of utensils :19-48% and Drink: 80%

Confusion Accelerometer motion axes.

Ground truth(Ax,Ay,Az)

Computer detected (Ax,Ay,Az)% Fork Spoon Drink Both

handSingle hand

Fork 23,20,21

49,56,51

4,5,6 13,9,10 12,10,1.4

Spoon 19,14,18

20,64,60

4.3,5,5 14,8.5,9 8,9,9

Drink 1,1,1 5,4.5,6 81,84,82

6,6,6.5 7,5,5

Both hand

8,6,7 30,36,31

18,28,37

28,18,16

17,12,10

Single hand

15,11,10

21,27,28

20,28,35

19,15,11

25,19,17

Accuracy 42,41,39 %

27

Confusion gyroscope motion axes.

Ground truth(Yaw,Pitch,Roll)

Computer detected (Yaw,Pitch,Roll)% Fork Spoon Drink Both

handSingle hand

Fork 42,14,49

31,33,17

14,25,8

6,13,12 7,14,14

Spoon 40,9,37

38,40,20

11,25,13

5,11,14 7,15,16

Drink 28,3,1.5

10,10,1.6

51,48,71

10,27,18

1.5,12,8

Both hand

41,6,4 17,19,6

28,31,40

9,31,32 6,13,19

Single hand

36,8,35

29,27,12

20,31,18

7,16,18 10,19,18

Accuracy 30, 31, 38

28

Confusion combining all 6 motion axes.

Ground truth(Ax,Ay,Az,Yaw,Pitch,Roll)

Computer detected (Ax,Ay,Az,Yaw,Pitch,Roll)% Fork Spoon Drink Both

handSingle hand

Fork 48 27 3 15 6

Spoon 30 38 6 21 5

Drink 1.5 1.5 80 10 8

Both hand

5 5 28 46 16

Single hand

33 13 14 22 19

Accuracy 46

29

Bite detectionTested on 22,383 total bitesDetection rate: 48%Positive predictive value: 75%No higher performance for different axes

anddifferent combinations of axes

30

Conclusions Food and drink bites appear to have different wrist motion patterns Different types of utensils for food bites also appear to have

different wrist motion patterns, however, they are not consistent enough to enable differentiation via template matching

Original threshold-based algorithm:  77% true detections, 86% PPVTemplate matching algorithm:  46% true detection, 75% PPVTemplate matching is too rigid for detecting bites; there is too much variability in appearance; interestingly, it yielded the close PPV in the threshold-based algorithm suggesting it might be useful for suppressing false positives

31

References[1] O. Amft, H. Junker, and G. Troster, \Detection of eating and drinking arm gestures using inertial body-worn sensors," in Proceedings of the ninth IEEE International Symposium on Wearable Computers, 2005, pp. 160-163.[2] O. Amft and G. Troster, \On-body sensing solutions for automatic dietary monitoring,"IEEE Pervasive Computing, vol. 8, no. 2, pp. 62-70, 2009.[3] G. Billington, Epstein, \Overweight, obesity, and health risk." Arch Intern Med,vol. 160, pp. 898,904- 2000.[4] M. Boninsegna and M. Rossi, \Similarity measures in computer vision," PatternRecognition Letters, vol. 15, no. 12, pp. 1255-1260, 1994.[5] C. Champagne, G. Bray, A. Kurtz, J. Monteiro, E. Tucker, J. Volaufova, andJ. Delany, \Energy intake and energy expenditure: a controlled study comparingdietitians and non-dietitians," Journal of the American Dietetic Association, vol.102, no. 10, pp. 1428-1432, 2002.[6] C. Ching, M. Jenu, and M. Husain, \Fitness monitor system," in Proceeding ofConference on Convergent Technologies for Asia-Pacic Region, vol. 4, 2003, pp.1399-1403.

32

[7] Y. Dong, \Tracking wrist motion to detect and measure the eating intake of free-living humans," Ph.D. dissertation, Electrical and Computer Engineering Department, Clemson University, 2012.[8] Y. Dong, J. Scisco, M. Wilson, E. Muth, and A. Hoover, \Detecting periods of eating during free living by tracking wrist motion," IEEE of Biomedical Health Informatics, 2013.[9] K. Flegal, M. Carroll, B. Kit, and C. Ogden, \Prevalence of obesity and trends in the distribution of body mass index among us adults, 1999-2010," The journal of the American Medical Association, vol. 307, no. 5, pp. 491-497, 2012.[10] J. Foreyt and W. Poston II, Overview and the future of obesity treatment. Springer, 1999.[11] C. Gagnadre, M. Billon, and S. Thuillier, \Fibre optic sensor for physiologicalparameters," Electronics Letters, vol. 34, no. 21, pp. 1991-1993, 1998.[12] C. Harland, T. Clark, and R. Prance, \High resolution ambulatory electrocardiographic monitoring using wrist-mounted electric potential sensors," Measurement Science and Technology, vol. 14, no. 7, pp. 923-928, 2003.[13] B. Howard and S. Howard, \Lightglove: Wrist-worn virtual typing and pointing,“ in Proceedings of IEEE Fifth International Symposium on Wearable Computers,2001, pp. 172-173.

33

[14] Z. Huang, \An assessment of the accuracy of an automated bite counting method in a cafeteria setting," Master's thesis, Electrical and Computer Engineering Department, Clemson University, 2013.[15] H. Junker, O. Amft, P. Lukowicz, and G. Troster, \Gesture spotting with bodyworn �inertial sensors to detect user activities," pattern Recognition, vol. 41, no. 6, pp. 2010-2024, 2008.[16] E. Kelly, Obesity: Health and Medical Issues Today. Greenwood Publishing Group, 2006.[17] S. Kumar, W. Nilsen, M. Pavel, and M. Srivastava, \Mobile health: Revolutionizing healthcare through transdisciplinary research." IEEE Computer, vol. 46, no. 1, pp. 28-35.[18] J. Lementec and P. Bajcsy, \Recognition of arm gestures using multiple orientation sensors: gesture classication," in Proceedings of the 7th IEEE International Conference on Intelligent Transportation Systems, 2004, pp. 965-970.[19] A. Mikhail, C. Frederic, and D. Philippe, \An algorithm for estimating all matches between two strings," INRIA, Tech. Rep., 2001.[20] C. Ogden, M. Carroll, L. Curtin, M. McDowell, C. Tabak, and K. Flegal, \Prevalence of overweight and obesity in the united states, 1999-2004," The journal of the American Medical Association, vol. 295, no. 13, pp. 1549-1555, 2006.[21] G. Ogris, T. Stiefmeier, H. Junker, P. Lukowicz, and G. Troster, \Using ultrasonic hand tracking to augment motion analysis based recognition of manipulative gestures," in proceedings of the Ninth IEEE International Symposium on Wearable Computers, 2005, pp. 152-159.

34

[22] W. H. Organization, \Overweight and obesity," http://www.who.int/ mediacentre/factsheets/fs311/en/index.html, 2008.[23] J. Salley, \Accuracy of a bite-count based calorie estimate compared to human estimates with and without calorie information available," Master's thesis, Psychology Department, Clemson University, 2013.[24] E. Sazonov and S. Schuckers, \The energetics of obesity: A review: Monitoring energy intake and energy expenditure in humans," IEEE Engineering in Medicine and Biology Magazine, vol. 29, no. 1, pp. 31-35, 2010.[25] D. Schmidt, R. Dannenberg, A. Smailagic, D. Siewiorek, and B. Biigge, \Learning an orchestra conductor's technique using a wearable sensor platform," in Proceeding of the 11th IEEE International Symposium on Wearable Computers, 2007, pp. 113-114.[26] A. Smailagic, D. Siewiorek, U. Maurer, A. Rowe, and K. Tang, \Ewatch: Context sensitive system design case study," in Proceedings of the IEEE Computer Society Annual Symposium on VLSI, 2005, pp. 98-103.[27] STMelectronics. (2013) Mems inertial sensor 3-axis linear accelerometer. http: //www.st.com/web/catalog/sense power/FM89/SC444/PF207281. [28] STMelectronics. (2013) Mems motion sensor 2-axis pitch and roll gyroscope. http://www.st.com/web/catalog/sense power/FM89/SC1288/PF248621.

35

[29] STMelectronics. (2013) Mems motion sensor 2-axis pitch and yaw gyroscope. http://www.st.com/web/catalog/sense power/FM89/SC1288/PF248616.[30] C. Sugimoto, H. Ariesanto, H. Hosaka, K. Sasaki, N. Yamauchi, and K. Itao, \Development of a wrist-worn calorie monitoring system using bluetooth," Mi-crosystem technologies, vol. 11, no. 8-10, pp. 1028-1033, 2005.[31] L. Terre, W. Poston II, and J. Foreyt, 12 Overview and the Future of Obesity Treatment. Springer, 2005.[32] N.Wellman and B. Friedberg, \Causes and consequences of adult obesity: health, social and economic impacts in the united states," Asia Pacic journal of clinical nutrition, vol. 11(s8), pp. S705-S709, 2002.

36

Thank you!

Questions?