Playful Toothbrush CHI 2008 presentation

1

Playful Toothbrush: Ubicomp Technology for Teaching Tooth Brushing to Kindergarten Children

CHI 2008, Florence Italy

Yu-Chen Chang, Chao-Ju Huang, Hao Chu, Peggy ChiNational Taiwan University

Jin-Ling Lo, Nan-Yi Hsu, Hisn-Yen Wang, Ya-Lin HsiehNational Taiwan University Hospital

2

Child Behavior Training

• Child behavior training as important but challenging parental responsibility– Potty training, self-dressing, cleaning room, self-

feeding, tooth brushing (this work).

3

Persuade Children to Change Behaviors

• The most common form of parental persuasion is verbal persuasion.– “If you don’t brush your teeth properly and thoroughly, you are not

allowed to go to sleep”.• Not effective, why not?

– Verbal persuasion alone lacks proper incentive to motivate children

4

A Case Study: Child Tooth Brushing

• Target kindergarten children (5-6Y)

– Learn to care for their own oral hygiene– Average 5Y children brush only 1/4 of teeth (Rugg-Gunn)– A common scenario: candies → improper brushing → cavities → dent

al clinic

5

Play-based Occupational Therapy

• Pediatric occupational therapy (O.T.)– Leverage the desire of children to play to induce their behavior change

willingly. • Children may not like to brush teeth, but like to play.• Add playfulness (game) into the brushing activity

– Effective – full of toys in pediatric O.T. clinic.

6

Play-Based O.T. Limitations

(1) Children treated in clinics during regular office hours (NTU Hospital)– Many child behavior problems not observable to therapists

• Eating (dinner time), brushing (before bedtime), etc.• Effective treatment is difficult.

(2) Train general performance skill rather than specific functional skill– Hand dexterity skill vs. tooth brushing skill– Improvement in general performance skills may not translate into the

target functional skills

7

HCI/UBICOMP Opportunities

• Embed digital technology into a child naturalistic living environment– Sensing to observe child behavior anytime, anywhere– Game playing to influence child behavior anytime, anywhere

• Occupational therapist perspective: – From treatment clinic– To the child actual living environment (functional behaviors)

• HCI/UBICOMP perspective:– From sense and track behaviors – To engage children to change behaviors

• Also called Persuasive technology (by Fogg, King, and others)

8

Playful Toothbrush Goals

• (Child) Dislike brushing– Make brushing more attractive

• (Child) Habitual brushing but not properly– Teach proper brushing skill

• Not replace adults’ supervision– Extend their effectiveness– Used together can overcome young

children’s limited physical and cognitive abilities, such that they can successfully learn proper brushing

9

• Sensing– Camera vision to detect children’s

brushing actions– Brushing actions become game inputs

• Playing– An interactive game of brushing teeth– Start with a mirror image of dirty teeth.– Brushing own teeth maps to cleaning

the same virtual teeth in the mirror image.

• Demo video

Playful Toothbrush

Brush extension

Web camera for tracking brushing motions

Camera #2 to record videos for later human analysis

Brushing game

10

Play-based Occupational Therapy Model

• Playful toothbrush is a treatment program / play activity (three steps).

Volition Performance Habituation

Bring enough enjoyment to attract a child to participate in the target activity.

Ensure a child will have a successful experience. Set appropriate level of difficulty.

Apply reinforcement to reward good performance, so increase change of repeating desirable behavior. Enough times to become a habit.

11

Design Considerations

• Enjoyment– Bring enough fun to attract children to brushing– Associate brushing with game playing and having fun

• Engagement– Simple Interaction (Not all young children can learn to operate complex devices)

– Use their natural brushing actions as game inputs • Automation

– Help children learn tooth brushing skill and internalize as habit• In brushing case, internalizing means doing without much thinking)

– Advice adults not to interrupt children and their own motor planning, necessary for internalization of brushing motor skill

12

Playful Toothbrush Prototype (two main components)

Vision-based brushing tracker

Brushing game

13

Vision-based Brushing Tracker• Use one camera to detect brushing

– Recognize brushing of 24 different teeth surface areas (or granularity of 2 adjacent teeth surfaces)

• Mouth closed during brushing – Bristle-teeth contact area not visible to an

external camera• Brush extension

– Each of four faces has unique LED-pattern– Used as marker to assist camera-vision

recognition– Children told not to block brush extension from the camera

14

How to recognize a brushing stroke on a specific teeth surface area?

• Take an example (brushing the outer surface of frontal teeth)

• Three features determines this brushing stroke– Bristle rotated toward the face– Brush oriented parallel to the face– Back-and-forward motion vector (parallel to the face)

15

Computer Vision Technique

• Use the brush extension marker to reconstruct– Bristle rotation angle (x-axis)– Brush orientation angle (z-axis)– Brush back-n-forward motion vector

• Infer the brushing stroke and its target teeth area

x (θroll)

z (θyaw)

y

brush extension

16

Accuracy Test (vision-based brushing tracker)

• 13 kindergarten children (72-81 months)• Recorded 48 minutes of their brushing videos• Compare human-read brushing strokes (ground-truth) and

machine-recognized brushing strokes• Machine-recognized accuracy 90%

– Need not be perfect. Occasional error in games has little effect.

17

Childrenfalse negative rate

(frame) (%)false positive rate

(frame) (%)

1 602/6521 9.2% 47/483 9.7%

2 880/5958 14.8% 49/332 14.8%

3 331/5130 6.5% 209/1304 16%

4 617/5270 11.7% 20/380 5.3%

5 209/4813 4.3% 60/634 9.5%

6 90/4551 2% 0/80 0%

7 290/5520 5.3% 0/196 0%

8 579/4302 13.5% 206/1288 12.2%

9 239/7104 3.4% 23/688 3.3%

10 32/3532 0.9% 115/1298 8.6%

11 695/6847 10.2% 111/630 17.6%

12 118/4974 2.4% 0/168 0%

13 241/6493 3.7% 41/343 12.0%

Average 6.8% 8.4%

18

Brushing Game

• Start with a virtual mirror image of the children’s own uncleaned teeth– Grouped into 24 teeth surface areas

• “1” appears next to the first cleaning target– 7 layers of plaques were drawn, requiring 7

brushing strokes– Each brushing stroke removes one layer of p

laque– Combine audio feedback: Do-Re-Mi-Fa ..– No game response for brushing other teeth

areas

19

Brushing Game

• After 1st area is complete, “2” appears next to the 2nd target area.

• Applause at the game completion– Visual-audio feedbacks provide a sen

se of accomplishment • Enforce a brushing sequence (Stillm

an’s brushing method)– Upper right, upper left, the lower righ

t, and lower left • Two benefits of sequencing

– Ensure brushing all teeth– Repeat this sequence enough time ->

“do without thinking”

2

23

20

User Study

• Two questions guided our user study– How effective is the Playful Toothbrush in improving the

brushing skills of kindergarten children?– What aspects of brushing behaviors were affected by

Playful Toothbrush?

21

User Study Test Subjects

• 13 young children (72 – 81 months) from a NTUH-run kindergarten class– 5 girls and 8 boys

• Written informed consent from parents• Tooth brushing is required after meals or snacks

• Setup– Install our system at the kindergarten’s restroom– A video camcorder to record children’s brushing sessions

22

User Study Procedure (11 days spanned over 3 weeks)

• Pre-trial (one day)– Familiarize with the therapists and our system

• Pretest (two days)– Brushed with their own toothbrushes – Established a baseline behavior

• Training (five days)– Brushed with our playful toothbrush– A trained therapist helps children understand the brushing game

• Posttest (two days)– Brushed with their own toothbrushes– Measured behavior improvement from pretest

• One week follow up (one day)– Brushed with their own toothbrushes– Measured behavior retention after a week

23

User Study: Brushing Effect

• Teeth cleaning effectiveness– Red plaque disclosing dye before each

brushing session (dye attached the plaque)

– Oral exam (counted teeth surfaces with dye before and after brushing)

– Plaque index• number of teeth surfaces with plaque /

total number of teeth surfaces– Cleaning effect

• Reduction of Plaque Index from before to after brushing

24

User Study: Brushing Behavior

• By analyzing/coding brushing videos, measured 3 aspects of brushing behaviors:– Length of brushing time– Number of brushing strokes on each of 24 teeth areas– Total number of brushing strokes

25

User Study: Cleaning Effect Results

Before brushing Mean(SD)

After brushingMean(SD)

Cleaning effectMean (SD)

Pre-test

Day 3 0.69(0.25) 0.37(0.18) 0.32(0.21)

Training

Day 4 0.76(0.14) 0.09(0.10) 0.67(0.15)

Day 6 0.68(0.28) 0.04(0.05) 0.64(0.26)

Day 8 0.79(0.18) 0.11(0.10) 0.68(0.17)

Average 0.74(0.19) 0.08(0.06) 0.66(0.17)

Post-test

Day 9 0.83(0.11) 0.16(0.10) 0.67(0.13)

Day10 0.86(0.16) 0.15(0.08) 0.70(0.15)

Average 0.85(0.10) 0.16(0.07) 0.66(0.28)

Follow-up

Day 11 0.88(0.14) 0.18(0.10) 0.70(0.14)

26

Number of brushing strokes Mean(SD)

Number of unbrushed teeth areas Mean(SD)

Brushing time Mean(SD) (sec)

Pretest

Day 2 212.31(137.77) 11.31(5.25) 84(53)

Day 3 168.62(157.03) 13.46(5.35) 67(46)

Average 190.46(138.38) 12.39(4.75) 76(45)

Training

Average

Posttest

Day 9 281.69(120.66) 7.46(4.89) 137(41)

Day 10 214.31( 71.91) 9.46(5.12) 99(31)

Average 248.00(87.12) 8.46(4.82) 118(30)

Follow-up

Day 11 239.62(107.48) 8.31(5.07) 120(36)

User Study: Brushing Behavior Results

27

Result Summary

• (Pretest) Child subject failed to clean 37% of their teeth surfaces• After 5 training days, improvements in

– Teeth cleaning effectiveness– Number of brushing strokes– Length of brushing time– Coverage of brushed teeth areas

28

Conclusion

• Demonstrate a case study of applying HCI/UBICOMP technology in play-based occupational therapy

• For teaching young children proper tooth brushing, user study results were encouraging

• Other similar repetitive development tasks for young children (self-feeding, potty training, cleaning room, etc.) – This technique can make training attractive and simple for

adults/children.

29

Examples of Other Persuasive Technologies

30

• Sense to recognize behavior

– Weight sensor underneath the tray to sense eating actions

– Eating actions as game input• Play to engage behavior change

– Interactive games: coloring cartoon character or penguin fishing

Case Study: Playful Tray Encourage good eating habit in young children

31

• Sense to recognize behavior

– Tilt sensor to detect drinking actions– Drinking actions are game inputs

• Play to engage behavior change

– Game metaphor: hydrating/dehydrating body -> watering/drying a tree

Case Study: Mug-Tree Encourage healthy habit of drinking fluid regularly

32

• Sense to recognize dress & dressing color

– Camera and Computer Vision• Play to engage behavior change

– Playful explore & experiment with how different clothing color look on people

• CHI 2008 poster on Tuesday

Case Study: ChroMirror Persuade people to explore more colorful dressing

33

Thanks

Q & A

34

Lesson #4

• Mediate, not automate – Mediation: not to replace children’s efforts but make the

experience of performing more enjoyable for children– Mediation works better than automation in this case.

35

Lesson #1

• Unpredictable behaviors from young children – Make activity recognition difficult

• Randomly switch from left to right hands, swing brush wildly, head movement brushing the brush …

Observe child behaviors carefully before designing and programming the system

36

Lesson #2

• Activity grading– Children have different physical and cognitive capabilities

• Some learned fast (not challenging enough); some learned slowly (frustration)

– Different levels of challenges fitting to each child’s ability

“Play-based O.T. is about the experience of performance, or the fitness between the level of challenge in an activity

and a child’s physical and cognitive capabilities.”

37

Lesson #3

• Personalization & customization– Easily personalized and customized to environmental or

human factors (preferences of children, changing performance of a child, different deployment environments).

– Left hand vs. right hand – Lighting condition– Child height– Preferences about game characters

38

Implementation Details

• Toothbrush extension weight– Not too heavy for children to cause inconvenience in brushing

• Prevent child from holding the brush extension– Block the brush extension from the overhead camera– Solution: a protruding cap as separator

• Camera placement– Not too high such that camera cannot see the brush extension clearly.– Not too low such that children can switch the brush extension outside

camera view