Automatic user state recognition for hand gesture based low cost television control system

Automatic User State Recognitionfor Hand Gesture Based Low-CostTelevision Control System

Automatic User State Recognitionfor Hand Gesture Based Low-CostTelevision Control System

2 CONTENTS

INTRODUCTION

HAND FREE CONTROL SYSTEM

PROPOSED SYSTEM

COMPONENTS OF AUSR SYSTEM

USER STATE DEFINITION

USER STATE INITIALIZATION

USER STATE TRANSITION

HAND GESTURE RECOGNITION

EXPERIMENT RESULTS

CONCLUSION

REFERENCES

AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM

3 INTRODUCTION

Innovation in TV display screen TV controller remains unchanged Fitt's Law Hand Free control system


4 HAND FREE CONTROL SYSTEM

Voice controlled devices

Hand gesture control system


5 PROPOSED SYSTEM

Low cost hand gesture in automatic manner


6

PROPOSED SYSTEM(contd..)

Automatic user state recognition(AUSR)

Ultra sonic distance array and camera module


7 IMPLEMENTATION OF THE SYSTEM


8 COMPONENTS OF AUSR SYSTEM

Hardware TVset-top boxsensors(RGB or depth cameras) ultrasonic distance array


9 COMPONENTS OF AUSR(contd…)

Embedded software componentsconnects sensors with TVProcess the sensor dataDrive the TV control process


10 USER STATE DEFINITION

Classification of user stateAbsentOther actionControllingWatching


11 USER STATE DEFINITION (contd..)


12 USER STATE INITIALIZATION

Action detection Distance detection Movement detection Single sensor based & sensor array based

action decision.


13 USER STATE INITIALIZATION (contd..)

L(t)=C.T(t)/2

L(t)=distance between object and sensor

C=sonic speed in air(340 m/s).

T(t)=time between the sensor emits and

receives the ultrasonic waves.

∆Lt =|L(t+P)-L(t)|



At = 0, ∆Lt <R (considering the threshold)

1,∆Lt ≥ R



Considering action sequence :

St=[∆ Lt-k+1 ,∆Lt-k+2 ,..., ∆Lt-1, ∆Lt ]

Based on threshold R for non intentional movements

SRt =[At-k+1 ,At-k+2 ,..,At-1 , At ]



Action status

A= 0, SUM(SRt )< Q

1, SUM(SRt )≥ Q

SUM(SRt )=∑ i=0K-1 At-i

S=2.√2.r=2√2.D.tgθ Action decision will be according to A= 0, max [A0 ,A1 ,A2 ,A3 ]=0 1, max [A0 ,A1 ,A2 ,A3 ]=1



Presentation detection

Face detection

Horizontal view angle α

α= arctg(W/2L)

arctg- inverse trigonometric function

W- width of tv screen

L-distance between user eye and TV screen.



Vertical view angle β

Β = arctg(H/2L)

H- height of TV screen Roll angle[φ] Lf ={F|-α≤Fh ≤α, -β ≤Fv ≤β and –φ ≤ Fr ≤φ }

B= o, if the face is not detected

1, if the face is detected







User state decision decided from results of action detection

and presentation detection.


22

USER STATE INITIALIZATION (contd..)

ALGORITHIM OF DECISION OF USER STATE

If (the intentional motion is detected)

If (the face watching TV is detected)

Set (A=1,B=1) - Controlling

Else

Set (A=1,B=0) - Other Action

Else if (the face watching TV is detected)

Set (A=0,B=1) - Watching

Else

Set (A=0,B=0) - Absent


23USER STATE INITIALIZATION (contd..)


24USER STATE TRANSITION If (the intentional motion is detected) If (A=1 and B=1)

Detect hand gesturesElse if (A=0 and B=0)

Set (A=1,B=0)Else if (A=1 and B=0)

If (the face watching TV is detected)Set (A=1,B=1)

ElseIf (the face watching TV is detected)Set (A=1,B=1)

Else if (the intentional motion is not detected for a long time)If (A=1,B=0)

Set (A=0,B=0)Else if(A=1,B=1)

Set (A=0,B=1)


25 USER STATE TRANSITION(contd..)


26 HAND GESTURE RECOGNITION

Hand gesture recognition module & camera activated when user state is controlling.

Gesture continuously detected in Tf

Tf is the threshold for restrict the time period permitted for the user to do hand gestures


27 EXPERIMENTAL RESULTS

User study

Out of 6 people 2young,2middle age and 2 old users are considered


28 EXPERIMENTAL RESULTS (contd..)

Automatic user state recognition


29 EXPERIMENTAL RESULTS (contd..)

Computational cost and power consumption


30 CONCLUSIONS

TV user’s states are defined according to finite state machine.

Camera system and gesture recognition is activated or closed by recognised user state.

Reduction in computational cost and power consumption



31 REFERENCES

[1]Shiguo Lian, Member, IEEE, Wei Hu, Kai Wang,” Automatic User State Recognition for Hand Gesture Based Low-Cost Television Control System”, IEEE Transactions on Consumer Electronics, Vol. 60, No. 1, February 2014.

[2]I. S. Mackenzie and W. Buxton, "Extending fitts' law to two dimensional tasks," in Proceedings of CHI'92, pp. 219-226, May 1992.

[3]S. K. Kim, G. H. Park, S. H. Yim, S. M. Choi, and S. J. Choi, “Gesture recognizing hand-held interface with vibrotactile feedback for 3D interaction,” IEEE Trans. Consumer Electronics, vol. 55, no. 3, pp.1169-1177, Aug. 2009.

[4]H. Heo, E. C. Lee, K. R. Park, C. J. Kim, and M. C. Whang, “A realistic game system using multi-modal user interfaces,” IEEE Trans. Consumer Electronics, vol. 56, no. 3, pp. 1364-1372, Aug. 2010.

[5]D. W. Lee, J. M. Lim, S. W. John, I. Y. Cho, and C. H. Lee, “Actual remote control: a universal remote control using hand motions on a virtual menu,” IEEE Trans. Consumer Electronics, vol. 55, no. 3, pp.1439-1446, Aug. 2009

32

Questions please??


THANK YOU

Engineering

Automatic user state recognition for hand gesture based low cost television control system