A New Writing Experience :Finger Writing in the Air Using a Kinect SensorXin Zhang, Zhichao Ye, Lianwen Jin,Ziyong Feng, and Shaojie Xu

MultiMedia, IEEE, 2013

FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR

Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu

IEEE International Conference onMultimedia and Expo Workshops (ICMEW), 2013

2

Outline• Introduction • Related Work• Proposed Method• Experimental Results• Conclusion

3

Introduction

4

Introduction• So far most of writing systems still rely on:

• Keyboard• Touch screen• …(Extra devices)

• Essential goal of HCI:

• Making interaction between user and computer more natural

5

Introduction• In this paper:

• Propose a finger-writing-in-the-air system (based on Kinect):

• Using depth, color and motion information

• Real-time

• User-friendly and unconstrained

6

Related Work

Related work• Hand Segmentation

• Skin color:• Gaussian (mixture) model[2]

• Illumination and hand-face overlapping

• Depth:• noise

• Motion:• Motion Cue[3]

• The hand should be the most distinct moving object.

X

X

X

7

Related work• Fingertip Detection

• Curvature[6]

• Template matching[1]

• Geodesic distance

[1] L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of JCSC, 16(3):421–436, 2007.

[2] S. L. Phung, A. Bouzerdoum, and D. Chai. Skin segmentation using color pixel classification: Analysis and comparison. IEEE Trans. on PAMI, 27:148–154, 2005.

[3] Jonathan Alon, Vassilis Athitsos, Quan Yuan and Stan Sclaroff. A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation. IEEE Trans. on PAMI, 31:1685–1699, 2009.

[6] D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Journal of ETRI, 33(3):415–422, 2011.

8

Related work• Writing-in-the-air system [10]:

Hand Segmentation

Data Conversion

Region Clustering

Fingertip Identification

Arm point

Fingertip

K-means

[10] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye and WeixinYang. Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in the Air System. In Proc. of IEEE ICIMCS, 2012.

9

10

ProposedMethod

Flow Chart• Hand Segmentation Fingertip Detection

11

12

• DSB-MM segmentation algorithm

Hand Segmentation

13

• Depth Model• Solve the issues:

• lighting • hand-face overlapping• moving background

• Hand D:

Hand Segmentation

R(n) : hand region at frame nω : : growth factor

depth

↑

↑

Hand Segmentation• Depth Model

A static hand A moving hand

14

15

• Skin Model• YCbCr color space

• Quantify Y Component into three regions:• Bright

• Normal

• Dark

• Gaussian classifier[2]:

Hand Segmentation

Reduce the storage size

skin

Non-skin

: mean vector of the i-th skin class covariance of the i-th skin class mean vector of the i-th non-skin covariance of the i-th non-skin class

(Squared Mahalanobis distance)

Hand Segmentation• Skin Model

Color Image Depth Model Skin Model Depth + Skin

16

17

• Background Model• Codebook background model[8] •

Hand Segmentation[8] K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005.

18

• Background Model• Codebook background model[8] •

Hand Segmentation

Color image A Color image BForeground result A Foreground result B

19

• DSB-MM segmentation algorithm

Hand Segmentation

20

• DSB-MM segmentation algorithm• Each model should have different reliabilities.• Adaptive voting system

• A pixel is kept as hand pixel by

Hand Segmentation

21

• Artificial Neural Network (ANN) • (1) All the models contribute to the final result.• (2) None of them is absolutely reliable.

Hand Segmentation

“1 0 0”, “0 1 0” or “0 0 1” representing 1/3, 1/2 or 2/3

Training:resilient back propagation algorithm (RPROP)

22

Hand Segmentation

Origin Depth Skin Background Mixture

Flow Chart• Hand Segmentation Fingertip Detection

23

24

• Side-mode & Frontal-mode

Fingertip Detection

-- (Red) : Side-modeㄧ (Blue) : Frontal-mode

25

• Side-mode

• Fingertip : the farthest point from the arm point

• Palm point: • Ellipse fitting technique (center point)

• Arm point: • The center of the increased region

Fingertip Detection

26

• Side-mode• The farthest distance to the arm point:

• Side-Mode Criterion:

Fingertip Detection

27

• Frontal-mode• Fingertip : the point with the smallest depth value

Fingertip Detection

28

ExperimentalResults

29

Experimental Results• Intel Core i5-2400 CPU

• 3.10 GHz and 4 Gbytes of RAM

• 20 frames per second(fps)

• 375 videos(44522 frames)

• Recognition of the classifier:• 6763 frequently used Chinese character• 26 English letters (upper case & lower case)• 10 digits

30

Experimental Results• Finger-writing character recognition

• Linking all detected fingertip positions + mean filter• Modified quadratic discriminant function (MQDF) character classifier[9]

[9] T. Long and L. Jin. Building Compact MQDF Classifier for Large Character Set Recognition by Subspace Distribution Sharing. Pattern Recognition, 41(9):2916-2926, 2008.

31

Experimental Results• Error distance (Fingertip detection):

32

Experimental Results

33

Experimental Results

34

Conclusion

35

Conclusion• Propose a real-time finger-writing-in-the-air system

• Hand Segmentation:• Depth + Skin + Motion• Adaptive depth threshold of hand region

• Fingertip Detection:• Side-mode• Frontal-mode