WLSACONVERGENCE SUMMIT

ZERO-EFFORT CAMERA-ASSISTED CALIBRATION TECHNIQUES FOR WEARABLE MOTION SENSORS

JIAN WUUNIVERSITY OF TEXAS, DALLAS

Zero-Effort Camera-Assisted Calibration Techniques for Wearable Motion

Sensors

Jian Wu and Roozbeh Jafari

Embedded Signal Processing LaboratoryDepartment of Electrical Engineering

University of Texas at Dallas

Motivation & Background

• Wearable motion sensor Inertial measurement unit (IMU) 3-axis accelerometer measures gravity and accelerations and 3-axis

gyroscope measures angular velocities• Where are they used?

Navigation systems Health and wellness monitoring Activity tracking

• Forms Mobile phones Wearable devices Fitness trackers

3

Motivation & Background

• Activity recognition plays an important role in pervasive wellness and health-care monitoring applications.

• The activity recognition algorithms are often designed to work with a known orientation of sensors on the body.

4

0 100 200 300 400 500-2

-1

0

1

2

Sample number

Ac

ce

lera

tio

n(g

)

x-axisy-axisz-axis

0 100 200 300 400 500-1.5

-1

-0.5

0

0.5

Sample number

Ac

ce

lera

tio

n(g

)

x-axisy-axisz-axis

Sit-to-stand

Sit-to-stand

X

Y

X

Y

Motivation & Background

• The orientation may be different from which when the system was calibrated and trained. Accidental displacement during motion The user may not wear the sensor properly

• Calibration of the sensor orientation is necessary. Current approaches: Investigate the statistical distribution of the features, and

adjust the features adaptively for slight displacement. Ask the user to perform a certain activity which will

require extra efforts.

5

Motivation & Background

• We propose a calibration method by leveraging the camera information (i.e., Kinect) which requires zero effort from the user.

6

Kinect

Inertial Sensor

Arbitrary movement

Frames Definition

7

Human body frame (back view)

Xe

Xs

Ys

Zs

Ye

Ze

gravity

Kinect frame

Sensor local frameSensor earth frameSensor front face

Problem Formulation

8

• Yaw rotation: sensor rotational displacement around Y-axis of human body frame

• Roll rotation: sensor rotational displacement along the Z-axis of the human body frame

• Objective: calibration of the sensor yaw rotation (case 1) and roll rotation (case 2) w.r.t. the human body frame

Angle symbol Angle representationβ Yaw rotation of the Kinect frame w.r.t. the

sensor local frameγ Roll rotation of the sensor local frame w.r.t. the

Kinect frameα Yaw rotation of the Kinect frame w.r.t. the

human body frameφ Yaw rotation of the sensor local frame w.r.t.

the human body frame

Proposed Approach

• For a body segment rotation, inertial sensor and Kinect measure the same rotation, and thus has the minimum rotation distance.

• First step yaw search, the sensor yaw rotation w.r.t Kinect frame is calibrated.

• Second step roll search, the sensor roll rotation w.r.t human body frame is calibrated

• In the last step, the sensor yaw w.r.t human body frame is calibrated.

9

Algorithm Preliminaries

• Motion sensor: Orientation of the sensor local frame w.r.t. the sensor earth frame, which is denoted as .

• Kinect sensor: The orientation of body segment w.r.t Kinect frame.

• Rotation Distance:

10

First Step Yaw Search

• First step search for yaw rotation β The tilt of Kinect is zero The user faces the Kinect

• β degrees yaw rotation between sensor earth frame and Kinect frame.

11

First Step Yaw Search

12

• For a body segment rotation Sensor measurement:

Kinect measurement:

Minimum rotation distance between and . min(d(, )

First Step Yaw Search

• Movement quality metric µ µ =

: rotation of body segment w.r.t gravity vector. 1 and 2 are two states during one movement. Chosen as 0.1, which is a reliable measure since it

works correctly for all our experiments.

13

First Step Yaw Search

14

Algorithm 1 First step yaw direction search Calculate; if µ < 0.1 The movement is not qualified, choose another one; else Continue; end for = 1:360 Calculate d(, ; ; end return .

Second Step Roll Search

• Two states: arbitrary and ideal• Rotation of the segment

Kinect measurement

Sensor measurement• =

Rotation distance•

15

Algorithm 2 Second step roll rotation search

for = 1:360 Calculate ;(); end return .

Sensor Yaw w.r.t the Human Body

• Through the first step search, the yaw of sensor body frame w.r.t the Kinect frame is calculated as -.

• Yaw rotation between body frame and Kinect frame obtained from Kinect API, denoted as α.

• Sensor yaw rotation w.r.t. the human body frame:

φ = -α

16

Experiment Setup

• Four yaw configurations• Two random roll rotations• 4 subjects

(3 male & 1 female)

• Activities

17

No# Activity No# Activity1 Walking 4 Sit-to-stand

2 Kneeling 5 Stand-to-sit

3 Leg lifting 6 Arm stretch

Yaw Search Results

18

1 2 3 4 5 6 7 8-50

0

50

100

150

200

250

300Yaw search for upper arm

Samples

An

gle

(d

eg

rees)

m1-0deg

m6-0deg

m1-90deg

m6-90deg

m1-180deg

m6-180deg

m1-270deg

m6-270deg

0 5 10 15 20 25 30 35 40-100

0

100

200

300

400Yaw search results for thigh

Samples

Ang

le (d

egre

es)

0 degree yaw90 degree yaw180 degree yaw270 degree yaw

y = 270

y = 180

y = 90

y = 0

Roll Search Results

• Roll search errors for different subjects for arm and thigh

19

Subject #

Arm Accuracy(RMSE in degrees)

Thigh Accuracy(RMSE in degrees)

Subject 1 10.733 5.98Subject 2 5.11 5.60Subject 3 13.5 5.32Subject 4 9.60 5.60

Total 10.73 5.59

Activity Recognition Accuracy

Method Activity #

1 2 3 4 5 6

Approach in [5] 93% 98% 92% 100% 93% 100%

Our approach 92% 98% 90% 100% 92% 100%

20

Performance: our approach offers similar performance as [5] for the activity recognition applications.

[5] A. Henpraserttae, S. Thiemjarus, and S. Marukatat, “Accurate activity recognition using a mobile phone regardless of device orientation and location,” in Body Sensor Networks (BSN), 2011 International Conference on, pp. 41–46, IEEE, 2011.

Conclusions

• Wearable motion sensors need to be calibrated for activity recognition applications.

• We proposed a zero-effort camera-assisted calibration method.

• Our results show good performance of the yaw calibration and an average RMSE of 10.73 degrees for arm and 5.59 degrees for leg for the roll calibration.

• Our method achieves similar performance as classic calibration techniques that require extra efforts.

21

Questions

Zero-Effort Camera-Assisted Calibration Techniques for Wearable Motion Sensors

Jian Wujian.wu@utdallas.eduEmbedded Signal Processing Lab, UT-Dallashttp://www.essp.utdallas.edu

22

WLSACONVERGENCE SUMMIT

www.wirelesshealth2014.org