43
M1205 Interactivity Topic 10: Motion Tracking Part II Spring 2012 SCM-CityU 1

SM1205 Interactivity Topic 10: Motion Tracking Part II Spring 2012SCM-CityU1

Embed Size (px)

Citation preview

SM1205 Interactivity

Topic 10: Motion Tracking Part II

Spring 2012 SCM-CityU 1

Clipping Regions• By default, motion tracking is done over the entire

video domain – Check out MotionTracker-Skeleton.fla

Spring 2012 SCM-CityU 2

SM1205 Interactivity

Clipping Regions

Spring 2012 SCM-CityU 3

Clipping Regions • What if motion only within specific regions (called

clipping regions) are needed?– Motion outside those clipping regions will be ignored

• Demo

Spring 2012 SCM-CityU 4

clipping regions

Clipping Regions • Specify a clipping region when creating a motion tracker

– cl, ct, cr, cb are with respect to top-left cornerof webcam

– trackX, trackY are also withrespect to top-left cornerof webcam

– You can have only ONE clipping region for one tracker

Spring 2012 SCM-CityU 5

tracker = new MotionTracker(webcam,

cl, ct, cr, cb);tracker = new MotionTracker(webcam,

cl, ct, cr, cb);

cl

cr

ct cb

Clipping Region

Multiple Clipping Regions • To have multiple clipping regions, you can create

multiple trackers – All trackers share the same Webcam instance – Each tracker has its own associated clipping region

Spring 2012 SCM-CityU 6

// initialize 1st motion tracker tracker = new MotionTracker(webcam, 0, 0, 200, 200);

// initialize 2nd motion tracker tracker2 = new MotionTracker(webcam, 300, 300, 500, 500);

// initialize 1st motion tracker tracker = new MotionTracker(webcam, 0, 0, 200, 200);

// initialize 2nd motion tracker tracker2 = new MotionTracker(webcam, 300, 300, 500, 500);

Class Exercise • Extend the previous example to have two clipping regions

as follows– Hint: create two trackers with their own clipping regions– How trackers work at the overlapped areas of clipping regions?

Spring 2012 SCM-CityU 7

Example: Ping-Pong Game• MotionTracker-PingPong-Skeleton.fla in W drive

Spring 2012 SCM-CityU 8

box

barL

ball

barR

box2

Class Exercise • Use tracker and tracker2 to control barL and barR,

respectively

Spring 2012 SCM-CityU 9

Class Exercise

• Make ball bounce against stage borders – Please refer to Ping-Pong example in Topic 05

Spring 2012 SCM-CityU 10

Hit Test between Ball and Bars

• Let’s just use hitTestObject method

Spring 2012 SCM-CityU 11

if (ball.hitTestObject(barR)) { trace("barR got hit");

}

if (ball.hitTestObject(barL)) { trace("barL got hit");

}

if (ball.hitTestObject(barR)) { trace("barR got hit");

}

if (ball.hitTestObject(barL)) { trace("barL got hit");

}

Hit Test between Ball and Bars

• Collision response

Spring 2012 SCM-CityU 12

Case 1: Ball is on left sideof barR

Case 2: Ball is on right sideof barR

Hit Test between Ball and Bars• But how to determine which case to handle?• Solution: use signs of ball’s speedX– speedX > 0 Case 1

• Response: put ball exactly at the left sideof barR

– speedX < 0 Case 2• Response: put ball

exactly at the right sideof barR

Spring 2012 SCM-CityU 13

ball.x = barR.x - barR.width * 0.5 - rball.x = barR.x - barR.width * 0.5 - r

Class Exercise

• Add sound effects when ball hits stage borders/bars – You can use Flash built-in sounds

• Hints– Export sounds for

ActionScript

Spring 2012 SCM-CityU 14

var ws:WoodSound = new WoodSound(); ws.play();var ws:WoodSound = new WoodSound(); ws.play();

Moving Speed • Use MotionTracker’s speedX and speedY to get the

moving speed of tracked object

• speedX (in pixels)– Positive value if object moves along positive x-axis – Negative value if objects moves along negative x-axis

• speedY (in pixels)– Positive value if object moves along positive y-axis – Negative value if objects moves along negative y-axis

Spring 2012 SCM-CityU 15

trace(tracker.speedX, tracker.speedY);trace(tracker.speedX, tracker.speedY);

Example • Transfer moving speed of tracked object to ball– E.g., hand

• Demo

Spring 2012 SCM-CityU 16

Example • Let’s modify the previous ping-pong example– Keep one tracker only, without any clipping region– No vertical bars– Add friction

Spring 2012 SCM-CityU 17

Example • Change ball’s speeds only if there is some intersection

between ball and moving object

Spring 2012 SCM-CityU 18

if (tracker.isTracked() && tracker.hitTestPoint(ball.x, ball.y, true)) {

ball_speed_x += tracker.speedX;ball_speed_y += tracker.speedY;

}

if (tracker.isTracked() && tracker.hitTestPoint(ball.x, ball.y, true)) {

ball_speed_x += tracker.speedX;ball_speed_y += tracker.speedY;

}

Summary of MotionTracker

• Specify source path pointing to the library

Spring 2012 SCM-CityU 19

Summary of MotionTracker

• Without clipping region

• With clipping region

– cl, ct, cr, cb are with respect to top-left cornerof webcam

Spring 2012 SCM-CityU 20

tracker = new MotionTracker(webcam);tracker = new MotionTracker(webcam);

tracker = new MotionTracker(webcam,

cl, ct, cr, cb);tracker = new MotionTracker(webcam,

cl, ct, cr, cb);

Summary of MotionTracker

• Properties – trackX, trackY: center of tracked region • With respect to top-left corner of webcam

– speedX, speedY• Moving speed of

tracked object

Spring 2012 SCM-CityU

2121

Spring 2010

21

(0, 0)

(trackX, trackY)

Summary of MotionTracker

• Methods– hitTestObject• To test if bounding boxes of display object and tracked

region intersect with each other

– hitTestPoint• To test if a given point is within the tracked region

– isTracked• To check if any moving object is tracked

– hideTrackedArea– hideTrackedCenter

Spring 2012 SCM-CityU 22

SM1205 Interactivity

Motion-Driven Buttons

Spring 2012 SCM-CityU 23

Button Click

• Usually done via mouse click

Spring 2012 SCM-CityU 24

Button Click • Can we click digital buttons by our fingers directly?

• Answer: Yes!– Through motion

– Of course, you canalso rely on touchsensors• Need extra hardware

Spring 2012 SCM-CityU 25

Today’s Examples

• Motion-driven button interaction

Spring 2012 SCM-CityU 26

Motion Tracking Library

• In cityu.scm package (developed by Hongbo Fu)

– Webcam class • An easy-to-use wrapper for webcam access

– MotionTracker class• Main class for motion tracking

–MotionTrackerButton class • Use motion to trigger button clicking

Spring 2012 SCM-CityU 27

Key Idea• Analyze motion over a button on the stage– If motion is detected over it, dispatch OVER event

• Cf. MouseEvent.MOUSE_OVER

– If motion continuously exists for a while, dispatch CLICKED event• Cf. MouseEvent.MOUSE_DOWN• To avoid undesired clicks after dispatching CLICKED event, clicking is

disabled for a short period (say 2 sec)

– If clicking is re-enabled, dispatch CLICKABLE event• Cf. MouseEvent.MOUSE_UP

Spring 2012 SCM-CityU 28

• To create a motion-driven button, you need – Step 1: create some display object (e.g., btn) on stage

• By either design tool or ActionScript programming

– Step 2: create a MotionTrackerButton object and associate it with btn

– Step 3: add event listeners for motionBtn • MotionTrackerButton.OVER MotionTrackerButton.CLICKED MotionTrackerButton.CLICKABLE

MotionTrackerButton Class

Spring 2012 SCM-CityU 29

var motionBtn:MotionTrackerButton = new MotionTrackerButton(webcam,

btn);

var motionBtn:MotionTrackerButton = new MotionTrackerButton(webcam,

btn);

Step 1: Create Display Object• MotionTracker-Skeleton.fla in W drive – Draw some shape on stage– Convert it to symbol – Assign an instance name (e.g., btn)

Spring 2012 SCM-CityU 30

Step 2: Create MotionTrackerButton Object

• Before adding a MotionTrackerButton object, we need to initialize a Webcam object– We don’t need to explicitly create a MotionTracker object,

since it is automatically generated for the associated button

Spring 2012 SCM-CityU 31

import cityu.scm.Webcam; // import Webcam class

var webcam:Webcam = new Webcam(640, 480); webcam.addEventListener(Webcam.READY, onCameraReady);

function onCameraReady(e:Event): void { addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha = 0.4;

}

import cityu.scm.Webcam; // import Webcam class

var webcam:Webcam = new Webcam(640, 480); webcam.addEventListener(Webcam.READY, onCameraReady);

function onCameraReady(e:Event): void { addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha = 0.4;

}

Step 2: Create MotionTrackerButton Object

• First import MotionTrackerButton class

• Then create a button with motion control

– 3rd parameter: amount of motion needed to activate clicking• How easy a button can be clicked

– 4th parameter: time (in milliseconds) needed to re-enable clicing after dispatching CLICKED event • How often a button can be clicked

Spring 2012 SCM-CityU 32

import cityu.scm.MotionTrackerButton; import cityu.scm.MotionTrackerButton;

var motionBtn:MotionTrackerButton =

new MotionTrackerButton(webcam, btn, 10, 500); var motionBtn:MotionTrackerButton =

new MotionTrackerButton(webcam, btn, 10, 500);

Step 2: Create MotionTrackerButton Object

• Motion detection is clipped to the button area only – To see tracking visualization, make the button a bit

transparent

Spring 2012 SCM-CityU 33

Step 3: Add Event Listeners• Event types – MotionTrackerButton.OVER– MotionTrackerButton.CLICKED– MotionTrackerButton.CLICKABLE

Spring 2012 SCM-CityU 34

motionBtn.addEventListener(MotionTrackerButton.OVER, onButtonOver);

motionBtn.addEventListener(MotionTrackerButton.CLICKED, onButtonClick);

motionBtn.addEventListener(MotionTrackerButton.CLICKABLE, onButtonClickableAgain);

motionBtn.addEventListener(MotionTrackerButton.OVER, onButtonOver);

motionBtn.addEventListener(MotionTrackerButton.CLICKED, onButtonClick);

motionBtn.addEventListener(MotionTrackerButton.CLICKABLE, onButtonClickableAgain);

function onButtonClick(e:Event): void {

trace("button clicked"); }

function onButtonClick(e:Event): void {

trace("button clicked"); }

Class Exercise• Task 1: when button is clicked by motion – If btn’s current frame label is “up”, go to frame “down”– If btn’s current frame label is “down”, go to frame “up”– Tips

• Add code in onButtonClick • Use currentLabel, gotoAndStop

Spring 2012 SCM-CityU 35

Class Exercise• Task 2: play with different values for 3rd and 4th

parameters of MotionTrackerButton. – 3rd parameter: how easy a button can be clicked– 4th parameter: how often a button can be clicked

• Compare – MotionTrackerButton(webcam, btn, 1, 500);– MotionTrackerButton(webcam, btn, 20, 500);

• Compare – MotionTrackerButton(webcam, btn, 5, 50);– MotionTrackerButton(webcam, btn, 5, 2000);

Spring 2012 SCM-CityU 36

Pausing, Resuming and Removing of MotionTrackerButtons

• Like MotionTracker’s pause, resume and dispose methods, MotionTrackerButton class has same set of methods for similar functionalities

Spring 2012 SCM-CityU 37

Pausing, Resuming and Removing of MotionTrackerButtons

Spring 2012 SCM-CityU 38

btnResume.addEventListener(MouseEvent.CLICK, buttonClick); btnPause.addEventListener(MouseEvent.CLICK, buttonClick); btnNextFrame.addEventListener(MouseEvent.CLICK, buttonClick);

function buttonClick(e:MouseEvent): void { if (e.target == btnResume) {

motionBtn.resume(); } else if (e.target == btnPause) {

motionBtn.pause(); } else if (e.target == btnNextFrame) {

motionBtn.dispose();

removeChild(webcam); webcam.dispose();

gotoAndStop(2); }

}

btnResume.addEventListener(MouseEvent.CLICK, buttonClick); btnPause.addEventListener(MouseEvent.CLICK, buttonClick); btnNextFrame.addEventListener(MouseEvent.CLICK, buttonClick);

function buttonClick(e:MouseEvent): void { if (e.target == btnResume) {

motionBtn.resume(); } else if (e.target == btnPause) {

motionBtn.pause(); } else if (e.target == btnNextFrame) {

motionBtn.dispose();

removeChild(webcam); webcam.dispose();

gotoAndStop(2); }

}

Example: Piano with Motion Control

• How to interact multiple MotionTrackerButton objects?

Spring 2012 SCM-CityU 39

Music Notes• Like previous digital piano example, we use an array to

store music notes

Spring 2012 SCM-CityU 40

var notes:Array = [new S1(), new S2(), new S3(), new S4(),

new S5(), new S6(), new S7()];

var notes:Array = [new S1(), new S2(), new S3(), new S4(),

new S5(), new S6(), new S7()];

Create Motion Buttons • We use arrays to store both piano keys and their

corresponding motion buttons – Corresponding key and motion button have same index

Spring 2012 SCM-CityU 41

// a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array = new Array(); var keys:Array = [key1, key2, key3, key4, key5, key6, key7];

// a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array = new Array(); var keys:Array = [key1, key2, key3, key4, key5, key6, key7];

for (var i:uint = 0; i < keys.length; i++) { keysMB.push(new MotionTrackerButton(webcam, keys[i], 5, 500)); }

for (var i:uint = 0; i < keys.length; i++) { keysMB.push(new MotionTrackerButton(webcam, keys[i], 5, 500)); }

Event Listeners

Spring 2012 SCM-CityU 42

for (var i:uint = 0; i < keys.length; i++) { // motion-driven

keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed);

keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased);

}

for (var i:uint = 0; i < keys.length; i++) { // motion-driven

keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed);

keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased);

}

function onKeyPressed(e:Event): void { for (var i:uint = 0; i < keys.length; i++) {

if (keysMB[i] == e.target) { trace("index of pressed key", i);

} }

}

function onKeyPressed(e:Event): void { for (var i:uint = 0; i < keys.length; i++) {

if (keysMB[i] == e.target) { trace("index of pressed key", i);

} }

}

Extension to Other Instruments

Spring 2012 SCM-CityU 43