Audio Manipulation Through Gesticulation

Garrett Fosdick, Jair RobinsonJosé Sanchez

Bradley University - Electrical & Computer EngineeringOctober 6, 2015

Overview

- Background- Design Approach- Economic Analysis- Schedule- Division of Labor- Societal and Environmental Impacts

2

Background

3

Problem

• Audio manipulation over a distance

4

• Audio manipulation while multi-tasking

• Interactivity with music

Problem Background• Similar products

- Playstation Eyetoy- Xbox Kinect

• Difference- Audio interactivity- Purely 2D image tracking

5

Microsoft 2014

Sony 2008

Solution

• Kinetis Tower• Visual and audio input• Programming based

- Hand tracking- Dynamic Time Warping- Tie audio effects to motions

6

Freescale 2014

Leopard Imaging

Solution - Audio Manipulation Through Gesticulation

• Audio manipulation over a distance

7

• Audio manipulation while multi-tasking

• Interactivity with music

- Control while several feet away

- Control with movement of a single hand

- Interact through motion- Reduce repetitiveness of songs

Hand Tracking Design

8

Color Matching Through Zeroing

9

Color Matching Through Zeroing

10

Color Matching Through Zeroing

11

Color Matching Through Zeroing

12

Color Thresholding

13

Color Thresholding

14

Color Thresholding

15

Color Thresholding

16

Motion Thresholding

17

Motion Thresholding

18

Motion Thresholding

19

Motion Thresholding

20

Motion Thresholding

21

Color and Motion Thresholding

22

Color and Motion Thresholding

23

Color and Motion Thresholding

24

Color and Motion Thresholding

25

Search Limiting

26

Results

27

28

Dynamic Time Warping

29

Calculate Difference

- =A B |A-B|

30

Example

359

- =A B |A-B|

359

359

444

222

333

115

137

026

4 2 3359

31

Calculate Cheapest Route To Bottom Right Corner

Difference Cost Search Area

32

Example

|A-B|

115

137

026

Cheapest Cost To Get To That

Square

1

33

Example

|A-B|

115

137

026

Cheapest Cost To Get To That

Square

12

24

34

Example

|A-B|

115

137

026

Cheapest Cost To Get To That

Square

127

249

24

35

Example

|A-B|

115

137

026

Cheapest Cost To Get To That

Square

127

249

2410

36

Example

|A-B|

115

137

026

Cheapest Cost To Get To That

Square

127

249

2410

37

Variations

• Time distortion cost

38

- Non-diagonal movements cost more

• Path killing- Routes over a certain cost are removed

Audio Processing

39

Audio Processing

• Processing time• Analog – digital conversion• 5 audio effects• Finalizing input and output audio

40

Low Pass Filtering• Passing lower Frequencies below cutoff

41Beausievers 2013

High Pass Filtering• Passing higher frequencies above cutoff

42Beausievers 2013

Distortion• Amplifies audio signal to threshold and clips

43Common Wikimedia 2011

Chorus• Sounds like audio is produced by multiple

sources

44Sound on Sound 2004

Reverb• Sound reflecting in a space

45Practical Musical Production 2012

Development and Testing

46

Development

• Tools- Bradley Computers

- Kinetis Freescale Tower- Leopard Imaging USB Camera

- Kinetis IDE- MATLAB

47

Freescale 2014

Leopard Imaging

Testing• Hand Tracking

- Must have a 80% success rate in the following conditions- Outdoor/Indoor lighting- Different movement speeds

(slow/medium/fast)- At least 3 different hands

- Success is tracking a hand for 30 seconds or more

48

Freescale 2014

Leopard Imaging

Testing

• Dynamic Time Warping- Test against already solved matrixes- Capable of gesture matching

success 90% of the time- Gesture matching will occur in the

same scenarios as the hand tracking

49

Freescale 2014

Leopard Imaging

Testing

• Audio processing- 5 audio effects- Quick processing time- Satisfactory auditory results- No crackling or static- Minimize lag to 100 ms or less

50

Freescale 2014

Leopard Imaging

Economic Analysis

51

Development Costs

52

• Software- Kinetis IDE: $ 0.00- MATLAB: $ 0.00 (Provided By School)

• Hardware- Kinetis Freescale Tower: $294.25- Leopard Imaging USB Camera: $91.94- Total: $386.19

Division of Labor

Decide On Hardware/Order

Parts

Research Video Processes, and Test in

Matlab

Research Audio Processes, and Test in

Matlab

Implement Visual Components on

Hardware

Implement Audio Components on

Hardware

Combine Visual and Audio Components

Test and Improve

Done

Garrett

Jair

Both

Key

53

Schedule

54

Societal and Environmental Impacts

• Is it right to alter an artists music• Liability from damage while gesturing• Liability of harm if used improperly

Disclaimer before use to protect

- Advise users to use caution - Check their surroundings

55

Societal and Environmental Impacts

• For avid music listeners• RoHS compliant

56

Conclusion- Background

- Need for more interactivity with music- Solution provides innovative experience with

personal music- Design Approach

- Dynamic Time Warping and hand recognition for gestures

- Program audio effects tied directly with gestures

57

Conclusion-Feasible project to finish within schedule-Environmentally safe-Socially safe with disclaimer and caution

58

Audio Manipulation Through Gesticulation

Garrett Fosdick, Jair RobinsonJosé Sanchez

Bradley University - Electrical & Computer EngineeringOctober 6, 2015

Extra Slides

60

Division of Labor

61

Test Procedures• Camera Input

- Display Footage On A Monitor - PASS/FAIL

• Hand Tracking- Display Footage At The End Of Each Step – PASS/FAIL- Tracks All Test Hands For A Full 30 Seconds

• Gesture Recognition- Light Up An LED When The Gesture Occurs- Must Be Right 90% Of The Time

62

Test Procedures• Audio Input

- Receiving Without Lag And Distortion- PASS/FAIL

• Audio Output- Audio Is Audible At Normal Hearing Level - PASS/FAIL- No Lag Above 100ms Or Unintended Distortion

• Audio Effects- No Lag from Gesture Trigger Above 100ms- Must Trigger from Correct Gesture 100% of time

63

Preliminary Test Results - Gesture

Matching Partial Random

Success Fail Fail64

Preliminary Test Results - Gesture

65

Block Diagram

66

Glass Block Diagram

67

Gantt Chart

68

Schedule

69

Schedule

70

Schedule

71