View
212
Download
0
Category
Tags:
Preview:
Citation preview
Audio Manipulation Through Gesticulation
Garrett Fosdick, Jair RobinsonJosé Sanchez
Bradley University - Electrical & Computer EngineeringOctober 6, 2015
Overview
- Background- Design Approach- Economic Analysis- Schedule- Division of Labor- Societal and Environmental Impacts
2
Background
3
Problem
• Audio manipulation over a distance
4
• Audio manipulation while multi-tasking
• Interactivity with music
Problem Background• Similar products
- Playstation Eyetoy- Xbox Kinect
• Difference- Audio interactivity- Purely 2D image tracking
5
Microsoft 2014
Sony 2008
Solution
• Kinetis Tower• Visual and audio input• Programming based
- Hand tracking- Dynamic Time Warping- Tie audio effects to motions
6
Freescale 2014
Leopard Imaging
Solution - Audio Manipulation Through Gesticulation
• Audio manipulation over a distance
7
• Audio manipulation while multi-tasking
• Interactivity with music
- Control while several feet away
- Control with movement of a single hand
- Interact through motion- Reduce repetitiveness of songs
Hand Tracking Design
8
Color Matching Through Zeroing
9
Color Matching Through Zeroing
10
Color Matching Through Zeroing
11
Color Matching Through Zeroing
12
Color Thresholding
13
Color Thresholding
14
Color Thresholding
15
Color Thresholding
16
Motion Thresholding
17
Motion Thresholding
18
Motion Thresholding
19
Motion Thresholding
20
Motion Thresholding
21
Color and Motion Thresholding
22
Color and Motion Thresholding
23
Color and Motion Thresholding
24
Color and Motion Thresholding
25
Search Limiting
26
Results
27
28
Dynamic Time Warping
29
Calculate Difference
- =A B |A-B|
30
Example
359
- =A B |A-B|
359
359
444
222
333
115
137
026
4 2 3359
31
Calculate Cheapest Route To Bottom Right Corner
Difference Cost Search Area
32
Example
|A-B|
115
137
026
Cheapest Cost To Get To That
Square
1
33
Example
|A-B|
115
137
026
Cheapest Cost To Get To That
Square
12
24
34
Example
|A-B|
115
137
026
Cheapest Cost To Get To That
Square
127
249
24
35
Example
|A-B|
115
137
026
Cheapest Cost To Get To That
Square
127
249
2410
36
Example
|A-B|
115
137
026
Cheapest Cost To Get To That
Square
127
249
2410
37
Variations
• Time distortion cost
38
- Non-diagonal movements cost more
• Path killing- Routes over a certain cost are removed
Audio Processing
39
Audio Processing
• Processing time• Analog – digital conversion• 5 audio effects• Finalizing input and output audio
40
Low Pass Filtering• Passing lower Frequencies below cutoff
41Beausievers 2013
High Pass Filtering• Passing higher frequencies above cutoff
42Beausievers 2013
Distortion• Amplifies audio signal to threshold and clips
43Common Wikimedia 2011
Chorus• Sounds like audio is produced by multiple
sources
44Sound on Sound 2004
Reverb• Sound reflecting in a space
45Practical Musical Production 2012
Development and Testing
46
Development
• Tools- Bradley Computers
- Kinetis Freescale Tower- Leopard Imaging USB Camera
- Kinetis IDE- MATLAB
47
Freescale 2014
Leopard Imaging
Testing• Hand Tracking
- Must have a 80% success rate in the following conditions- Outdoor/Indoor lighting- Different movement speeds
(slow/medium/fast)- At least 3 different hands
- Success is tracking a hand for 30 seconds or more
48
Freescale 2014
Leopard Imaging
Testing
• Dynamic Time Warping- Test against already solved matrixes- Capable of gesture matching
success 90% of the time- Gesture matching will occur in the
same scenarios as the hand tracking
49
Freescale 2014
Leopard Imaging
Testing
• Audio processing- 5 audio effects- Quick processing time- Satisfactory auditory results- No crackling or static- Minimize lag to 100 ms or less
50
Freescale 2014
Leopard Imaging
Economic Analysis
51
Development Costs
52
• Software- Kinetis IDE: $ 0.00- MATLAB: $ 0.00 (Provided By School)
• Hardware- Kinetis Freescale Tower: $294.25- Leopard Imaging USB Camera: $91.94- Total: $386.19
Division of Labor
Decide On Hardware/Order
Parts
Research Video Processes, and Test in
Matlab
Research Audio Processes, and Test in
Matlab
Implement Visual Components on
Hardware
Implement Audio Components on
Hardware
Combine Visual and Audio Components
Test and Improve
Done
Garrett
Jair
Both
Key
53
Schedule
54
Societal and Environmental Impacts
• Is it right to alter an artists music• Liability from damage while gesturing• Liability of harm if used improperly
Disclaimer before use to protect
- Advise users to use caution - Check their surroundings
55
Societal and Environmental Impacts
• For avid music listeners• RoHS compliant
56
Conclusion- Background
- Need for more interactivity with music- Solution provides innovative experience with
personal music- Design Approach
- Dynamic Time Warping and hand recognition for gestures
- Program audio effects tied directly with gestures
57
Conclusion-Feasible project to finish within schedule-Environmentally safe-Socially safe with disclaimer and caution
58
Audio Manipulation Through Gesticulation
Garrett Fosdick, Jair RobinsonJosé Sanchez
Bradley University - Electrical & Computer EngineeringOctober 6, 2015
Extra Slides
60
Division of Labor
61
Test Procedures• Camera Input
- Display Footage On A Monitor - PASS/FAIL
• Hand Tracking- Display Footage At The End Of Each Step – PASS/FAIL- Tracks All Test Hands For A Full 30 Seconds
• Gesture Recognition- Light Up An LED When The Gesture Occurs- Must Be Right 90% Of The Time
62
Test Procedures• Audio Input
- Receiving Without Lag And Distortion- PASS/FAIL
• Audio Output- Audio Is Audible At Normal Hearing Level - PASS/FAIL- No Lag Above 100ms Or Unintended Distortion
• Audio Effects- No Lag from Gesture Trigger Above 100ms- Must Trigger from Correct Gesture 100% of time
63
Preliminary Test Results - Gesture
Matching Partial Random
Success Fail Fail64
Preliminary Test Results - Gesture
65
Block Diagram
66
Glass Block Diagram
67
Gantt Chart
68
Schedule
69
Schedule
70
Schedule
71
Recommended