Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Earphones: The Next Computing Platform
Romit Roy Choudhury
after Smartphones
Nura
Bose
Nokia
Apple
Samsung
Dolby, Oculus
Nura
Bose
Nokia
Apple
Samsung
Dolby, Oculus
… just the tip of the iceberg
Mic IMUSpkr. BLE
Sensors, hardware
Mic IMUSpkr. BLE Vibration Optical Neural ArraysIn-Ear
Sensors, hardware
Mic IMUSpkr. BLE Vibration Optical Neural ArraysIn-Ear
Acoustics,Speech
Wireless(BLE, WiFi, 60GHz)
Multi-modal Sensing
Embedded Systems
Machine Learning
Sensors, hardware
Primitives, Algorithms
Mic IMUSpkr. BLE
Sensors, hardware
Acoustic
Health Sense
Security
Navigate me
Acoustics,Speech
Wireless(BLE, WiFi, 60GHz)
Multi-modal Sensing
Embedded Systems
Machine Learning
Primitives, Algorithms
Vibration Optical Neural ArraysIn-Ear
Translate
2005 2015 20232018
But why Earables? Why now?
Reading and typing, a cognitive re-focus
Voice + hearing seamless for humans
But why Earables? Why now?
1
Reading and typing, a cognitive re-focus
Phone, watch, fitbit = quantified lower body
Voice + hearing seamless for humans
Earables gateway to upper body, head
But why Earables? Why now?
1
2
Reading and typing, a cognitive re-focus
Socially well accepted (unlike, Google Glass)
Phone, watch, fitbit = quantified lower body
Voice + hearing seamless for humans
Earables gateway to upper body, head
Takes off one important risk factor
But why Earables? Why now?
1
2
3
But what are hurdles … show-stoppers?
Many Hurdles …
Energy
Many Hurdles …
1
Energy
Discomfort
Many Hurdles …
1
2
Energy
Discomfort
Privacy / Shy
Many Hurdles …
1
2
3
Energy
Discomfort
Privacy / Shy
Health, radiation
Many Hurdles …
1
2
3
4
Energy
Discomfort
Privacy / Shy
Health, radiation
Many Hurdles …but not Insurmountable
1
2
3
4
Contact Mic.
Hollow
Whisper Zone
Low Frequency
We are building a
software libraryfor Earable Computing
With particular interest in the more challenging problems …
We are building a
software libraryfor Earable Computing
Voice assistants
Motion tracking
In-Ear Anatomy (health, security)
Face/Jaw Activity (eating, drinking)
Spatial acoustics
Noise Cancellation
Sensor and Hardware Layer
Beamforming to Speech
Voice assistants
Motion tracking
In-Ear Anatomy (health, security)
Face/Jaw Activity (eating, drinking)
Spatial acoustics
Noise Cancellation
Sensor and Hardware Layer
Beamforming to Speech
We are building a
software libraryfor Earable Computing
Voice assistants Low SNR Speech Recognition (whisper)
Voice assistants Low SNR Speech Recognition (whisper)
WhisperDecoder ASR
Voice assistants
Motion tracking
In-Ear Anatomy (health, security)
Face/Jaw Activity (eating, drinking)
Spatial acoustics
Noise Cancellation
Sensor and Hardware Layer
Beamforming to Speech
Spatial Acoustics
Different propagation delays at two ears
Spatial Acoustics
Different propagation delays at two ears
Inject artificial delays as if sounds coming from a specific direction
Spatial Acoustics
Different propagation delays at two ears
But delays are not unique along hyperbola
Inject artificial delays as if sounds coming from a specific direction
Spatial Acoustics
Different propagation delays at two ears
But delays are not unique along hyperbola
Inject artificial delays as if sounds coming from a specific direction
Personal Transfer Function
3D Gazing Direction
Spatial Acoustics
Different propagation delays at two ears
But delays are not unique along hyperbola
Inject artificial delays as if sounds coming from a specific direction
Personal Transfer Function
FollowMe
Starbuckshere
Acoustic Augmented Reality (AAR)
Spatial Acoustics
Voice Escort
Voice assistants
Motion tracking
In-Ear Anatomy (health, security)
Face/Jaw Activity (eating, drinking)
Spatial acoustics
Noise Cancellation
Sensor and Hardware Layer
Beamforming to Speech
We are building a
software libraryfor Earable Computing
Motion tracking Indoor localization
Motion tracking Indoor localization
Motion tracking Indoor localization
Motion tracking Indoor localization
Motion tracking Indoor localization
Motion tracking Indoor localization
Body/skeleton is a natural low pass filter
Motion tracking Indoor localization
LocationTracking
Lower body IMU (smartphone)
Head IMU (earphone)
Kinematic Motion models
Body/skeleton is a natural low pass filter
Voice assistants
Motion tracking
In-Ear Anatomy (health, security)
Face/Jaw Activity (eating, drinking)
Spatial acoustics
Noise Cancellation
Sensor and Hardware Layer
Beamforming to Speech
We are building a
software libraryfor Earable Computing
Beamforming to Speech Estimate Angle of Arrival (AoA)
Beamforming to Speech Estimate Angle of Arrival (AoA)
AoA Estimate.
.
.
Path #2(1st Echo)
Path #3(2nd Echo)
Path #1(Direct Path)
Estimate Angle of Arrival (AoA) Beamforming to Speech
Path #2(1st Echo)
Path #3(2nd Echo)
Path #1(Direct Path)
A B C D E F G H I JA B C D E F G
……A B C …
A B C D E F G H I JA B C D E. …
…A B C …
Iterative Align and Cancel (IAC)
Algorithm
Voice assistants
Motion tracking
In-Ear Anatomy (health, security)
Face/Jaw Activity (eating, drinking)
Spatial acoustics
Noise Cancellation
Sensor and Hardware Layer
Beamforming to Speech
We are building a
software libraryfor Earable Computing
Noise Cancellation
Ear Drum
Noise
Anti-Noise
Bone Conduction
Speaker
NoiseCancelation
Cancellation with Hollow earbuds
Face/jaw activity
Different eating, drinking, tooth brushing activity
Activities = Eating, Drinking, Brushing
In-ear health sensing Ear Impulse Response (EIR)
Voice assistants
Motion tracking
In-Ear Anatomy (health, security)
Face/Jaw Activity (eating, drinking)
Spatial acoustics
Noise Cancellation
Sensor and Hardware Layer
Beamforming to Speech
We are building a
software libraryfor Earable Computing
Questions thus far …
Part II
Design a hollow earable
that can still cancel noise
Existing Solutions
This Project
Existing SolutionsBetter Noise Cancellation
9dB Better
This Project
Existing Solutions
Better Form Factor(Hollow)
Better Noise Cancellation
Comparable Performance
9dB Better
Wireless Networking +
Our System: MUTE
Noise Cancellation
Alice
1. Sound starts
Alice
1. Sound starts
2. IoT relay forwards sound over wireless
Alice
1. Sound starts
2. IoT relay forwards sound over wireless
Alice
Wireless radios travel a million times faster than sound
1. Sound starts
2. IoT relay forwards sound over wireless 3. Wireless arrives at
ear-device earlier
Alice
1. Sound starts
2. IoT relay forwards sound over wireless 3. Wireless arrives at
ear-device earlier
4. Actual sound arrives later
Alice
1. Sound starts
2. IoT relay forwards sound over wireless 3. Wireless arrives at
ear-device earlier
Lookahead
4. Actual sound arrives later
Alice
Lookahead allows us to cover ears in time
Analogy: Light travels much faster than sound
MUTE: Leverage lookahead for noise cancellation
Talk Outline
How can MUTE leverage lookahead?
Wideband Cancellation
Non-Causal Filtering
Sound Source Profiling
Timing Gain
Signal Processing Gain
Application-Specific Gain
Talk Outline
How can MUTE leverage lookahead?
Wideband Cancellation
Non-Causal Filtering
Sound Source Profiling
Timing Gain
Signal Processing Gain
Application-Specific Gain
Noise Cancelling Headphones-- What is inside?
Noise Cancellation Primer
Ref. Mic. DSP Anti-NoiseSpeaker
Noise Cancellation Primer
Ref. Mic. DSP Anti-NoiseSpeaker
Noise Cancellation Primer
Ref. Mic. DSP Anti-NoiseSpeaker
Noise Cancellation Primer
Ref. Mic. DSP Anti-NoiseSpeaker
Noise Cancellation Primer
Ref. Mic. DSP Anti-NoiseSpeaker
Noise Cancellation Primer
Ref. Mic. DSP Anti-NoiseSpeaker
Feedback Mic.
(Error Feedback)
How do they perform today?
500 1000 1500 2000 2500 3000 3500
Frequency (Hz)
-30
-20
-10
0
Can
cella
tion
(dB)
Bose_Active
Active noise cancellation only at low frequencies
Bose QC35
Good
Bad
500 1000 1500 2000 2500 3000 3500
Frequency (Hz)
-30
-20
-10
0
Can
cella
tion
(dB)
Bose_Active
Active noise cancellation only at low frequencies
Bose QC35
Good
Bad
500 1000 1500 2000 2500 3000 3500
Frequency (Hz)
-30
-20
-10
0
Can
cella
tion
(dB)
Bose_Active
Active noise cancellation only at low frequencies
Bose QC35
Good
Bad
Sound absorbing material blocks high frequencies
Active noise cancellation only at low frequencies
500 1000 1500 2000 2500 3000 3500
Frequency (Hz)
-30
-20
-10
0
Can
cella
tion
(dB)
Bose_Active
Bose_Overall
Sound absorbing material blocks high frequencies
Bose QC35
Good
Bad
Why does noise cancellation not work at higher frequencies?
Let’s look into the headphone again.
Ref. Mic. Fb. Mic.DSP Speaker
Time
Ref. Mic. Fb. Mic.DSP Speaker
𝑡𝑡1
𝑡𝑡2
Time
Ref. Mic. Fb. Mic.DSP Speaker
𝑡𝑡1
𝑡𝑡2
Time
Ref. Mic. Fb. Mic.DSP Speaker
𝑡𝑡2
Time
𝑡𝑡1
Ref. Mic. Fb. Mic.DSP Speaker
𝑡𝑡1
𝑡𝑡2
Time budget (lookahead):𝑡𝑡2 − 𝑡𝑡1 ≈ 30 μs
1 cm
Time
Ref. Mic. Fb. Mic.DSP Speaker
𝑡𝑡1
𝑡𝑡2
𝑡𝑡3
Time budget (lookahead):𝑡𝑡2 − 𝑡𝑡1 ≈ 30 μs
Time
Ref. Mic. Fb. Mic.DSP Speaker
𝑡𝑡1
𝑡𝑡2
𝑡𝑡3
Time budget (lookahead):𝑡𝑡2 − 𝑡𝑡1 ≈ 30 μs
Time
Time lag: Δ𝑡𝑡 = 𝑡𝑡3 − 𝑡𝑡2
Ref. Mic. Fb. Mic.DSP Speaker
Time
High Frequency:
Anti-Noise
Noise
Residual Error
Low Frequency:
Anti-Noise
Noise
Residual Error
Ref. Mic. Fb. Mic.DSP Speaker
Time
500 1000 1500 2000 2500 3000 3500
Frequency (Hz)
-30
-20
-10
0
Can
cella
tion
(dB)
Bose_Active
Let’s now look at noise cancellation in MUTE ...
Ref. Mic. Fb. Mic.DSP Speaker
𝑡𝑡1
𝑡𝑡2
Time
Fb. Mic.DSP Speaker
𝑡𝑡2
𝑡𝑡1
Ref. Mic.
Time
Fb. Mic.DSP Speaker
𝑡𝑡1
𝑡𝑡2
Ref. Mic.
Time
Fb. Mic.DSP Speaker
𝑡𝑡2
Ref. Mic.𝑡𝑡1
1 m
Time budget (lookahead):𝑡𝑡2 − 𝑡𝑡1 ≈ 3 ms(100X larger)
Time
Fb. Mic.DSP Speaker
𝑡𝑡2
Ref. Mic.𝑡𝑡1
Time budget (lookahead):𝑡𝑡2 − 𝑡𝑡1 ≈ 3 ms(100X larger)
Time
Fb. Mic.DSP SpeakerRef. Mic.𝑡𝑡1
Time budget (lookahead):𝑡𝑡2 − 𝑡𝑡1 ≈ 3 ms(100X larger)
Cancel high frequencies
Anti-Noise
Noise
Residual Error
Time
Talk Outline
How can MUTE leverage lookahead?
Wideband Cancellation
Non-Causal Filtering
Sound Source Profiling
Timing Gain
Signal Processing Gain
Application-Specific Gain
Talk Outline
How can MUTE leverage lookahead?
Wideband Cancellation
Non-Causal Filtering
Sound Source Profiling
Timing Gain
Signal Processing Gain
Application-Specific Gain
Noise Cancellation Primer
Ref. Mic. Feedback Mic.
DSP
Noise Cancellation Primer
Ref. Mic. Feedback Mic.
DSP
𝑓𝑓(•)
Noise Cancellation Primer
Ref. Mic. Feedback Mic.
DSP
𝑔𝑔(•)
𝑓𝑓(•)
Noise Cancellation Primer
Ref. Mic. Feedback Mic.
𝑓𝑓(•)
𝑔𝑔(•)
−𝑓𝑓−1 ∗ 𝑔𝑔
−𝑓𝑓−1 ∗ 𝑔𝑔
Future PastCurrent
Channel Impulse Response
Ref. Mic. Feedback Mic.
−𝑓𝑓−1 ∗ 𝑔𝑔
Future PastCurrent
Today’s Headphones (Causal)
MUTE (Non-Causal)
Channel Impulse Response
Lookahead
“Lookahead” Non-Causal Filtering Better Cancellation
Ref. Mic. Feedback Mic.
Talk Outline
How can MUTE leverage lookahead?
Wideband Cancellation
Non-Causal Filtering
Sound Source Profiling
Timing Gain
Signal Processing Gain
Application-Specific Gain
Talk Outline
How can MUTE leverage lookahead?
Wideband Cancellation
Non-Causal Filtering
Sound Source Profiling
Timing Gain
Signal Processing Gain
Application-Specific Gain
Alice Bob Alice
Alice Bob Alice
Alice Bob Alice
With lookahead …
Alice Bob Alice
Alice Bob Alice
Alice Bob Alice
Talk Outline
How can MUTE leverage lookahead?
Wideband Cancellation
Non-Causal Filtering
Sound Source Profiling
Timing Gain
Signal Processing Gain
Application-Specific Gain
Implementation & Evaluation
MUTE Platform
Noise Source
MUTE PlatformIoT Relay
Noise Source
MUTE Platform
Wireless Receiver
Noise SourceIoT Relay
MUTE Platform
DSP Processor
Wireless Receiver
Noise SourceIoT Relay
MUTE Platform
Feedback Microphone
Wireless Receiver
Noise SourceIoT Relay
DSP Processor
MUTE Platform
Anti-Noise SpeakerWireless Receiver
Noise SourceIoT Relay
DSP Processor
Feedback Microphone
MUTE Platform
MUTE (Hollow)
MUTE Platform
MUTE (Hollow)
Evaluation
Goal: Comparable Performance
MUTE (Hollow) Ear Blocking Headphone
500 1000 1500 2000 2500 3000 3500
Frequency (Hz)
-40
-30
-20
-10
0
Can
cella
tion
(dB)
Bose
MUTE (Hollow)
2. MUTE hollow design (comfort)1. Bose headphone
Evaluation
Goal: Better Performance
MUTE (Non-Hollow) Ear Blocking Headphone
500 1000 1500 2000 2500 3000 3500
Frequency (Hz)
-40
-30
-20
-10
0
Can
cella
tion
(dB)
Bose
MUTE (Hollow)
MUTE (Non-Hollow)
2. MUTE hollow design (comfort)3. MUTE non-hollow design (performance)
1. Bose headphone
+
Zooming Out
Wireless Signal
MUTE Tabletop Relay
DSP
MUTE Tabletop Relay
DSP
MUTE Tabletop Relay
Noise Cancellation as an Edge Service
DSP
Noise Cancellation as an Edge Service
DSP
Anti-Noise for Alice
Feedback
MUTE Enabled Noise Sources
Conclusion
Acoustic Data Plane
Wireless Control Plane
MUTETabletop Relay
MUTEEdge Service
MUTENoise Source
AcousticNoise
Cancellation
Conclusion
AcousticNoise
Cancellation
On-Ear Machine Learning
Acoustic Augmented
Reality…
MUTETabletop Relay
MUTEEdge Service
MUTENoise Source
Acoustic Data Plane
Wireless Control Plane
Closing Thoughts …
The Wearable Market Projections
WiFore: https://www.nickhunn.com/wp-content/uploads/downloads/2014/08/The-Market-for-Smart-Wearables.pdfhttps://www.nickhunn.com/wp-content/uploads/downloads/2014/08/The-Market-for-Smart-Wearables.pdf
Voice assistants
Motion tracking
In-Ear Anatomy (health, security)
Face/Jaw Activity (eating, drinking)
Spatial acoustics
Noise Cancellation
Sensor and Hardware Layer
Beamforming to Speech
We are building a
software libraryfor Earable Computing
Thanks to my excellent
students and collaborators …
Thanks to all of youJay PrakashVisiting Student