Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
The Visual Microphone: Passive Recovery of Sound from Video
Copyright of figures and other materials in the paper belongs original authors.
Presented by Ki-hoon Kim
2015.09.09
Computer Graphics @ Korea University
Abe Davis et al.SIGGRAPH 2014
Ki-hoon Kim | 2016-01-07 | # 2Computer Graphics @ Korea University
3D Position Measurement From Image
Image Data
3D Geometry Data
Ki-hoon Kim | 2016-01-07 | # 3Computer Graphics @ Korea University
Emotion Recognition
Image Data
Sound DataEmotion
Recognition
Ki-hoon Kim | 2016-01-07 | # 4Computer Graphics @ Korea University
Introduction
Ki-hoon Kim | 2016-01-07 | # 5Computer Graphics @ Korea University
• Recording the reflection of a laser pointed at the surface.
Record phase of a reflected laser
Recover high quality audio
Depend on precise positioning(Laser & Receiver)
Surface must be appropriate reflectance
Related Work - Laser microphone
Ki-hoon Kim | 2016-01-07 | # 6Computer Graphics @ Korea University
Using high-speed camera
Record changes in the speckle pattern of laser light
Greater flexibility in the positioning of a receiver
Depend on recording reflected laser light
Related Work - Laser microphone
Heart beat Human Voice
Simultaneous remote extraction of multiple speech sources and heart beats from secondary speckles pattern[Zalevsky et al./Opt. Express 2009.]
Ki-hoon Kim | 2016-01-07 | # 7Computer Graphics @ Korea University
Related Work – Extract subtle motions
Ki-hoon Kim | 2016-01-07 | # 8Computer Graphics @ Korea University
Overview
Ki-hoon Kim | 2016-01-07 | # 9Computer Graphics @ Korea University
• 0. Decompose the input video into spatial sub-bands
Corresponding to different orientations and scales
• 1. Computing the Local Motion Signals(Phase Variations)
• 2. Computing the Global Motion Signal
• 3. Denoising
Recovering Sound from Video
Ki-hoon Kim | 2016-01-07 | # 10Computer Graphics @ Korea University
• Complex Image can be expressed(Riesz Pyramid Transform)
𝐴 : Amplitude
𝜑 : Phase
𝜃 : Transform Orientation
𝑟 : Transform Scale
• Compute phase variations
Recovering Sound from VideoLocal Motion Signals
Ki-hoon Kim | 2016-01-07 | # 11Computer Graphics @ Korea University
• Weigh each local signal by its squared amplitude
• Shift in time to align phase signals
• Global motion signal is
Recovering Sound from VideoGlobal Motion Signal
Ki-hoon Kim | 2016-01-07 | # 12Computer Graphics @ Korea University
• Modeling object motion
(a,b) 300 Hz pure tone, [0.1,1] Pa
(c) 20 Hz to 2,200 Hz frequency
Object Response
Ki-hoon Kim | 2016-01-07 | # 13Computer Graphics @ Korea University
• Modeling pixel motion
𝑚 : Magnification of surface in [mm/pixel]
𝜃 : Viewing angle of camera
𝜔 : Frequency
• SNR of recovered sound relationship
𝑛𝑝 : The number of pixels
𝜎𝑛 : Image Noise Standard Deviation
Processing
Ki-hoon Kim | 2016-01-07 | # 14Computer Graphics @ Korea University
Experiments - Frequency Range
Input Sound
ReconstructedSound
Ki-hoon Kim | 2016-01-07 | # 15Computer Graphics @ Korea University
Experiments – Recovery Comparison
• VM – Our Method, LDV – Laser Doppler Vibrometer
• Bold: Better score
• VM is comparable to LDV
• LDV is required the retro-reflective tape
Ki-hoon Kim | 2016-01-07 | # 16Computer Graphics @ Korea University
Recovering Sound with Normal Video Cameras- What is Rolling Shutter?
Ki-hoon Kim | 2016-01-07 | # 17Computer Graphics @ Korea University
Recovering Sound with Normal Video Cameras- How to use Rolling Shutter
• aaaa
Ki-hoon Kim | 2016-01-07 | # 18Computer Graphics @ Korea University
• Assume exposure time 𝐸 ≈ 0,
𝐼𝑛 : Rest State Image
𝐵𝑛 : Image with Rolling Shutter Effect
𝑛 : Frame number
𝑇 : Frame Period
𝑑 : Line Delay
Recovering Sound with Normal Video Cameras- Equation Modeling
Ki-hoon Kim | 2016-01-07 | # 19Computer Graphics @ Korea University
Recovering Sound with Normal Video Cameras- Convert to audio signal
• Missing Sample by Frame Delay
Audio interpolation
Ki-hoon Kim | 2016-01-07 | # 20Computer Graphics @ Korea University
Results
Ki-hoon Kim | 2016-01-07 | # 21Computer Graphics @ Korea University
Q&A