Raskar Banff

  • View
    1.113

  • Download
    0

Embed Size (px)

DESCRIPTION

 

Text of Raskar Banff

  • Media Lab, MIT Cambridge, MA From 4D Capture to 6D Display: A mask-based approach Ramesh Raskar
  • Discussion Topics
    • What is the info content of a 3D scene?
      • Encoding appearance and geometric complexity
    • What are the dimensions beyond viewpt?
      • Lighting?
    • What other optical sensors we can use?
  • R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns Siggraph 2007 Prakash: Lighting-Aware Motion Capture Using Photosensing Markers and Multiplexed Illuminators
  • Vicon Motion Capture High-speed IR Camera Medical Rehabilitation Athlete Analysis Performance Capture Biomechanical Analysis
  • Imperceptible Tags under clothing, tracked under ambient light Hidden Marker Tags Outdoors Unique Id
  • Labeling Space (Indoor GPS) Each location receives a unique temporal code But 60Hz video projector is too slow Projector Tags Pos=0 Pos=25 5 Time
  • Pattern MSB Pattern MSB-1 Pattern LSB
    • For each tag
    • From light sequence, decode x and y coordinate
    • Transmit back to RF reader ( Id , x, y )
    0 1 1 0 0 X=12
  • Inside of Multi-LED Emitter
  • Tag
  • Analog Space Labeling Multi-LED Beacon 1 Beacon 2 Beacon 3 Tag N ?
  • Imperceptible Tags Location
  • Location Orientation
  • 3D Overlay Orientation
  • Imperceptible Tags Incident Illumination
  • Inverse Optical Mo-Cap
    • High Speed Camera
    • Detect blobs in each frame
    • Reflective/Emitting Marker
    • Disambiguate in camera
    • Only Location
    • High Speed Projector
    • Label the 3D space
    • Photosensing Marker
    • Find ego-position
    • Location, Orientation, Illum
  • On-set MoCap: Location + Orientation + Incident Illumination
  • Coded Illumination Sensor Skin
    • 500 Hz with Id for each Marker Tag
    • Capture in Natural Environment
      • Visually imperceptible tags
      • Photosensing Tag can be hidden under clothes
      • Ambient lighting is ok
    • Unlimited Number of Tags
      • Light sensitive fabric for dense sampling
    • Non-imaging, complete privacy
    • Base station and tags only a few 10s $
    • Body scan + bio
      • Elderly, patients, athletes, performers
  • Project Topics
    • Structured Light Scanning
      • Fast Stripping
        • Can you build a scanner using very low cost hardware?
        • Without full 2D cameras or video projectors?
      • Global-direct Separation
        • Can you scan difficult (global effect) using direct/global separation?
  • Towards a 6D Display Passive Reflectance Field Display Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008 1 2 1 1 1 MPI Informatik, Germany 2 MIT
  • Martin Fuchs
    • [Lippman 1908] [Nakajima et al. 2001] ...
    Martin Fuchs
  • electronic: [Nayar et al. 2004] slit based / different patterns: [Scharstein 1996] Martin Fuchs
  • Martin Fuchs
  • Martin Fuchs
  • Martin Fuchs
  • Martin Fuchs
  • Improved Design Martin Fuchs
  • Variance with Observer Martin Fuchs
    • recall:
  • Martin Fuchs
  • Martin Fuchs
  • Martin Fuchs
  • Observer-Variance Martin Fuchs
  • 6D Construction Martin Fuchs
  • Illumination + Spatial Variation Martin Fuchs
  • Variance with Observation Angle Martin Fuchs
  • Towards 6D Martin Fuchs
  • 6D Results Martin Fuchs
  • Future Work
    • Efficient manufacturing
    • scale
    • precision
    • How fine can we get our structures?
      • is 6D really practical?
    • Extensions for local illumination ?
    Martin Fuchs
  • Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern
  • In Focus Photo LED
  • Out of Focus Photo: Open Aperture
  • Out of Focus Photo: Coded Aperture
  • Captured Blurred Photo
  • Refocused on Person
  • Mask? Sensor 4D Light Field from 2D Photo: Heterodyne Light Field Camera Full Resolution Digital Refocusing: Coded Aperture Camera Mask? Sensor Mask Sensor Mask? Sensor Mask Sensor
  • Light Field Inside a Camera
  • Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Light Field Inside a Camera
  • Stanford Plenoptic Camera [Ng et al 2005]
    • 4000 4000 pixels 292 292 lenses = 14 14 pixels per lens
    Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125 square-sided microlenses
  • Digital Refocusing [Ng et al 2005] Can we achieve this with a Mask alone?
  • Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor
  • Heterodyne Light Field Camera Scanner sensor Mask [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor
  • How to Capture 4D Light Field with 2D Sensor ? What should be the pattern of the mask ?
  • Radio Frequency Heterodyning Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz Baseband Audio Signal
  • Optical Heterodyning Photographic Signal (Light Field) Carrier Incident Modulated Signal Reference Carrier Main Lens Object Mask Sensor Recovered Light Field Software Demodulation Baseband Audio Signal Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz
  • Captured 2D Photo Encoding due to Mask
  • 2D FFT Traditional Camera Photo Heterodyne Camera Photo Magnitude of 2D FFT 2D FFT Magnitude of 2D FFT
  • Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9
  • A Theory of Mask-Enhanced Camera
    • Mask == Light Field Modulator
    • Intensity of ray gets multiplied by Mask
    • Convolution in Frequency domain
    Main Lens Object Mask Sensor
  • Related Work
    • Ligh