Upload
satya-harish
View
164
Download
0
Embed Size (px)
Citation preview
1 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Implementing Computational Camera
Michael Mangan Product Manager, Senior Qualcomm Technologies, Inc.
3 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Computational camera: What is it?
4 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Computational camera Utilizes both the ISP HW and Qualcomm® Snapdragon™ processing to create new and exciting camera effects!
• Example: Image segmentation and filtering an image allow for the digital re-creation of a sophisticated portraiture effect, called the “Bokeh” effect.
Sens
or 1
Sens
or 2
ISP 1
ISP 2
Qualcomm® KRAIT CPU
Qualcomm® ADRENO™ GPU
Qualcomm® HEXAGON™ DSP
Qualcomm Snapdragon, Qualcomm Adreno, Qualcomm Krait and Qualcomm Hexagon are products of Qualcomm Technologies, Inc.
5 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Computational camera Depth calculation designed to enable shoot once, focus later
Sens
or 1
Se
nsor
2
ISP 1
ISP 2
KRAIT CPU
ADRENO GPU
HEXAGON DSP
6 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Computational camera Depth calculation designed to enable range finding for instant auto-focus
Sens
or 1
Se
nsor
2
ISP 1
ISP 2
7 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Computational photography examples
8 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Qualcomm® UbiFocus™ camera feature
Qualcomm UbiFocus is a product of Qualcomm Technologies, Inc.
9 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
UbiFocus UbiFocus
Description UbiFocus is a camera feature that generates an all in focus image from a mobile camera with a limited depth of field. Additionally, this feature allows users to selectively refocus their image in post-processing.
KPI
1. No fewer than 5 images. 2. For 13M images
1. Take no longer than 1.0s to capture images. 0.7s with further optimization 2. Take no longer than 0.5s after the capture to process input images generate an all in focus image.
OEM feature deliverable
1. UbiFocus available to licensees provided as part of the MSM8974 LA software release as optional software. 2. UbiFocus provided in binary form, and called by OEM enabled camera applications via an API interface defined by Qualcomm Technologies. 3. Use-Case
a) All-In-Focus Mode: a) When ‘All-in-Focus’ mode for UbiFocus is selected, the snapshot operation will result in an ‘All-in-Focus’ JPEG image from the
mobile camera. b) Dynamic Focus Mode:
a) User will need to select ‘Dynamic Focus Mode’ option within Camera App. b) UbiFocus will allow the end-user to touch an object in a scene and bring it in focus. c) The user can either share the refocus-able object on the web, or; d) The user can select the desired focus point and save the image as a final JPEG image.
10 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
UbiFocus • Mobile camera’s have a limited Depth-Of-Field
Out of focus In focus Out of focus
Input image 1 Input image 2
In focus
11 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
UbiFocus • From a Limited Depth-of-field Camera
• UbiFocus generates an all-in-focus image
• Refocus at a desired location as post processing
Out of focus Out of focus
Input image 1 Input image 2 UbiFocus
In focus In focus
In focus
12 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Video: UbiFocus
13 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
UbiFocus
Use case
• Refocus at a user-desired location
• Share images that can be re-focused (the recipient does not need UbiFocus to perform dynamic re-focus)
• Generates an all-in-focus image
Benefits
• Eliminates the need for auto-focus
• Focus-on-demand “live” images
• Same effect as Lytro and Pelican Imaging, but with much higher resolution and no additional hardware
• Preserves resolution (unlike Lytro)
• Integrated with Snapdragon camera pipeline
14 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
OptiZoom camera feature
OptiZoom is a product of Qualcomm Technologies, Inc.
15 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
OptiZoom OptiZoom
Description
OptiZoom is a multi-frame super-resolution feature that enhances the resolution by 2x on each side (i.e., 4x number of pixels) that mimics an optical zoom. OptiZoom uses a Qualcomm Technologies-designed multi-frame super-resolution technique to allow an end-user to zoom into an image beyond normal digital zoom levels. OptiZoom designed to increase image resolution and also reduce noise. OptiZoom, can be used to super-resolve only a region of interest or it can be used to super-resolve the entire image.
KPI
1. Eight images of 12MP shall be taken in less than 0.6s. 2. Processing time for converting a 4MP image into a 16MP image is ~700ms on 8974. Processing time varies almost linearly
wrt the number of pixels.
OEM feature deliverable
1. OptiZoom, licensed to OEMs as optional software, as part of the MSM 8974 software release. 2. OptiZoom provided in binary form, and called by OEM enabled camera applications via an API interface defined by
Qualcomm Technologies. 3. Use Case:
a) The end-user zooms into the area of interest. b) The end-user initiates OptiZoom on the area of interest. c) The OptiZoom algorithm enhances the resolution of the area of interest.
16 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
OptiZoom use cases
Option 1 When user presses the “take image” button, the device takes N full-size images.
OptiZoom is invoked with the full frame
The user can zoom in to any area of the image without invoking OptiZoom again
Option 2 User first zooms-in in the preview and specifies RoI and then N images are taken.
OptiZoom is invoked with the RoI The user gets a 4x image of the RoI.
Option 3
When user presses the “take image” button, the device takes N full-size images. Then user zooms in and selects an RoI
OptiZoom is invoked with the RoI
The user gets a 4x image of the RoI. If the user want another region of the image to be super-resolved, OptiZoom can be invoked again.
17 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
OptiZoom - Disabled
18 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
OptiZoom - Enabled
19 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
OptiZoom Disabled Enabled
20 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
OptiZoom
Use case
• Increasing the image resolution by 4x number of pixels
• See previous slide for more details
• Useful for allowing users to zoom in on key regions within an image.
- Read fine text - See faces clearly
Benefits
• Increases photo resolution and reduces image noise.
• Fast and scalable
• Utilizes Snapdragon Hardware.
• Integrated with Snapdragon camera pipeline
21 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Qualcomm® ChromaFlash™ camera feature
Qualcomm Cromaflash is a product of Qualcomm Technologies, Inc.
22 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
ChromaFlash ChromaFlash
Description
ChromaFlash takes two images in quick succession: one with flash enabled and one without flash enabled to produce a beautiful composite image. Flash images usually suffer from over-exposedness and bluish tint and dark background. Our ChromaFlash technology overcomes these problems algorithmically and produces and image that preserves color, texture, and brightness.
KPI
1. ChromaFlash Images shall be taken in less than 0.135s on top of flash image capture time to capture 8/13MP input images. 2. ChromaFlash algorithm processing shall take no more than 700ms for two 13MP images on 8974. Processing time varies
almost linearly with the number of pixels.
OEM feature deliverable
1. ChromaFlash, licensed to OEMs as optional software, as part of the MSM8974 LA software release. 2. ChromaFlash provided in binary form, and called by OEM enabled camera applications via an API interface defined by
Qualcomm Technologies. 3. Use-Case/Default Operation
a) ChromaFlash will engage whenever normal Flash is used, effectively replacing ‘normal’ flash operation. b) ChromaFlash 2.0 can handle dynamic scenes (i.e., scenes that contain motion) as well as camera motion due to
handshake
23 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
No-flash image Flash image ChromaFlash image
Another ChromaFlash example
Note: Projector quality may limit ability to see enhancements visible on laptop monitor.
24 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
ChromaFlash
Use case
• Flash that preserves color
• Enhances Low-light imaging
• Enhances Dynamic Range
Benefits
• No extra hardware is required
• Reduces the noise observed in no-flash images taken in low-light.
• Maintains color correctness lost using the flash.
• Increases the dynamic range
• Integrated with Snapdragon camera pipeline
25 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Low-light video enhancement
26 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Low light vision enhancements
Description
Low-light video enhancement performs two main functions: - Adaptively brightens regions of the video frame that are too dark. If the scene is sufficiently bright already, Low-light video enhancement does not interfere. - Reduces noise
KPI Processing time is ~7ms on a 1080p frame on 8974. Processing time varies almost linearly wrt the number of pixels.
Tier 1 OEM feature deliverable
1. Low-light video enhancement, licensed to OEMs as optional software, as part of the MSM8974 LA software release. 2. Low-light video enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API
interface defined by Qualcomm Technologies. 3. Use-Case: Low-light video
Competition
1. Arcsoft NightHawk 2. Morpho
27 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
What is Low-light video enhancement? Low-light video enhancement is an algorithm for adaptively brightening areas of video/images that are too dark
Attributes
• It works for dark scenes and scenes that are generally bright, but include dark regions
• It is a low-power and low-complexity algorithm. As a result, it can run comfortably on even low-tier chips @ 30fps for 1080p resolution
• It also includes a fast and efficient noise-reduction to attenuate noise in the areas that were under-exposed in the original frame
28 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Video: Low light enhancement
29 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Low-light video enhancement comparison Original Revision-LTM
30 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Low-light video enhancement comparison Original Low-light video enhancement
31 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Noise reduction
No Temporal NR With Temporal NR
32 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Video: Preserving sharpness
33 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Low-light video enhancement enhancing low light video
Original Low-light video enhancement
34 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Portrait enhancement
35 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Portrait enhancement
Description
A a camera feature that takes a regular portrait image taken with a camera phone and synthesizes shallow depth-of-field (aka Bokeh effect) as if the image is produced by an DSLR camera. This feature also brings a static image to life by automatically generating 3D parallax and zoom effect (see the videos in upcoming slides) The input image does not even have to be taken by a camera equipped with portrait enhancement.
KPI
1. Requires an image with 1-3 people in the scene (the max number of people is adjustable if needed) 2. Processing time on 8974:
a) Initial segmentation: 1.2s b) Re-segmentation after user edit: 200ms c) Bokeh effect: 1.1s for 13MP output d) Parallax effect: 250ms e) Dolly Zoom effect: 250ms f) Horizontal motion blur effect: 1.1s for 13MP output g) Sketch effect: 550ms for 13MP output h) Fusion effect: 450ms for 13MP output
OEM feature deliverable
1. Portrait enhancement licensed as optional software as part of the MSM8974 LA software release. 2. Portrait enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API interface defined by Qualcomm
Technologies. 3. Use-Case:
- Synthesize a shallow-depth-of-field automatically even on images that have been taken previously without portrait enhancement - Generates 3D Parallax and Dolly Zoom effects
36 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Portrait enhancement • What is it?
− It takes a single image and automatically segments out humans in the image. Then one of several effects (see following slides) can be applied applied to the background
• Attributes
− portrait enhancement can be applied even to images that are taken previously using an ordinary camera
− If needed, it allows the user to edit the foreground and background by providing small hints
37 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Portrait enhancement vs. DSLR
Camera phone (input image)
Portrait enhancement DSLR w/ expensive lens
38 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Input image
39 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Shallow depth-of-field (mimics portrait mode of DSLR cameras)
40 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Parallax
41 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Dolly-zoom
42 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Horizontal motion blur
43 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Sketch
44 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Still camera image enhancement
45 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Still camera image enhancement
Description
Takes a single image and enhances the dynamic range of the still image by automatically brightening regions of the image that are too dark and remove haze from the image. It allows the user to move a slider and change the amount of adaptive enhancement applied to the image. The input image does not even have to be taken with a camera that is equipped with Still camera image enhancement.
KPI Processing time for 13MP image on 8974 is ~80ms. Processing time varies almost linearly wrt the number of pixels.
OEM feature deliverable
1. Still camera image enhancement, licensed as optional software, provided as part of the MSM8974 LA software release. 2. Still camera image enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API
interface defined by Qualcomm Technologies. A sample app is included as a reference to guide OEM’s implementation of the feature
3. Use-Case: - Enhance the image quality dramatically by adaptively and automatically removing haze and brightening regions of the image that are too dark
46 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Video: Still camera image enhancement
47 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
General image segmentation
48 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Image segmentation
• Accurate image segmentation with user interactive
• Applications include
– Photo editing for re-focusing
– Multi-layer color editing
– Multi-layer image filtering
• Available – October 2014
Segm
enta
tion
Mat
ting
49 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Video: Image segmentation
50 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Touch to Track A general object tracker
51 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Touch to Track (T2T) v2.0
Track, detect, and learn
• Lucas-Kanade optical flow tracking
• Decision Forest and NCC detection
• Online learning bootstrapping binary classifier
• Leverages Motion Vector Information from Venus Video Core.
Applications
• Camera and video auto focus, exposure, zoom, etc.
• Robot control with T2T
• Advanced video recording / snapshot
52 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Video: Touch to Track (T2T) v2.0
53 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Touch to Track – Power optimized performance • Below are the power consumption in performance-mode* and power-optimized-mode** of Touch to Track on 8974
using the same test sequences
• In performance mode, Touch to Track power consumption is about 200-300% of those in the power-optimized-mode. The ratio could be higher in full frame detection
382
367
210
395
109 123 113
183
0
50
100
150
200
250
300
350
400
450
video1 video2 video3 video4
mW
performance mode power-optimized mode
* Performance mode (CPU Only) and power-optimized (DSP & CPU) mode selectable ** Power saving achieved by utilizing multiple processors including DSP on 8974 Note: No tracking performance differences between performance and power-optimized modes.
54 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Touch to Track
Use case
• Tracker enhanced 3A algorithms • Single Camera Picture in Picture in preview and
recording • Artistic special effects in camera preview and video
recording
Benefits
• Multiple object tracking - Track 2, up to 4 objects
• Robust tracking algorithm tightly integrated with camera 3A and auto-zoom algorithms, enhances: - Auto Exposure - Auto Focus - Auto Zoom.
• Allow selected region of image to be displayed in single-camera PIP (Picture in Picture)
• Tracker is Snapdragon HW accelerated • Tracking on 1080p @30fps
55 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Object removal & cloning
56 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Object removal and cloning 2.0 – single frame This feature is different from multiframe motion based object removal • Content based on-device image editing
• Object Removal allows the user to select and remove and object
• Cloning allows users to replicate and insert an image Object removal Object cloning
57 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Object removal and cloning
Use case
Object Removal
• Allows on-device, user selectable, removal of blemishes within the image.
• Uses advanced background matching algorithm to match the background.
Object Cloning
• Allows on-device, user selectable, cloning of new images onto existing photographs.
Benefits
• Low power, high quality, image processing features • Easily integrated into existing Camera Application to
fit existing User Interface. • Written using OpenCL to fully utilize Adreno GPU
58 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
For more information on Qualcomm, visit us at: www.qualcomm.com & www.qualcomm.com/blog
©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Snapdragon, Adreno, and Hexagon are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Krait, Uplinq, UbiFocus and ChromaFlash are trademarks of Qualcomm Incorporated. All Qualcomm Incorporated trademarks are used with permission. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.
Thank you FOLLOW US ON: