Advances in iOS Photography - Apple Inc....Camera Capture Manual Controls (iOS 8 / Yosemite) WWDC...

Live Photo, RAW, and Wide Color Capture with AV Foundation

Media #WWDC16

Session 501

Advances in iOS Photography

Brad Ford Apple

developer.apple.comPast Sessions

What’s New In Camera Capture (iOS 6) WWDC 2012

What’s New In Camera Capture (iOS 7) WWDC 2013

Camera Capture Manual Controls (iOS 8 / Yosemite) WWDC 2014

Agenda

New AVCaptureOutput

Agenda

New AVCaptureOutputNew photography-driven features in iOS 10!

Agenda

New AVCaptureOutputNew photography-driven features in iOS 10!• Live Photos

Agenda

New AVCaptureOutputNew photography-driven features in iOS 10!• Live Photos• RAW and DNG

Agenda

New AVCaptureOutputNew photography-driven features in iOS 10!• Live Photos• RAW and DNG• Preview (Thumbnail) images

Agenda

New AVCaptureOutputNew photography-driven features in iOS 10!• Live Photos• RAW and DNG• Preview (Thumbnail) images• Wide Color

Review: AV Foundation Capture Objects

AVCaptureSession

AVCaptureDeviceInput

AVCaptureDevice (Camera)

AVCaptureSession

AVCaptureMovieFileOutput

AVCaptureSession

AVCaptureStillImageOutput

AVCaptureConnection AVCaptureConnection

AVCaptureSession

AVCapturePhotoOutput

NEWAVCapturePhotoOutput Design Features

Functional programming model

Functional programming modelPhoto settings encapsulation

Functional programming modelPhoto settings encapsulationA delegate-style interface for tracking the progress of photo capture requests

Functional programming modelPhoto settings encapsulationA delegate-style interface for tracking the progress of photo capture requestsResolving of photo settings to an immutable object

Using AVCapturePhotoOutput

Read-only properties

Feature opt-in properties

Methods

isLivePhotoCaptureSupportedavailablePhotoPixelFormatTypes

availablePhotoCodecTypes

Methods

isHighResolutionCaptureEnabledisLivePhotoCaptureEnabled

Methods

isHighResolutionCaptureEnabledisLivePhotoCaptureEnabled

capturePhoto(with:, delegate:)

Using AVCapturePhotoOutput NEW

AVCapturePhotoSettingsFeatureXFeatureYFeatureZ

-capturePhoto(with:, AVCapturePhotoSettingsFeatureXFeatureYFeatureZ

-capturePhoto(with:,

AVCapturePhotoCaptureDelegatephotoEventAHappenedphotoEventBHappenedphotoEventCHappened

-capturePhoto(with:,

delegate:)

AVCapturePhotoSettings

Atomic

AtomicUnique

AtomicUniqueYour order form copy

Single set of callbacks per photo settings AVCapturePhotoCaptureDelegatephotoEventAHappenedphotoEventBHappenedphotoEventCHappened

Single set of callbacks per photo settingsOrdering is documented

Single set of callbacks per photo settingsOrdering is documentedVehicle for resolving indeterminate settings

AVCapturePhotoCaptureDelegate Usage

capturePhoto(with:delegate:

flashMode = .auto, autoSIS = trueAVCapturePhotoCaptureDelegate

flashMode = .auto, autoSIS = true

AVCapturePhotoCaptureDelegate

AVCaptureResolvedPhotoSettingsflash = On, SIS = Off

willBeginCaptureForResolvedSettings

AVCaptureResolvedPhotoSettingsflash = On, SIS = Off

willCapturePhotoForResolvedSettings

didCapturePhotoForResolvedSettings

CMSampleBuffer

didFinishProcessingPhotoSampleBuffer

didFinishCaptureForResolvedSettings

AVCapturePhotoCaptureDelegate Specifics

Delegate callbacks track a single photo capture request

Delegate callbacks track a single photo capture requestAVCapturePhotoOutput holds a weak reference to your delegate

Delegate callbacks track a single photo capture requestAVCapturePhotoOutput holds a weak reference to your delegateAll callbacks are marked @optional

Delegate callbacks track a single photo capture requestAVCapturePhotoOutput holds a weak reference to your delegateAll callbacks are marked @optionalSome callbacks are required at runtime depending on your photo settings

Delegate callbacks track a single photo capture requestAVCapturePhotoOutput holds a weak reference to your delegateAll callbacks are marked @optionalSome callbacks are required at runtime depending on your photo settingsAll callbacks pass an instance of AVCaptureResolvedPhotoSettings

// Initiating a photo capture using AVCapturePhotoOutput

func takeHighResolutionPhoto() { let settings = AVCapturePhotoSettings() settings.isHighResolutionPhotoEnabled = true photoOutput.capturePhoto(with: settings, delegate: self) }

func takeFlashPhoto() { let settings = AVCapturePhotoSettings() settings.flashMode = .auto

photoOutput.capturePhoto(with: settings, delegate: self) }

func takeBGRAPhoto() { let bgraFormat: [String : AnyObject] = [kCVPixelBufferPixelFormatTypeKey as String : NSNumber(value: kCVPixelFormatType_32BGRA)] let settings = AVCapturePhotoSettings(format: bgraFormat) photoOutput.capturePhoto(with: settings, delegate: self) }

AVCaptureResolvedPhotoSettings Properties

public var uniqueID: Int64 { get }

public var uniqueID: Int64 { get }public var photoDimensions: CMVideoDimensions { get }

public var uniqueID: Int64 { get }public var photoDimensions: CMVideoDimensions { get }public var isFlashEnabled: Bool { get }

public var uniqueID: Int64 { get }public var photoDimensions: CMVideoDimensions { get }public var isFlashEnabled: Bool { get }public var isStillImageStabilizationEnabled: Bool { get }

Bracketed Capture Support

Review 2014 Session 508: Camera Capture Manual Controls

Review 2014 Session 508: Camera Capture Manual ControlsAuto Exposure and Manual Exposure Brackets

Review 2014 Session 508: Camera Capture Manual ControlsAuto Exposure and Manual Exposure BracketsAVCapturePhotoBracketSettings is a subclass of AVCapturePhotoSettings

Review 2014 Session 508: Camera Capture Manual ControlsAuto Exposure and Manual Exposure BracketsAVCapturePhotoBracketSettings is a subclass of AVCapturePhotoSettings public var bracketedSettings: [AVCaptureBracketedStillImageSettings] { get }public var isLensStabilizationEnabled: Bool

func capture(_ captureOutput: AVCapturePhotoOutput, didFinishProcessingPhotoSampleBuffer photoSampleBuffer: CMSampleBuffer?, previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: NSError?) { // bracketSettings tells you which item in your array of bracketedSettings // this image corresponds to. }

Deprecated in iOS 10

AVCaptureDevice.flashActive

AVCaptureDevice isFlashModeSupported(_:)

AVCaptureDevice.flashMode

Deprecated in iOS 10

AVCaptureDevice.flashActive

AVCaptureDevice isFlashModeSupported(_:)

AVCaptureDevice.flashMode

AVCapturePhotoOutput.isFlashScene

AVCapturePhotoOutput.supportedFlashModes

AVCapturePhotoSettings.flashMode

AVCapturePhotoOutput Benefits

Easier bookkeeping

Easier bookkeepingImmediate settings resolution

Easier bookkeepingImmediate settings resolutionConfident request tracking

Easier bookkeepingImmediate settings resolutionConfident request trackingExpandable palette of callbacks

Capturing memoriesLive Photos

“A still photo captures an instant frozen in time. With Live Photos, you can turn those instants into unforgettable living memories.”

Moment or memory?What is a Live Photo

It’s a moment!

It’s a moment!• Full resolution still image (12 MP JPEG)

It’s a moment!• Full resolution still image (12 MP JPEG)• Same quality as non-Live Photo (SIS / OIS)

It’s a moment!• Full resolution still image (12 MP JPEG)• Same quality as non-Live Photo (SIS / OIS)• “Frictionless” capture

It’s a memory!

It’s a memory!• A short movie encompassing the still time

It’s a memory!• A short movie encompassing the still time• Screen resolution (1440x1080 or 1290x960)

It’s a memory!• A short movie encompassing the still time• Screen resolution (1440x1080 or 1290x960)• Includes audio

It’s a memory!• A short movie encompassing the still time• Screen resolution (1440x1080 or 1290x960)• Includes audioIn iOS 9.1 — “Shoe shot” auto trimming

It’s a memory!• A short movie encompassing the still time• Screen resolution (1440x1080 or 1290x960)• Includes audioIn iOS 9.1 — “Shoe shot” auto trimmingNew in iOS 10 — Video stabilization

It’s a memory!• A short movie encompassing the still time• Screen resolution (1440x1080 or 1290x960)• Includes audioIn iOS 9.1 — “Shoe shot” auto trimmingNew in iOS 10 — Video stabilizationNew in iOS 10 — Interruption-freemusic during capture

It’s a moment and a memoryWhat is a Live Photo

Live Photo .MOV Asset

Metadata

0.75 1.5 2.25 3

Photo Display Time

H.264, 1440x1080, 15 FPS, 7 Mbps

44.1 kHz, PCM, Mono or Stereo

Capturing Live Photos

Check AVCapturePhotoOutput.isLivePhotoCaptureSupported

Check AVCapturePhotoOutput.isLivePhotoCaptureSupportedLive Photo capture is only supported with the AVCaptureSessionPresetPhoto

Check AVCapturePhotoOutput.isLivePhotoCaptureSupportedLive Photo capture is only supported with the AVCaptureSessionPresetPhotoOpt in using AVCapturePhotoOutput.isLivePhotoCaptureEnabled = true

Check AVCapturePhotoOutput.isLivePhotoCaptureSupportedLive Photo capture is only supported with the AVCaptureSessionPresetPhotoOpt in using AVCapturePhotoOutput.isLivePhotoCaptureEnabled = trueAdd an AVCaptureDeviceInput for the microphone to your AVCaptureSession

Check AVCapturePhotoOutput.isLivePhotoCaptureSupportedLive Photo capture is only supported with the AVCaptureSessionPresetPhotoOpt in using AVCapturePhotoOutput.isLivePhotoCaptureEnabled = trueAdd an AVCaptureDeviceInput for the microphone to your AVCaptureSessionPresence of AVCaptureMovieFileOutput disables Live Photo capture

// Live Photo Capture Example

func takeLivePhoto() { let settings = AVCapturePhotoSettings() settings.isHighResolutionPhotoEnabled = true settings.livePhotoMovieFileURL = URL(fileURLWithPath: "\(NSTemporaryDirectory())/\(_uniqueAppSignature)_\(settings.uniqueID)") // Optional Movie-level Metadata var metadataItems = [AVMetadataItem]() let authorMetadataItem = AVMutableMetadataItem() authorMetadataItem.keySpace = AVMetadataKeySpaceCommon authorMetadataItem.key = AVMetadataCommonKeyAuthor authorMetadataItem.value = "Brad Ford" metadataItems.append(authorMetadataItem) settings.livePhotoMovieMetadata = metadataItems photoOutput.capturePhoto(with: settings, delegate: self) }

willBeginCaptureForResolvedSetting

Live Photo Delegate Methods

livePhotoMovieURL = someURL

AVCaptureResolvedPhotoSettingslivePhotoMovieDimensions = {1440x1080}

CMSampleBuffer

didFinishRecordingLivePhotoMovieForEventualFileAtURL

didFinishProcessingLivePhotoToMovieFileAtURL

didFinishCaptureForResolvedSettings

// Live Photo Capture Delegate Methods

func capture(_ captureOutput: AVCapturePhotoOutput, didFinishProcessingPhotoSampleBuffer photoSampleBuffer: CMSampleBuffer?, previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: NSError?) { if let photoSampleBuffer = photoSampleBuffer, data = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer: photoSampleBuffer, previewPhotoSampleBuffer: previewPhotoSampleBuffer) { do { try data.write(to: _uniqueJPEGFilePath, options: .atomicWrite) } catch { // Handle error } } }

AVCapturePhotoCaptureDelegate Tip

Some types of photo captures deliver multiple assets

Some types of photo captures deliver multiple assets Instantiate a new AVCapturePhotoCaptureDelegate object for each photo request

Some types of photo captures deliver multiple assets Instantiate a new AVCapturePhotoCaptureDelegate object for each photo requestAggregate assets in the delegate object

Some types of photo captures deliver multiple assets Instantiate a new AVCapturePhotoCaptureDelegate object for each photo requestAggregate assets in the delegate objectDispose of the object after didFinishCaptureForResolvedSettings:

PHLivePhoto PHLivePhotoViewPHImageManager

PHLivePhotoEditingContext

DemoLive Photo Capture and Editing

AVCam and LivePhotoEditor

Live Photo Capture in AVCam

Separate video recording and photo modes

Separate video recording and photo modesShows proper “Live” badging technique

Separate video recording and photo modesShows proper “Live” badging techniqueSaves assets to Photo Library

Separate video recording and photo modesShows proper “Live” badging techniqueSaves assets to Photo LibrarySample code available now!

More Live Photo Editing in Session 505

Live Photo Editing and RAW Processingwith Core Image Nob Hill Thursday 11:00AM

Live Photo Capture Suspension

P1 Obnoxious Foghorn

P2P1 Obnoxious Foghorn

output.isLivePhotoCaptureSuspended = true

output.isLivePhotoCaptureSuspended = false

Live Photo Capture Support

iPhone 6s

iPhone 6s Plus

iPhone SE

9.7-inch iPad Pro

with a dash of DNG on the sideRAW Photo Capture

What is RAW?

Color Filter Array

What is RAW?

Color Filter Array

Light From Scene

What is RAW?

Sensor Array

Color Filter Array

Light From Scene

What is Stored in a RAW file?

Intensity of red, green, or blue light hitting the sensor

Intensity of red, green, or blue light hitting the sensorBayer pattern as metadata

Intensity of red, green, or blue light hitting the sensorBayer pattern as metadataA lot of other metadata

What Do RAW Converters Do?

Demosaic Bayer filter patternApply white balanceInterpret Colorimetric InfoCorrect GammaReduce NoiseSharpen

Why RAW?

Bake-time flexibility

Why RAW?

Bake-time flexibilityNo compression

Why RAW?

Bake-time flexibilityNo compressionMore bits

Why RAW?

Bake-time flexibilityNo compressionMore bitsLots of headroom for editing

Why RAW?

Bake-time flexibilityNo compressionMore bitsLots of headroom for editingGreater artistic freedom to interpret the image data in post

Why JPEG?

Lovingly baked by Apple

Why JPEG?

Lovingly baked by AppleMuch faster rendering

Why JPEG?

Lovingly baked by AppleMuch faster renderingMultiple image fusion (e.g. for stabilization)

Why JPEG?

Lovingly baked by AppleMuch faster renderingMultiple image fusion (e.g. for stabilization)Smaller file size

// CVPixelBuffer.h

/* Bayer 14-bit Little-Endian, packed in 16-bits, ordered G R G R... alternating with B G B G... */

kCVPixelFormatType_14Bayer_GRBG = OSType(“grb4”), /* Bayer 14-bit Little-Endian, packed in 16-bits, ordered R G R G... alternating with G B G B... */

kCVPixelFormatType_14Bayer_RGGB = OSType(“rgg4”),

/* Bayer 14-bit Little-Endian, packed in 16-bits, ordered B G B G... alternating with G R G R... */

kCVPixelFormatType_14Bayer_BGGR = OSType(“bgg4”),

/* Bayer 14-bit Little-Endian, packed in 16-bits, ordered G B G B... alternating with R G R G... */

kCVPixelFormatType_14Bayer_GBRG = OSType(“gbr4”),

Capturing RAW with AVCapturePhotoOutput

RAW is only supported when using a photo format ( AVCaptureSessionPresetPhoto )

RAW is only supported when using a photo format ( AVCaptureSessionPresetPhoto )Rear camera only

RAW is only supported when using a photo format ( AVCaptureSessionPresetPhoto )Rear camera onlyRAW bracketed captures are supported

// Capturing RAW Photos

func takeRawPhoto()

let rawFormat = photoOutput.availableRawPhotoPixelFormatTypes.first!.uint32Value

let settings = AVCapturePhotoSettings(rawPixelFormatType: rawFormat)

// RAW photo settings have autoStillImageStabilizationEnabled set to NO

// highResolutionPhotoEnabled is also NO as it’s meaningless in RAW

photoOutput.capturePhoto(with: settings, delegate: self)

func takeRawPhoto()

// RAW related AVCapturePhotoCaptureDelegate methods

func capture(_ captureOutput: AVCapturePhotoOutput,

didFinishProcessingRawPhotoSampleBuffer rawSampleBuffer: CMSampleBuffer?,

previewPhotoSampleBuffer: CMSampleBuffer?,

resolvedSettings: AVCaptureResolvedPhotoSettings,

bracketSettings: AVCaptureBracketedStillImageSettings?,

error: NSError?)

// Handle the RAW sample buffer

error: NSError?)

RAW + Processed Image Support

Processed image may be JPEG, or an uncompressed format

Processed image may be JPEG, or an uncompressed formatThe processed image is delivered to didFinishProcessingPhotoSampleBuffer:

Processed image may be JPEG, or an uncompressed formatThe processed image is delivered to didFinishProcessingPhotoSampleBuffer: The RAW image is delivered to didFinishProcessingRAWPhotoSampleBuffer:

Processed image may be JPEG, or an uncompressed formatThe processed image is delivered to didFinishProcessingPhotoSampleBuffer: The RAW image is delivered to didFinishProcessingRAWPhotoSampleBuffer:RAW + processed brackets are supported

Processed image may be JPEG, or an uncompressed formatThe processed image is delivered to didFinishProcessingPhotoSampleBuffer: The RAW image is delivered to didFinishProcessingRAWPhotoSampleBuffer:RAW + processed brackets are supportedRAW + still image stabilization processed image is NOT supported

func takeRAWPlusJPEGPhoto()

let processedFormat = photoOutput.availablePhotoCodecTypes.first!

let settings = AVCapturePhotoSettings(rawPixelFormatType: rawFormat, processedFormat: [AVVideoCodecKey : processedFormat])

// highResolutionPhotoEnabled YES or NO applies only to the processed photo

Storing RAW Buffers

// Writing RAW to DNG

func capture(_ captureOutput: AVCapturePhotoOutput, didFinishProcessingRawPhotoSampleBuffer rawSampleBuffer: CMSampleBuffer?, previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: NSError?) { if let rawSampleBuffer = rawSampleBuffer, data = AVCapturePhotoOutput.dngPhotoDataRepresentation(forRawSampleBuffer: rawSampleBuffer, previewPhotoSampleBuffer: previewPhotoSampleBuffer) { let filePath = (_uniqueFilePath as NSString).appendingPathExtension(".dng")! do { try data.write(to: URL(fileURLWithPath: filePath), options: .atomicWrite) } catch { // Handle error } } }

// Writing RAW to DNG

DemoRAW Capture and Editing

AVCamManual and RawExpose

Advances in iOS Photography - Apple Inc....Camera Capture Manual Controls (iOS 8 / Yosemite) WWDC...

Documents

Photography. Photography as Witness MoMA Photography Theme

WATER RESOURCES ASSOCIATION OF YOLO COUNTY · 2. Approve Agenda and Adding Items to the Posted Agenda: ... (FA14) • Lower Cache Creek Aerial Photography Project (FA16) WATER RESOURCES

Agenda Digital Photography Free Response Will follow, please read this! Complete the worksheet from…

introduction of ios development , iOS Classes Ahmedabad , iOS Training Ahmedabad

Cisco Unified Communications Manager and Cisco IOS … · About Cisco IOS and Cisco IOS XE Software Documentation Documentation Organization iv Cisco IOS Documentation Set Cisco IOS

AGENDA - norinrad10.files.wordpress.com 08, 2018 · AGENDA 2601 Bransford Avenue, Nashville, ... MNPS fleet of over 8,300 Apple IPads and other IOS devices for the ... Memorandum

Multicasting in IOS, IOS XE, IOS XR for IPv4 and IPv6 · Multicasting in IOS, IOS XE, IOS XR for IPv4 and IPv6 Disclaimer. I don’t pretend to cover everything about the protocols

iOS Photography

site.judbrasov.ro · apmsed mppap as soŒ1 ap .10)uaaua a!ios/ios as.ms tup a!ios/ios 00.1011 ap auk) us.uns a!ios/ios • 79 ti!P '9 a!ios/ios a!ios/ios ti!P a!ios/ios .101!.munq

RAD Studio XE4 - iOS Tutorials: Delphi iOS Application ...docs.embarcadero.com/products/rad_studio/radstudioXE4/iOS Tutorial... · iOS Tutorials: Delphi iOS Application Development

Smart Phone Photography July 20, 2015 · Smart Phone Photography July 20, 2015 Bob Walker Mike Schlitt . Agenda •Introductions – Bob and Mike –oth Apple/Mac users –Examples

iOS Demo Day - iOS Dev Portugal

iOS Tutorial - univie.ac.atvda.univie.ac.at/Teaching/HCI/16s/materials/hci-ios-tutorial.pdf · iOS Tutorial Human-Computer-Interaction and Psychology ... Android iOS Cordova • Languages:

Today’s Agenda - University of Michigan...2016/01/01 · Today’s Agenda • Welcome • New Hires • Show and Tell: Giving Blueday • Show and Tell: Michigan Photography •

Comparing iOS 7 with iOS 6 design

What is ios, shutter speed and aperture photography

audacityit · 2020. 4. 9. · Rokomari Place your screenshot here. ANDROID JeteChao Place your screenshot here. iOS BAT LOS2017. iOS IT Finder. iOS Book Reader. iOS Foodviser. iOS

Harnessing iOS to Create Magic in Your Apps...Engineering Manager, Mobile Apps Harnessing iOS to Create Magic in Your Apps. Apple Store App. Agenda We are here ... •Calendar Events

Build Once, Deploy Everywhere: Support Both iOS and Android · PDF fileBuild Once, Deploy Everywhere: Support Both iOS and Android Andrea Schiller . #mstrworld Agenda • Introduction

FILE OPEN MODES File open modes n ios::app n ios::ate n ios::binary n ios::in n ios::out n ios::trunc n ios::nocreate n ios::noreplace