Single Image Portrait Relighting™天成.pdf · Single Image Portrait Relighting Tiancheng Sun1,...

Single Image Portrait Relighting

Tiancheng Sun1, Jonathan T. Barron2, Yun-Ta Tsai2, Zexiang Xu1, Xueming Yu3, Graham Fyffe3, Christoph Rhemann3, Jay Busch3,

Paul Debevec3, Ravi Ramamoorthi1

1University of California, San Diego, 2Google Research, 3Google

Photography & RecordingAllowed

Overview

light from the back

Overview

light from the back shadow on the face

Overview

want to add dramatic lightinglight from the back shadow on the face

Overview

want to add dramatic lightinglight from the back shadow on the face

Change the lighting of any portrait after capture using post-processing algorithm

Overview

Portrait Relighting System

Overview

another lighting

Overview

another lighting

Overview

I want to rotate the lighting a little bit.

Overview

I want to rotate the lighting a little bit.

Previous work

Previous work• Light Stage

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

: capture ~100 images and do image-based relighting

Previous work

• Deep image-based relighting

Xu, Zexiang, et al. "Deep image-based relighting from optimal sparse samples." SIGGRAPH 2018

Previous work

capture 5 images and do relighting via neural network

Previous work

• Portrait lighting transfer

Shu, Zhixin, et al. "Portrait lighting transfer using a mass transport approach." SIGGRAPH 2018

Previous work

transfer lighting from one portrait to another

Previous work

Normal

Shading

Albedo

• SfSNet

Sengupta, Soumyadip, et al. "SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild'." CVPR 2018.

Previous work

Normal

Shading

Albedo

• SfSNet

: deep intrinsic decomposition mostly trained on synthetic faces

Previous work

Normal

Shading

Albedo

• SfSNet

: deep intrinsic decomposition mostly trained on synthetic faces

Previous work

Overview

• Goal: practical relighting on single portrait image

Overview

• Practical in detail:

• Robust to the pose and camera view

• Work well on natural lightings

• Adapt to high-resolution images

• Run at interactive rate

Overview

• Practical in detail:

• Robust to the pose and camera view

• Work well on natural lightings

• Adapt to high-resolution images

• Run at interactive rate

• Solution: Deep Neural Network + Real Face Data.

Overview

Method

portrait under lighting A

Method

(Neural Network)portrait under

lighting A

Method

lighting Alighting B

Method

lighting Aportrait under

lighting Blighting B

Method

lighting Blighting A lighting B

Method

How can we get the portrait pair for training?

Method: Data

Method: DataLight Stage

Method: DataLight Stage One-Light-At-a-Time scans (OLAT)

Method: Data

captured OLAT captured OLATDebevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

lighting

Method: Data

captured OLAT captured OLATDebevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

lighting

Method: Data

captured OLAT captured OLAT

latitude-longitude representation

lighting

Method: Data

lighting

Method: Data

lighting

x x+ + …… =

Method: Data

lighting

x x+ + …… =

Method: Data

captured OLAT captured OLAT relit image (background removed)

lighting

x x+ + …… =

Method: Data

captured OLAT captured OLAT relit image (background removed)

Wadhwa, Neal, et al. "Synthetic depth-of-field with a single-camera mobile phone." SIGGRAPH 2018Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

Method: Data

• OLAT images• 22 people (18 training, 4 validation), each 3~5 facial expressions

Method: Data

• Each OLAT captured with 7 cameras in 6 seconds.

Method: Data

• HDR lighting environments

Method: Data

• HDR lighting environments• ~2000 indoor HDR lighting

from Laval Dataset

Method: Data

from Laval Dataset

• ~1000 outdoor HDR lighting from the web

Method: Data

from Laval Dataset

• ~1000 outdoor HDR lighting from the web

Method: Data

• Total: 226,800 portrait and lighting pairs for training

Method: Training

Encoder DecoderBottleneck

Method: Training

• Task 1: Complete relighting

Method: Training

source image

Method: Training

source light

source image

Method: Training

source light target light

source image

Method: Training

source image target image

Method: Training

• Task 1: Complete relighting L1 loss

Log L1 loss

Method: Training

Method: Training• Task 2: Illumination retargeting

source image

source light

source image

source light

source image

Rotate

source image

Rotate

L1 loss

Log L1 loss

Method: Training

Method: Training• Network structure• U-Net

k x k conv layer concatenation

kweighted average tiling

kk-dimensional

input/label

kk-dimensional

activation

256 x 256 128 x 128 64 x 64 32 x 32 16 x 16Spatial Resolution:

Method: Training• Network structure• U-Net

kk-dimensional

input/label

kk-dimensional

activation

SourceImage 64 643 128 1283 256 2563 512 5123 5123 5123

OutputSourceLight

16 x 32 x 33

297 16 x 32 x 13 3 3 3 33

TrueSourceLight

Method: Training• Network structure• U-Net• Predict and feed in light at bottleneck

kk-dimensional

input/label

kk-dimensional

activation

SourceImage 64 643 128 1283 256 2563 512 5123 5123 5123

OutputSourceLight

16 x 32 x 33

297 16 x 32 x 13 3 3 3 33

TrueSourceLight

confidence learning module

kk-dimensional

input/label

kk-dimensional

activation

SourceImage 64 643 128 1283 256 2563 512 5123 5123 5123

OutputSourceLight

16 x 32 x 33

297 16 x 32 x 13 3 3 3 33

TrueSourceLight

confidence learning module

kk-dimensional

input/label

kk-dimensional

activation

SourceImage 64 643 128 1283 256 2563 512 5123 5123 5123

OutputSourceLight

16 x 32 x 33

297 16 x 32 x 13 3 3 3 33

TrueSourceLight

3 512TargetLight3

256128

51225625612812864643 3 3 3

TrueTargetImage

OutputTargetImage

Method: Training

Method: Training• Confidence learning

Method: Training

Several conv layers

• Confidence learning

Method: Training

Several conv layers

Method: Training

Several conv layers

resolution of the light

Method: Training

Several conv layers

Light prediction on each image patch

Method: Training

Several conv layers

Confidence of prediction on each

image patch

• Confidence learning• Predict the confidence of light prediction

Method: Training

Several conv layers

image patch

• Confidence learning• Predict the confidence of light prediction

Reshape

Method: Training

Several conv layers

image patch

• Confidence learning• Predict the confidence of light prediction• Allow network to say “I don’t know”

Reshape

Results: Validation

Results: Validation➤ Relit images for complete relighting

source image

source image target image ours [Barron & Malik 2015][Sengupta et al. 2018] [Li et al. 2018]

Results: Validation

➤ Relight images with predicted light as target light

Results: Validation

source image

Results: Validation

source image with self-supervision

withoutself-supervision

Results: Validation

Results: Validation➤ Comparison with portrait lighting transfer

reference image

source image

reference image

source image

reference image

Extract light from reference

source image

reference image

Apply to source image

source image

groundtruth oursreference image

source image

groundtruth ours

[Shih et al. 2014] [Shu et al. 2018]

reference image

Results: Validation

Results: Validation➤ Evaluation on lighting prediction

source image

ground truth

source image

ground truth

ours w/o confidence learning

source image

ground truth

ours w/o confidence learning

[Barron & Malik 2015]

[Sengupta et al. 2018]

Results: Images in the wild

Results: Images in the wildInput Image Relit Image

Results: Images in the wildInput Image Relit ImageInput Image Relit Image

Results: Images in the wild

Limitations

LimitationsInput Image Relit Image

• Complex shadows

• Specular highlights

• Overexposed pixels

• Complex shadows

• Over-smoothing

• Complex shadows

• Over-smoothing

• Unseen high-saturation color

Conclusion

• Learn the relighting function on portraits using Light Stage data

Conclusion

• Take home message:

Conclusion

• For human faces, use real data.

Conclusion

• For human faces, use real data.

• End-to-end training vs assuming models.

Acknowledgement• This work was funded in part by a Jacobs Fellowship, the Ronald L.

Graham Chair, and the UC San Diego Center for Visual Computing.

• Thanks to Zhixin Shu and Yichang Shih for the help on baseline algorithms

• Thanks to Jean-François Lalonde for providing the indoor lighting dataset.

• Thanks to Peter Denny for coordinating dataset capture.

• Thanks to all the anonymous volunteers in the dataset from Google, UCSD and UCLA

Also presenting in poster session today at 12:15-1:15.

input imagegenerated portraits under new illuminations

Single Image Portrait Relighting™天成.pdf · Single Image Portrait Relighting Tiancheng Sun1,...

Documents

Video Relighting Using Infrared Illumination · Video Relighting Using Infrared Illumination Oliver Wang and James Davis and Erika Chuang and Ian Rickard and Krystle de Mesa and Chirag

PORTRAIT PHOTOGRAPHY - National Portrait Gallery · PDF fileTeachers’ Resource Portrait Photography National Portrait Gallery Information and Activities for Secondary Teachers of

Interactive Virtual Relighting of Real Scenes · Interactive Virtual Relighting of Real Scenes C´eline Loscos, George Drettakis and Luc Robert Abstract – Computer augmented reality

Data-intensive Image Based Relighting Biswarup Choudhury Sharat Chandran

Object Image Relighting through Patch Match Warping and ...jinxin.me/downloads/papers/022-ICVRV2016/ICVRV2016-Final.pdfbased object relighting through patch match warping and color

AbstractNithin Raghavan 1Utkarsh Singhal Ravi Ramamoorthi3 Jonathan T. Barron2 Ren Ng1 1University of California, Berkeley 2Google Research 3University of California, San Diego Abstract

Portraits:€¦ · Web viewContents. Portraits:2. Portrait 13. Portrait 24. Sources:5. Portrait 36. Portrait 48. Sources:9. Portrait 59. Bibliography11. Image Source11. Appendix12

A Neural Rendering Framework for Free-Viewpoint Relighting

BP Portrait Award, National Portrait Gallery | Culture Whisper · BP Portrait Award, National Portrait Gallery | Culture Whisper

Face Relighting with Radiance Environment Maps · 2018-01-04 · Face Relighting with Radiance Environment Maps Zhen Wen University of Illinois Urbana Champaign zhenwen@ifp.uiuc.edu

Cosine Lobe Based Relighting from Gradient Illumination … · 2009. 5. 6. · Cosine Lobe Based Relighting from Gradient Illumination Photographs Graham Fyffe University of Southern

Virtual Cinematography: Relighting through …cs194-26/fa14/Papers/debevec06.pdfVirtual Cinematography: Relighting through Computation Figure 1.The ﬁrst photograph,Joseph Nicéphore

Image Based Relighting - Linköping Universitywebstaff.itn.liu.se/.../Assignments/Lab3/prac3.pdf · 2006-10-09 · Image Based Relighting TNM078 – Image Based Rendering Jonas Unger∗,

Cigarette relighting –implications for tobacco ... · Masoni c Cancer Center Comprehensive Cancer Center designated by the National Cancer Institute GOAL Cigarette relighting –implications

Ultrasensitive detection and characterization of ... · E. Hendry1,T.Carpy2, J. Johnston2,3,M.Popland2,R.V.Mikhaylovskiy1, A. J. Lapthorn2,S.M.Kelly4, L. D. Barron2, N. Gadegaard3

2012 BP Portrait Award der National Portrait Gallery, London 2010 BP Portrait Award der National Portrait Gallery, London 2009 BP Portrait Award der

Image-based relighting with office room lighting

Eyes for Relighting Extracting environment maps for use in integrating and relighting scenes (Noshino and Nayar)

A Portrait of Canadian Foundation Philanthropypfc.ca/.../portrait-cdn-philanthropy-sept2017-en.pdf · a portrait of Canadian foundation philanthropy a portrait of Canadian foundation

Relighting Acquired Models of Outdoor Scenesallen/PAPERS/3dim05.pdf · 2005. 8. 8. · Relighting Acquired Models of Outdoor Scenes Alejandro Troccoli Peter K. Allen Department of