Deep Learning for Graphics Supervised...

Niloy Mitra Iasonas Kokkinos Paul Guerrero Vladimir Kim Kostas Rematas Tobias Ritschel

UCL UCL/Facebook UCL Adobe Research U Washington UCL

Deep Learning for Graphics

Supervised Applications

EG Course Deep Learning for Graphics

Timetable

Niloy Iasonas Paul Vova Kostas Tobias

Introduction X X X X

Theory X

NN Basics X

Supervised Applications X X X

Data X

Unsupervised Applications X

Beyond 2D X X X

Outlook X X X X X X

Fast (shared convolutions) Simple (dense)

Fully-Convolutional Network (FCN)

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. CVPR, 2015

FCN-based semantic segmentation

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. Yuille, Deeplab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, PAMI 2016

FCN-CRFs: Deeplab

Ground truth FCN FCN-DCRF

Deeplab v2 results

Ground truth FCN FCN-DCRF

Object Detection: Fast(er)-RCNN

• Fast/Faster R-CNNGood speed

Good accuracy

Intuitive

Easy to use

Ross Girshick. “Fast R-CNN”. ICCV 2015.Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”. NIPS 2015.

Mask R-CNN

• Mask R-CNN = Faster R-CNN with FCN on RoIs

Faster R-CNN

FCN on RoI

EG Course “Deep Learning for Graphics”

Mask R-CNN results on COCO

• 1 keypoint = 1-hot “mask”

• Human pose = 17 masks

• Softmax over spatial locations• e.g. 562-way softmax on 56x56

Mask R-CNN for Human Keypoint Detection

Mask R-CNN frame-by-frame

I. Kokkinos, UberNet: Training a Universal CNN for Low- Mid- and High-Level Vision, CVPR 2017

https://github.com/jkokkin/UberNet

UberNet: a “universal” network for all tasks

“Inverse graphics”: understand how an image was generated from a scene

If we focus on a single object category: surface-based models

UberNet: Universal Network

DensePose: Unified model

What is the ultimate vision task?

R. A. Guler, G. Trigeorgis, E. Antonakos, P. Snape, S. Zafeiriou, I. Kokkinos,

DenseReg: Fully Convolutional Dense Shape Regression In-the-Wild, CVPR 2017

DenseReg: dense image-to-face regression

R. A. Guler, N. Neverova, I. Kokkinos “DensePose: Dense Human Pose Estimation In The Wild”, CVPR’18

DensePose-RCNN: ~25 FPS

DensePose: dense image-to-body correspondence

SFSNet: incorporating image formation in model

SfSNet: Learning Shape, Reflectance and Illuminance of Faces ‘in the wild' SoumyadipSengupta Angjoo Kanazawa Carlos D. Castillo David W. Jacobs, CVPR 2018

Beyond single frames: end-to-end optical flow

End-to-end Structure From Motion

• DeMoN: Depth and Motion Network for Learning Monocular Stereo, B. Ummenhofer, et al, CVPR 2017

• Unsupervised learning of depth and ego-motion from video, T Zhou, M Brown, N Snavely, DG Lowe, CVPR 2017

Monocular depth & normal estimation

• D. Eigen and R. Fergus, Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture, ICCV 2015

Graphics applications

Sketch Simplification

• Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup, Simon-Serra et al., 2016

• Deep Extraction of Manga Structural Lines, Li et al., 2017

Sketch Simplification: Learning to Simplify

Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup, Simo-Serra et al.

Sketch Simplification: Learning to Simplify

• Loss for thin edges saturates easily

• Authors take extra steps to align input and ground truth edges

Pencil: inputRed: ground truth

Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup, Simo-Serra et al.

Image Decomposition

• A selection of methods:

• Direct Instrinsics, Narihira et al., 2015

• Learning Data-driven Reflectance Priors for Intrinsic Image Decomposition, Zhou et al., 2015

• Decomposing Single Images for Layered Photo Retouching, Innamorati et al. 2017

Image Decomposition: DecomposingSingle Images for Layered Photo Retouching

Colorization

• Concurrent methods:• Let there be Color!, Iizuka et al., 2016

• Colorful Image Colorization, Zhang et al. 2016

• Learning Representations for Automatic Colorization, Larsson et al., 2016

• Real-Time User-Guided Image Colorization with Learned Deep Priors, Zhang et al. 2017

Colorization: Let There Be Color!

Let there be Color!: Iizuka et al.

Colorization: Colorful Image Colorization

inputdirect regression probability distr.

output

Image Credit: Colorful Image Colorization, Zhang et al.

Mesh Labeling / Segmentation

3D Mesh Labeling via Deep Convolutional Neural Networks, Guo et al. 2016

Mesh Labeling / Segmentation

3D Mesh Labeling via Deep Convolutional Neural Networks, Guo et al.

LDR to HDR Image Reconstruction:

• Concurrently:

• Deep Reverse Tone Mapping, Endo et al. 2017

• HDR image reconstruction from a single exposure using deep CNNs, Eilertsen et al. 2017

Reflectance Maps

• Paint a sphere as if it is made of a material under a certain illumination of another object in a photo

Deep Reflectance Maps. Rematas et al. CVPR 2015

DeLight

• Factor BRDF and (HDR) Illumination

Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning. Georgoulis et al. PAMI 2017

3D volumes form Xrays

Single-Image Tomography: 3D Volumes from 2D Cranial X-Rays. Henzler et al. EG 2018

Deep Shading

• Paint a z-buffer like a path tracer (AO, DOF, MB)

Deep Shading, Nalbach et al. EGSR 2017

Rendering Atmospherics

Speed up approx. 24 x Speed up approx. 24 x

Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks, Kallweit et al. SIGGRAPH Asia 2017

Rendering Atmospherics: RPNN

Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks, Kallweit et al. SIGGRAPH Asia 2017

In: Hierarchical representation of a cloud patchOut: incoming indirect radiance at patch center(incoming direct radiance is computed directly)

Denoising Renderings

• Concurrent:

• Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings, Bako et al. 2017

• Interactive Reconstruction of Monte Carlo Image Sequences using a Recurrent Denoising Autoencoder, Chaitanya et al. 2017 (more on Autoencoders later)

Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings, Bako et al.

Denoising Renderings:

Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings, Bako et al. SIGGRAPH 2017

Geometry Abstraction / Simplification

Learning Shape Abstractions by Assembling Volumetric Primitives, Tulsiani et al. 2016

Geometry Abstraction / Simplification:

Learning Shape Abstractions by Assembling Volumetric Primitives, Tulsiani et al. 2016

Procedural Parameter Estimation

Interactive Sketching of Urban Procedural Models, Nishida et al. 2016

Procedural Parameter Estimation:Interactive Sketching of Urban Procedural Models

Interactive Sketching of Urban Procedural Models, Nishida et al.

Audio-driven facial animation

Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion, Karras et al. 2017

3D Pose Estimation: VNECT

VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera, Mehta et al., SIGGRAPH 2017

skeleton joint heatmapand 3d positions50

Thank you!

http://geometry.cs.ucl.ac.uk/dl4g/

Deep Learning for Graphics Supervised...

Documents

Lecture 3: Deep Learning Optimizations - GitHub Pagesuvadlc.github.io/lectures/nov2019/lecture3.pdf · Lecture 3: Deep Learning Optimizations Deep Learning @ UvA. UVA DEEP LEARNING

Large-scale Deep Unsupervised Learning using Graphics Processors

Introduction to Deep Learning - Computer Graphics€¦ · Introduction to Deep Learning CS468 Spring 2017 Charles Qi. What is Deep Learning? Deep learning allows computational models

ARM A55 Cortex - University Of Illinois · NEON Pipeline SIMD architecture extension Audio/Video encoding/decoding 2D/3D Graphics Rendering AI (Machine Learning/Deep Learning/Computer

Deep Learning Lab Course 2017 (Deep Learning …ais.informatik.uni-freiburg.de/teaching/ws17/deep...Deep Learning Lab Course 2017 (Deep Learning Practical) Labs: (Computer Vision)

UNSUPERVISED DEEP LEARNING - TAUweb.eng.tau.ac.il/.../uploads/2016/12/Unsupervised-Deep-Learning.pdf · UNSUPERVISED DEEP LEARNING Erez Aharonov Noam Eilon Deep Learning Seminar School

Deep Convolutional Inverse Graphics Network

Learning Deep Learning

Real Learning graphics

Deep Convolutional Inverse Graphics Network - arXiv · Deep Convolutional Inverse Graphics Network Tejas D ... decoder networks to enable ... the basic structure of the Deep Convolutional

Deep Learning in real world @Deep Learning Tokyo

Chapter 1: Introduction to Messaging System...Chapter 3: Deep Dive into Kafka Producers. Graphics Bundle [ 10 ] Graphics Bundle [ 11 ] Chapter 4: Deep Dive into Kafka Consumers. Graphics

Deep Learning - Oxford Statisticspalamara/teaching/SML19/HT19_lecture15.pdf · SB2b/SM4 - Deep Learning Deep Learning Infrastructure! Computational Infrastructure critical to deep

Deep Learning for Graphics - UCL · Course: “Deep Learning for Graphics” End-to-end: Loss • Old days • Evaluation came after • It was a bit optional: • You might still

Deep Convolutional Inverse Graphics Networkpapers.nips.cc/paper/5851-deep-convolutional-inverse-graphics-network.pdfDeep Convolutional Inverse Graphics Network Tejas D. Kulkarni*1,

A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning...A Brief Introduction to Deep Learning ... distributed representation, ... Antoine Bordes, and Yoshua

Deep Learning for Laser Based Odometry Estimation - …juxi.net/workshop/deep-learning-rss-2016/papers/Nicolai - Deep... · Deep Learning for Laser Based Odometry Estimation ... deep

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

The World’s First Blockchain-based, Deep Learning (AI ... · The World’s First Blockchain-based, Deep Learning ... Royalty Free Stock Graphics Platform & Marketplace ... in innovation

Deep Learning for Graphics Unsupervised Learning · EG Course “Deep Learning for Graphics” InfoGAN mple:generator:discriminator maximize mutual information varying Image Credit: