Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... ·...

VisualDynamics:ProbabilisticFutureFrameSynthesisviaCrossConvolutionalNetworks

TianfanXue* JiajunWu* KatieBouman BillFreeman

NIPS2016

VGGReadingGroup,24Feb2017AnkushGupta

Frame2

Task:futureframeprediction

Frame1 Frame2Deterministicneuralnetwork

Deterministicpredictionsfailtomodeluncertainty

Frame1 Deterministicneuralnetwork

Prediction

Whatistheproblem?

Frame1 Deterministicneuralnetwork

Prediction

Whatistheproblem?

SynthesisnetworkInputframe Sampledfutureframe

Sampledifferentfutureframes

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Inputrandommotionvector𝑧~𝑝$(𝑧)

SynthesisnetworkInputframe

Sampledifferentfutureframes

Sampledfutureframe

Inputframe Anothersampledfutureframe

Segments Transformedsegments

Synthesizeusingdifferenttransformations

Sampledfutureframe

Motionvector𝑧

SynthesisnetworkInputframe

Encodingnetwork

Futureframe(groundtruth)

Training

Motionvector𝑧

Encodingnetwork

Synthesisnetwork Futureframe

(prediction)Trainingsamples

(Label-free)

Training

Inputframe

Futureframe(groundtruth)

Futureframe𝐼()*(prediction)

Motionvector𝑧

Encodingnetwork

Synthesisnetwork

Training

Futureframe𝐼+,(groundtruth)

Inputframe

Objectivefunction:𝐼()* − 𝐼+, + 𝐷01(𝒛||𝑁(𝟎, 𝐈))

Reconstructionloss

Inputframe

Encodingnetwork

Synthesisnetwork

Training Objectivefunction:𝐼()* − 𝐼+, + 𝐷01(𝒛||𝑁(𝟎, 𝐈))

KL-divergenceloss

Motionvector𝑧

Variational Autoencoder[Kingma andWelling,2014]

Synthesisnetwork

Testing

Encodingnetwork

Inputframe

Realoutputfromournetwork

Inputframe Futureframe

TransformsegmentsFindsegments

Inputrandommotionvector𝑧

Synthesizebytransformingsegments

Imagesegments Convolution

Movementcanbesynthesizedthroughconvolution

Imagesegments

Applyingmotiontoeachsegment

Motionkernels

Thedecodingnetworkgeneratesamotionkernelforeachcorrespondingsegment

Decodingnet

Motionvector𝑧

[Brabandere etal.2016][Finnetal.2016]

Motionvector𝑧

Inputframe

Futureframe

Synthesisnetwork

Futureframe

Whatisencodedinthemotionvector?

Encodingnetwork

Motionvector𝑧 Upwardmotionwhenchangingthisdimension

Eachdimensionencodesatypeofmotion

Motionvector𝑧 Legmotionwhenchangingthisdimension

Eachdimensionencodesatypeofmotion

• Simulatedshapes

• Trainingsamples

Results:toyexample

Learnedsegments

Networkautomaticallydetectssegments

Triangles

Circles

Input SamplednextframeGroundtruthdistribution

Sampledistribution

Networklearnsthecorrelationbetweenappearanceandmotion

Input Sampledfutureframes

Results:real-worldimages

Mainidea NetworkstructureOutline Whatnetworklearns Result

Challenge:largemotion

Input TwosampledfutureframesArtifactsappearwhenmotionislarge

Baseline:Transferflow 25.5%Ourmethod 31.3%

Labeledasreal

MechanicalTurkstudytoassesssynthesisquality

Idealsynthesisalgorithmachieves50%

• Samplemultiplefutureframesthatareconsistentwiththeinput

• Synthesizeframesbytransformingsegments

• Learnamotionrepresentationwithoutsupervision

Contributions

Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... ·...

Documents

Applying Combinatory Logic Synthesis To Work With Existing … · 2017-10-17 · Combinatory Logic Synthesis (CLS) then is our solution to the large frame-work problem. Application

PROBABILISTIC ANALYSIS OF REINFORCED CONCRETE ...ethesis.nitrkl.ac.in/6887/1/Probabilistic_Prusty_2015.pdfnonlinear analysis of RC plane frame structures including both structural

Automated Verification and Strategy Synthesis for ...qav.comlab.ox.ac.uk/talks/marta-atva13inv.pdf · Automated Verification and Strategy Synthesis for Probabilistic Systems Marta

Strategy Synthesis for Autonomous Agents using PRISMgethin/papers/nfm18.pdfThe probabilistic model checker PRISM. PRISM [19] is a probabilistic model checker that allows for the analysis

SRL Approaches: Frame-based Probabilistic models February 11, 2005

Video Frame Synthesis Using Deep Voxel Flow...Video Frame Synthesis using Deep Voxel Flow Ziwei Liu1 Raymond A. Yeh2 Xiaoou Tang1 Yiming Liu3∗ Aseem Agarwala4 1The Chinese University

Probabilistic Circuits: A New Synthesis of Logic and Machine Learningguyvdb/slides/ArrowheadCon... · 2019-05-26 · A New Synthesis of Logic and Machine Learning Guy Van den Broeck

ImprovedReproductionofStopsinNoiseReductionSystemswith ......based analysis-synthesis system using overlap-add for signal synthesis, the delay is given by the frame length of the synthesis

Multivariate Probabilistic Synthesis of Cellular Networks

Probabilistic Circuits: A New Synthesis of Logic and Machine Learningweb.cs.ucla.edu/~guyvdb/slides/KULeuven18.pdf · Probabilistic and Logistic Circuits: A New Synthesis of Logic

A frame w ork for the design, synthesis and cycle-accurate ...akjones/Alex-K-Jones/Enabling... · A frame w ork for the design, synthesis and cycle-accurate simulation of multiprocessor

Probabilistic Seismic Vulnerability assessment of RC Frame

Foundations of Probabilistic Programming/Program SynthesisOverview Foundations of Probabilistic Programming Probabilistic Programs Probabilistic programs extend sequential programs

LTL Synthesis on Probabilistic Systems

A Probabilistic Model for Component-Based Shape Synthesis

Probabilistic Reasoning - Department of Computer Science ... · Probabilistic Description Logics Probabilistic Datalog+/– Probabilistic Ontological Data Exchange Description Logics:

probabilistic logics and the synthesis of reliable organisms from

Joint registration and synthesis using a probabilistic

Minimization of the Probabilistic p-frame Potential...Minimization of the Probabilistic p-frame Potential M. Ehlera,b,, K. A. Okoudjoub aNational Institutes of Health, National Institute

PROBABILISTIC RELIABILITY ENGINEERING - …gnedenko-forum.org/library/Ushakov/Probabilistic Reliability...Probabilistic Reliability Engineering, Boris Gnedenko and Igor Ushakov. PROBABILISTIC