Introduce the field

“…a real convergence…” – “…computer vision finally becoming useful

Pikachu © Nintendo - http://www.freelargeimages.com/wp-content/uploads/2014/12/Pikachu_04.pngHoloLens © Microsoft - http://www.glassappsource.com/wp-content/uploads/2016/01/hololens.jpg

1

https://cdn1.vox-cdn.com/uploads/chorus_asset/file/6625539/pokemon-go-nick_statt-screenshots-1.0.jpg

Niantic – Ingress

Pokemon Go © Nintendo, NianticImage © Independent - https://static.independent.co.uk/s3fs-public/thumbnails/image/2016/07/08/15/pokemon-go-nick-statt-screenshots-1.0.jpghttp://www.independent.co.uk/arts-entertainment/pokemon-go-uk-release-date-and-how-to-get-it-now-on-iphone-and-android-apk-a7127326.html

Logo © Nintendo - http://assets.pokemon.com/assets/cms2/img/video-games/video-games/pokemon_go/169.jpg

2

Snapchat

Photo - http://thetechjournal.com/how-to/tutorial-face-swap-snapchat.xhtmlhttp://www.pleated-jeans.com/2016/03/02/21-snapchat-face-swaps-that-went-horribly-wrong/

3

Images © Microsofthttps://compass-ssl.surface.com/assets/13/b5/13b57736-9875-4332-af9f-dca0accbc331.jpg?n=HoloLens_Homepage_Audiences1_1920_1001.jpghttps://i.ytimg.com/vi/SKpKlh1-en0/maxresdefault.jpghttps://cdn0.vox-cdn.com/thumbor/aotdquRv2ZmMp1l-NTU_HrHcQtA=/cdn0.vox-cdn.com/uploads/chorus_asset/file/3327268/a67d3d33-e1e5-4cf7-bf3d-dbe1befc8d8c.0.jpg

Like magic! Where did that come from?!Even a company building a competitor – called MagicLeap – that has put out a few screen recordings.

4

5

6

Top: Bregler et al.Bottom – Beowolf production shots

7

Furukawa et al.

8

[Bokeloh et el. EUROGRAPHICS 2009]

9

[Ma et al. EGSR 2007]

10

Explain the diagram.

So what happens when the estimation of these models is not useful – when they are incorrect or imprecise or insufficient?

11

Question audience – what is wrong here?

Video - Playing Pokémon GO in New York - Tech Insiderhttps://www.youtube.com/watch?v=aQfkME59s9s

12

Image - http://www.pleated-jeans.com/2016/03/02/21-snapchat-face-swaps-that-went-horribly-wrong/

What went wrong here?

13

Image - http://www.pleated-jeans.com/2016/03/02/21-snapchat-face-swaps-that-went-horribly-wrong/

What went wrong here?

14

Image - http://blogs.solidworks.com/solidworksblog/wp-content/uploads/sites/2/6a00d83451706569e2017ee8115a91970d.jpg

Question audience – what is wrong here?

Next slide: but when you get it right, it can be transformative.

15

Magic Leap video - https://www.youtube.com/watch?v=kw0-JRa9n94

Where else are these models useful?

Half face shot is next.

16

https://www.wired.com/2013/03/luxion-keyshot/

Beyond augmented reality to VFX

Can anyone guess what is going on in this image? What artifact do you see?

17

Ex Machina – VFX Showreel – Double Negative -https://www.youtube.com/watch?v=qN5cjt2O6js

“A young programmer is selected to participate in a ground-breaking experiment in synthetic intelligence by evaluating the human qualities of a breath-taking humanoid A.I.”

Academy Award for best visual effects, 2016.

How did they fill in the bits _behind_ the newly transparent parts?

18

19

Image © Jason Corso - http://web.eecs.umich.edu/~jjcorso/r/career/index.html

20

21

OK, so before I give you a little introduction about myself, who do we have in the room?

Undergrads?Masters?PhDs?

And what’s your experience in visual computing?

Who has taken a graphics course?Vision course?Machine learning? Interaction?

22

I’ve been working in visual computing research now for 10 years, and in that time I’ve combined state-of-the-art computer vision techniques to create new interactivegraphics applications.

23

Well, he’s one example result.

The user provides start and end pair of images, and the system generates an

exploration of the video collection that takes in those views. This is computed

completely automatically, just from the visual information.

In this result, the system smoothly and automatically moves from looking at Big Ben to looking at the London Eye – in this composite motion made from 4 videos, the camera moves from one side of the river to the other and rotates 180 degrees to take in both views.

Tompkin et al. SIGGRAPH 2012 Videoscapes

24

Here’s what it looks like in motion.When we move through a video’s timeline, the video foci moves spatially in the context. We can see the comings and goings of the people in the scene, across different times – potentially across many months.In this way, by embedding the videos within a shared context, we can move away from the typical linear sequential presentation of videos, to build new interactions to compare and contrast spatio-temporal events.

FOCUS + CONTEXT -> great for YURT resolution demo

Tompkin et al. UIST 2013 Vidicontexts

25

Tompkin et al. UIST 2013 Vidicontexts

26

How might we paint in free space within a volume, with a hand-held pen, and have those paintings appear exactly where you placed them?

Tompkin et al. UIST 2015 5D light field pen sensing

27

Now, in my time as a masters and doctoral student, many things were hard for me.

28

Especially tough right at the beginning of your studies.

29

30

Especially tough right at the beginning of your studies.

31

32

One person is assigned at random at the beginning of the seminar to lead the discussion.Everybody leads discussion at least once in the course.The discussion leader receives a digest of the submitted questions just before the seminar.The discussion leader raises questions appropriately throughout the discussion, covers future work aspects, and finally provides a summary of the strengths and weaknesses of the techniques and of the discipline.

33

34

35

36

37

38

You spend a lot of time in high school and later on in college, under certain disciplines, learning to write.But learning to read efficiently when we move to reading academic texts is an important skill for your dissertations or thesis, and it’s not really taught as much.

39

40

PatchMatch+Global Patch Collider or TreeCANN

Image © Barnes et al., PatchMatch, SIGGRAPH 2009http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/patchmatch.pdf

41

Stereo diagram © MATLAB - https://www.mathworks.com/matlabcentral/mlc-downloads/downloads/submissions/41739/versions/1/screenshot.jpgModel column architecture – Goesele et al. – Multi-view Stereo Revisited -http://www.cs.washington.edu/node/2322http://www.gcc.tu-darmstadt.de/media/gcc/papers/Goesele-2006-MSR.pdf

42

https://stimulant.com/depth-sensor-shootout-2/

Learning to be a depth camerahttps://www.youtube.com/watch?v=n89EsXM69d4http://research.microsoft.com/en-us/projects/depth4free/default.aspx

Project TangoVenture Beat - http://1u88jj3r4db2x4txp44yqfj1.wpengine.netdna-cdn.com/wp-content/uploads/2015/08/Project-Tango-Google2-930x585.jpg

43

Dou et al., SIGGRAPH 2016 -http://perceptiveio.com/publications/2016/7/13/fusion4d-real-time-performance-capture-of-challenging-scenes

44

Left – Shotton et al., Real-Time Human Pose Recognition in Parts from a Single Depth Image - http://research.microsoft.com/apps/pubs/default.aspx?id=145347

Right – Gall et al., http://files.is.tue.mpg.de/jgall/projects/skelsurf/skelsurf.htmlMotion Capture Using Joint Skeleton Tracking and Surface Estimation

45

Debevec et al. – Rendering Synthetic Objects into Real Scenes (SIGGRAPH 1998)https://www.fxguide.com/wp-content/uploads/2012/05/relit_cglighting.jpg?8f1314

NOTE: This is _not_ a difficult illumination estimation paper, but it is one. Uses HDR images of environment maps to light the virtual world.We will read papers on more advanced problems.

46

Image © Song et al. - http://ait.inf.ethz.ch/projects/2015/3Dfrom2D/ - Joint Estimation of 3D Hand Position and Gestures from Monocular Video for Mobile Interaction

47

Video – WIRED - https://www.youtube.com/watch?v=BLkFWq_ipCc

Has anyone heard of Magic Leap?

$800 million seed round

WIRED

48

Image © Wikipedia - https://en.wikipedia.org/wiki/Mixed_reality

49

A great literature review

50

Video – Märki et al. - https://graphics.ethz.ch/~perazzif/bvs/index.html

51

Video – Märki et al. - https://graphics.ethz.ch/~perazzif/bvs/index.html

52

Images © MIT intrinsic image database http://www.cs.toronto.edu/~rgrosse/intrinsic/Roger Grosse, Micah K. Johnson, Edward H. Adelson, and William T. Freeman, Ground truth dataset and baseline evaluations for intrinsic image algorithms, in Proceedings of the International Conference on Computer Vision (ICCV), 2009.

53

Video © Laffont et al., Coherent Intrinsic Images from Photo Collections, http://www-sop.inria.fr/reves/Basilic/2012/LBPDD12/

54

Video © Face2Face: Real-time Face Capture and Reenactment of RGB Videos -http://people.mpi-inf.mpg.de/~mzollhoef/Papers/CVPR2016_FF/page.htmlJustus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias NießnerCVPR 2016

55

56

57

58

59

60

61