CS 2770: Computer Vision

Introduction

Prof. Adriana KovashkaUniversity of Pittsburgh

January 5, 2017

About the Instructor

Born 1985 in Sofia, Bulgaria

Got BA in 2008 atPomona College, CA(Computer Science & Media Studies)

Got PhD in 2014at University of Texas at Austin(Computer Vision)

Course Info

• Course website: http://people.cs.pitt.edu/~kovashka/cs2770

• Instructor: Adriana Kovashka(kovashka@cs.pitt.edu)

– Use "CS2770" at the beginning of your Subject

• Office: Sennott Square 5325

• Office hours: Tue/Thu, 3:30pm - 5:30pm

TA

• Keren Ye (yekeren@cs.pitt.edu)

• Office: Sennott Square 5501

• Office hours: TBD

– Do the Doodle by the end of Friday:

http://doodle.com/poll/v3m8acmcdsiydqhq

Textbooks

• Computer Vision: Algorithms and Applicationsby Richard Szeliski

• Visual Object Recognition by Kristen Graumanand Bastian Leibe

• More resources available on course webpage

• Your notes from class are your best study material, slides are not complete with notes

Course Goals

• To learn about the basic computer vision tasks and approaches

• To get experience with some computer vision techniques

• To learn/apply basic machine learning (a key component of modern computer vision)

• To think critically about vision approaches, and to see connections between works and potential for improvement

Policies and Schedule

http://people.cs.pitt.edu/~kovashka/cs2770/

Should I take this class?

• It will be a lot of work!

– But you will learn a lot

• Some parts will be hard and require that you pay close attention!

– But I will have periodic ungraded pop quizzes to see how you’re doing

– I will also pick on students randomly to answer questions

– Use instructor’s and TA’s office hours!!!

Questions?

Plan for Today

• Introductions

• What is computer vision?

– Why do we care?

– What are the challenges?

– What is the current research like?

• Overview of topics (if time)

Introductions

• What is your name?

• What one thing outside of school are you passionate about?

• Do you have any prior experience with computer vision?

• What do you hope to get out of this class?

• Every time you speak, please remind me your name

Computer Vision

What is computer vision?

Done?

Kristen Grauman (adapted)

"We see with our brains, not with our eyes“ (Oliver Sacks and others)

• Automatic understanding of images and video

– Computing properties of the 3D world from visual

data (measurement)

– Algorithms and representations to allow a

machine to recognize objects, people, scenes, and

activities (perception and interpretation)

– Algorithms to mine, search, and interact with

visual data (search and organization)

Kristen Grauman

What is computer vision?

Vision for measurement

Real-time stereo Structure from motion

NASA Mars Rover

Pollefeys et al.

Multi-view stereo for

community photo collections

Goesele et al.

Slide credit: L. LazebnikKristen Grauman

sky

water

Ferris

wheel

amusement park

Cedar Point

12 E

tree

tree

tree

carouseldeck

people waiting in line

ride

ride

ride

umbrellas

pedestrians

maxair

bench

tree

Lake Erie

people sitting on ride

Objects

Activities

Scenes

Locations

Text / writing

Faces

Gestures

Motions

Emotions…

The Wicked

Twister

Vision for perception, interpretation

Kristen Grauman

Visual search, organization

Image or video

archives

Query Relevant

content

Kristen Grauman

Related disciplines

Cognitive

science

Algorithms

Image

processing

Artificial

intelligence

GraphicsMachine

learningComputer

vision

Kristen Grauman

Vision and graphics

ModelImages Vision

Graphics

Inverse problems: analysis and synthesis.

Kristen Grauman

Why vision?• Images and video are everywhere!

Personal photo albums

Surveillance and security

Movies, news, sports

Medical and scientific images

Adapted from Lana Lazebnik

144k hours uploaded to YouTube daily

4.5 mil photos uploaded to Flickr daily

10 bil images indexed by Google

• As image sources multiply, so do applications

– Relieve humans of boring, easy tasks

– Human-computer interaction

– Perception for robotics / autonomous agents

– Organize and give access to visual content

– Description of image content for the visually

impaired

– Fun applications (e.g. transfer art styles to my

photos)

Adapted from Kristen Grauman

Why vision?

Faces and digital cameras

Setting camera

focus via face

detection

Camera waits for

everyone to smile to

take a photo [Canon]

Kristen Grauman

Face recognition

Devi Parikh

Linking to info with a mobile device

kooaba

Situated search

Yeh et al., MIT

MSR Lincoln

Kristen Grauman

Snavely et al.

Kristen Grauman

Exploring photo collections

Special visual effects

The Matrix What Dreams May Come

Mocap for Pirates of the Carribean, Industrial Light and Magic

Source: S. Seitz

Kristen Grauman

Yong Jae Lee

Interactive systems

Video-based interfaces

Human joystick

NewsBreaker Live

Assistive technology systems

Camera Mouse

Boston College

Kristen Grauman

YouTube Link

Vision for medical & neuroimages

Image guided surgery

MIT AI Vision Group

fMRI data

Golland et al.

Kristen Grauman

Safety & security

Navigation,

driver safety Monitoring pool (Poseidon)

SurveillancePedestrian detection

MERL, Viola et al.Kristen Grauman

Healthy eating

Im2calories by Myers et al., ICCV 2015

figure source

FarmBot.io

YouTube Link

Pirsiavash et al., “Assessing the Quality of Actions”, ECCV 2014

Self-training for sports?

Image generation

Radford et al., ICLR 2016

Reed et al., ICML 2016

YouTube link

Seeing AI

Obstacles?

Kristen GraumanRead more about the history: Szeliski Sec. 1.2

What the computer gets

Adapted from Kristen Grauman and Lana Lazebnik

Why is this problematic?

Why is vision difficult?

• Ill-posed problem: real world much more

complex than what we can measure in

images

– 3D 2D

• Impossible to literally “invert” image formation

process with limited information

• Need information outside of this particular

image to generalize what image portrays (e.g.

to resolve occlusion)

Adapted from Kristen Grauman

Challenges: many nuisance parameters

Illumination Object pose Clutter

ViewpointIntra-class

appearanceOcclusions

Kristen Grauman

Think again about the pixels…

Challenges: intra-class variation

slide credit: Fei-Fei, Fergus & Torralba

CMOA Pittsburgh

Challenges: importance of context

slide credit: Fei-Fei, Fergus & Torralba

• Thousands to millions of pixels in an image

• 3,000-30,000 human recognizable object categories

• 30+ degrees of freedom in the pose of articulated

objects (humans)

• Billions of images indexed by Google Image Search

• 1.424 billion smart camera phones sold in 2015

• About half of the cerebral cortex in primates is

devoted to processing visual information [Felleman

and van Essen 1991]

Kristen Grauman

Challenges: Complexity

Challenges: Limited supervision

MoreLess

Kristen Grauman

Challenges: Vision requires reasoning

Antol et al., “VQA: Visual Question Answering”, ICCV 2015

• Ok, clearly the vision problem is deep and

challenging… time to give up?

• Active research area with exciting progress!

• How datasets changed:

…… …

… ……

… …

…

Kristen Grauman

Datasets today

ImageNet: 22k categories, 14mil images

Microsoft COCO: 80 categories, 300k images

PASCAL: 20 categories, 12k images

SUN: 5k categories, 130k images

Some Visual Recognition Problems

Recognition: What is this?

Recognition: What objects do you see?

carriagehorse

person

person

truck

street

building

table

balcony

car

Detection: Where are the cars?

Activity: What is this person doing?

Scene: Is this an indoor scene?

Instance: Which city? Which building?

Visual question answering: What are all these people participating in?

The Latest at CVPR 2016

* CVPR = IEEE Conference on

Computer Vision and Pattern

Recognition

Our ability to detect objects has gone

from 34 mAP in 2008

to 73 mAP at 7 FPS (frames per second)

or 63 mAP at 45 FPS

in 2016

Redmon et al., CVPR 2016

You Only Look Once:

Unified, Real-Time Object Detection

Force from Motion:

Decoding Physical Sensation from a First Person Video

Park et al., CVPR 2016

MovieQA: Understanding Stories in Movies through

Question-Answering

Tapaswi et al., CVPR 2016

Visually Indicated Sounds

Owens et al., CVPR 2016

Anticipating Visual Representations from Unlabeled Video

Vondrick et al., CVPR 2016

Gatys et al., CVPR 2016

Image Style Transfer Using

Convolutional Neural Networks

DeepArt.io – try it for yourself!(Image Style Transfer Using Convolutional Neural Networks)

Images:

Styles:

DeepArt.io – try it for yourself!(Image Style Transfer Using Convolutional Neural Networks)

Results:

Seeing Behind the Camera:

Identifying the Authorship of a Photograph

Thomas and Kovashka, CVPR 2016

Is computer vision solved?

• Given an image, we can guess with 81% accuracy what object categories are shown (ResNet)

• … but we only answer “why” questions about images with 14% accuracy!

Why does it seem that it’s solved?

• Deep learning makes excellent use of massive data (labeled for the task of interest?)– But it’s hard to understand how it does so

– It doesn’t work well when massive data is not available and your task is different than tasks for which data is available

• Sometimes the manner in which deep methods work is not intellectually appealing, but our “smarter” / more complex methods perform worse

Overview of Topics

Overview of topics

• Lower-level vision– Analyzing textures, edges and gradients in images,

without concern for the semantics (e.g. objects) of the image

• Higher-level vision– Making predictions about the semantics or higher-

level functions of content in images (e.g. objects, attributes, styles, motion, etc.)

– Involves machine learning; we’ll cover some basics of this then go back to low-level tasks

Features and filters

• Transforming and

describing images;

textures, colors, edgesKristen Grauman

• Detecting distinctive + repeatable features

• Describing images with local statistics

Features and filters

Indexing and search

Kristen Grauman

• Matching features and

regions across images

• How does light in

3d world project

to form 2d

images?

Kristen Grauman

Image formation

Hartley and Zisserman

Lowe

• Multi-view geometry,

matching, invariant

features, stereo vision

Fei-Fei Li

Kristen Grauman

Multiple views

[fig from Shi et al]

• Clustering,

segmentation,

fitting; what parts

belong together?Kristen Grauman

Grouping and fitting

• Recognizing objects

and categories,

learning techniques

Kristen Grauman

Visual recognition

• Detecting novel instances of objects

• Classifying regions as one of several categories

Object detection

• Describing the high-level properties of objects

• Allows recognition of unseen objects

Attribute-based description

Convolutional neural networks

• State-of-the-art on many recognition tasks

ImagePrediction

Yosinski et al., ICML DL workshop 2015

Krizhevsky et al.

Recurrent neural networks

• Sequence processing, e.g. question answering

Wu et al., CVPR 2016

• Tracking objects, video analysis

Tomas Izo

Kristen Grauman

Motion and tracking

Pose and actions

• Automatically annotating human pose (joints)

• Recognizing actions in first-person video

Your Homework

• Fill out Doodle

• Read entire course website

• Do first reading

Next Time

• Linear algebra review

• Matlab tutorial