Computer Vision For Fun and Profit

Computer Vision for Fun and ProfitImage Processing: The Lowdown

!

!

Steven Mitchell, Ph.D. Componica, LLC

Copyright 2011 - Componica, LLC (http://www.componica.com/)

What does Componica have to offer?Decades of Experience and Expertise

A large library of code generated internally

A community of innovative Computer Engineers and Programmers

Access to academic papers and a history of effectively applying archived research

libSeal - A long term project to take previously written code and turn it into a single library:

SEAL - Steve's Evil (or Eclectic) Algorithms Library

http://www.componica.com


Image Processing

Any sort of signal processing done to an image

The Acquisition of an image

Compression and storage of an image

Enhancement and restoration

Registration of an image to another.

Measurement of data from the image (height of building, speed of car)

Interpretation and Recognition of objects.



Getting the Images in the First Place

Standard image files:

GIF, PNG, JPG: Standard. I tend to always use PNGs

TIFF, PDF: More challenging, requires an external library

Video files:

FFMpeg: Without it, it would be a nightmare to read video files but, if it can't read a video file, I can't read that video file.

Web Cam: OpenCV has a simple way of doing it. Outside OpenCV, there are scattered libraries

Mobile Devices

Scanner

MRI and CT scanners: Use a file format called DICOM



A Menu of Tools:Image Enhancement - Computer...uncrop and enhance!

Image Segmentation - These pixels belong to this, those pixels belong to that.

Image Registration - Line this image up with that image.

Object Recognition - This is a image of a frog and that's an image of a cheeseburger.

Image Compression - Crush this image and make sure the process is undo-able. (Not Covered...there are plenty of free libraries to do this.)



Image EnhancementSimple Pixel Stuff:

Normalizing brightness and contrast.

Gamma correction (the non-linear effects of TV)

Histogram equalization - maximize the global contrast.

Color Correction...color temperature and tint. White balancing.



Geometric Stuff:

Interpolation. Make the image bigger or smaller. This is not as easy as it sounds if you want non-aliased results. Trips up web developers whenever they try to roll their own thumbnail generator.

Warping an image from one geometry to another.

Simple rotation, scale, and translation. You need two or three landmark points.

Perspective (aka Projection or Homogeneous) transforms. You need four landmark points.

Lens distortions. Images pinch or barrel out in camera lens. You can calibrate and correct for that with enough landmark points.

General warping. Typically used for image morphing in special effects.

Image Enhancement - Geometry



Image Enhancement - Interpolation

AMATEUR

PRO

UPSAMPLING DOWNSAMPLINGORIGINALS



Image Enhancement - Geometry

ORIGINAL IMAGE PERSPECTIVE CORRECTEDThe original image is warped to a perspective corrected version.

The four black dots indicated the landmark points used to normalize the image to this artificial view.

The black regions are areas that falls outside the original image. Unknown data.



Noise removing. Median filter, average filter, etc.

De-blurring. You estimate the cause of the blurring and then undo it using deconvolution.

Motion blur - Estimate the camera movements and undo it.

Focus blur - Estimate the lens blur and undo it.

Image Enhancement

http://cg.postech.ac.kr/research/fast_motion_deblurring/


http://cg.postech.ac.kr/research/fast_motion_deblurring/


Divide an image into known parts:

It's not quite object recognition because images are typically interpreted on a pixel basis using edge detection or colors.

Sometimes it's good enough because you just care about the borders, not content

Border of tumor vs. healthy tissue.

The bright red ball in the color picture.

Is this pixel part of a letter ‘q’ or paper?

Often the first step to a bigger solution.

Image Segmentation



Make a decision based on a single pixel:

Simple thresholding - is this pixel darker than 160?

Slightly better - is this pixel red?

Even more better - is this pixel statistically more likely to be paper or letter based on it's RGB value.

The works - I'm modeling the uneven-ness of the lighting on this paper and made a statistical model of RGB, is this damn pixel ink or paper?

The downfall of this is you're looking at single pixels without understand how it's neighbors relate to it, missing the whole picture. Sometimes it works. Fast and easy to do.

Image Segmentation - Pixel Classes

ORIGINAL

SIMPLE THRESHOLD

ADAPTIVE BINARIZATION



Minimizing or maximizing a path which borders between elements. This one is a common technique, with many variants:

Assign a cost to all the pixels. For example, edge detection - it will cost me a lot to cross an edge. Or color transition, it will cost me to cross to a different color.

Use an optimization technique from classic data structures (typically used in graph theory if you still remember) to compute the cheapest path from one location of the image to a different location.

Dynamic Programming - Strange name but all it means is compute the cheapest path from one side of an image to another. Works best for paths that tend to be linear.

Minimum Graph Cut - Find the cheapest way to split the image in two regions. This works well for circular paths and 3D. Tends to be much more complicated than dynamic programming and slower.

Check out this: http://www.youtube.com/watch?v=6NcIJXTlugc

Image Segmentation - Least Cost Path


http://www.youtube.com/watch?v=6NcIJXTlugc


Random Decision Forests applied to pixels Apply twenty questions to pixel and it's surrounding neighbors to create decision trees to guess what this pixel is over.

Have a number of these decision trees (forest) and aggregate the results. Strangely it tends to works in many cases.

Image Segmentation - Bleeding Edge



I have a source image that I can transform (move, resize, rotate, or bend). Make it best fit a target image.

Translation Only - Shift the source image until it best fits the target.

Similarity Transform - Adjust the scale, rotation, and translation until the source image overlaps the target.

Perspective Transform - Move the source image’s four corners until it matches the target in a perspective manner.

Non-Rigid Warping - The source image is on a rubber sheet. Warp it onto the target image.

Obvious Applications:

Augmented Reality

Image Stitching

Object Detection

Image Registration



The most common way to register as image is the following:

Find the most interesting points on the two images (usually blobs and corners).

Scale-invariant feature transform - SIFT (Patented and slow)

Speeded Up Robust Features - SURF (Recently Patented and fast)

Features from Accelerated Segment Test - FAST (Very fast but noisy)

With SURF and SIFT you get a position, an angle, scale factor and a 64 element vector to compare. With FAST you get a position and maybe a rotation.

Compare all the interesting points from one image to the other forming matching pairs of points between images. Naive implementations are SLOW.

Use RANSAC to find a consensus (next page).

Image Registration - Interesting Points

SURF

FAST



Matching points locally doesn’t yield global matches.

RANdom Sample Consensus - RANSAC - prunes away the mismatches and computes the transform that converts the source image to the target.

It works as followed:

Most transforms can be described by a minimal number of points. For similarity transforms it’s two points, for projection transforms it’s four points.

Pick two (or four) matched pair of points at random.

Compute a transform from those two (or four) sets of points.

Transform all the source points using this transform and see how many points are close to the target.

Repeat this and keep the best transform that matches the most points.

Image Registration - Interesting Points

Here the green lines indicate pairs of matched points that fit the transform (looks like a similarity transform) and the red lines are matched pairs that failed to fit this transform and therefore rejected.!!RANSAC may seem ad hoc, but it works surprisingly well.



A very accurate way to line up a template onto an image is to use derivates and linear algebra.

Works very well only when the images are very close to each other in the first place.

It’s usually the polishing step after using an Interesting Points / RANSAC method.

Known as Lucas-Kanade Tracker -or- Baker-Matthews Tracker.

It’s the secret sauce to how the “Predator” algorithm works.

Image Registration - The Mathy Way

http://www.wedesoft.demon.co.uk/lucas-kanade-tracker.html


http://www.wedesoft.demon.co.uk/lucas-kanade-tracker.html


Making computers recognize objects in images

Many different algorithms, and each algorithm has appropriate uses depending on the objects being detected. For example:

AdaBoost - Awesome at detecting and locating faces, sucks at recognizing who’s face it is.

Deep Neural Networks - Works great for highly distorted text, but too slow for generalized OCR.

Active Appearance Models - Works great for both recognition and segmentation, but only on generally fixed shapes like faces and hearts. Sucks for livers and cancer.

Most recognizers depend on training the algorithm on known objects offline and then testing. Which brings up the topic of data...

Object Recognition



The data can be the most valuable part of a trainable system as many algorithms will generally function with somewhat similar hit rates.

Often ideas fail to take into account where the data comes from. It’s the killer of many ideas.

The basics of training something:

Data is normally split into a training and testing set. You train a thingy with the training set and test how well it works with the testing set.

Why? Most trainable thingies are prone to overfitting. Splitting the data into two sets prevents this problem because you use the test set to know when to stop training.

Disadvantage, you effectively need twice as much data. Sucks.

Object Recognition - It’s all about the Data

It’s obvious this data is best represented as a line, but!if the model over-fits the data, it may compute a!relationship that’s nonsensical.

As your algorithm learns from a training set (blue line),!the error decreases for that set, but in the testing set,!it will hit a point where overfitting is happening and!will increase the overall error in the real world. You stop!training at the point the testing set starts getting worse.



Steve’s crappy breakdown of object recognition algorithms:

A ship of fools approach - Armies of stupid algorithms that together become smart. Kinda like democracy sort of...not really.

Let’s create a brain, Igor - So what happens if you simulate brain tissue? It’s grown quite a bit since the neural network hype in the late 80s / early 90s. Fun Fact: This will eventually kill us all in a bloody uprising.

If the shoe fits... - Well if this template StarBucks logo fits somewhere on my image using object registration as described above, then I’m guessing this StarBucks logo is present in the image. Duh.

Find Features to Filter and Fit in a Feed-forward Fashion - You see this pattern all the time and it lacks of creativity. Find interesting features in the image, and feed them into a trainable function like a neural network, a non-linear regression, or a support vector machine. Boring.

I’ve stared at a wall for 20 minutes now and I think everything out there is either one or a combination of these general ideas. Hmm...That’s it? Disappointed.

Object Recognition - The Menu


End of Part One...

Technology

Computer Vision For Fun and Profit