OpenCV – introductionMašinska vizija, 2017.

OpenCV -History

• OpenCV (Open Source Computer Vision) is a library of programming functions mainly aimed at real-time computer vision.

• Intel team from Nizhny Novgorod created the library• Vadim Pisarevsky – the largest single contributor to the library• Gary Bradski – in 1999. launched OpenCV with the hopes of accelerating computer vision and artificial

intelligence by providing a solid infrastructure for everyone working in the field.

• OpenCV has received much of its support over the years from Intel and Google, but especially from Itseez (recently acquired by Intel), which did the bulk of the early development work.

• Intel also donated the built-in performance primitives code, which provides for seamless speedup on Intel architectures.

• open-source BSD liscence• imposes minimal restrictions on the redistribution of covered software• allows Linking, Distribution, Modification, Private use and Subliscencing• requires that the code under BSD must be licensed under BSD if redistributed in source code format

Nizhny Novgorod

Why OpenCV?

• OpenCV was designed for computational efficiency and with a strong focus on real-time applications. It is written in optimized C++ and can take advantage of multicore processors. => IT IS FAST

• The library is written in C and C++ and runs under Linux, Windows, and Mac OS X. There is active development on interfaces for Python, Java, MATLAB, and other languages, including porting the library to Android and iOS for mobile applications. => GOOD PORTABILITY

• One of OpenCV’s goals is to provide a simple-to-use computer vision infrastructure that helps people build fairly sophisticated vision applications quickly. The OpenCV library contains over 500 functions that span many areas in vision, including factory product inspection, medical imaging, security, user interface, camera calibration, stereo vision, and robotics. => DIVERSE USAGE and provides good starting points in each area

• Functions are well documented, but they lack in example diversity.

• OpenCV is not great if you just want to manipulate images.

• OpenCV is great for teaching the computer how to see something!

According to one well-known story, in 1966, Marvin Minsky at MIT asked his undergraduate student Gerald Jay Sussman to “spend the summer linking a camera to a computer and getting the computer to describe what it saw” – That is how computer/machine vision started!

Learning OpenCV x.x – A.Kaehler, G. Bradski

• The intent of this book is tutorial

• Supplemental material (code examples, exercises, etc.) is available for downloadat https://github.com/oreillymedia/Learning-OpenCV-3_examples.

• The purpose of this book is to:• Comprehensively document OpenCV by detailing what function calling conventions really

mean and how to use them correctly• Give the reader an intuitive understanding of how the vision algorithms work• Give the reader some sense of what algorithm to use and when to use it• Give the reader a boost in implementing computer vision and machine learning algorithms by

providing many working code examples to start from• Suggest ways to fix some of the more advanced routines when something goes wrong

OpenCV Block Diagram

• OpenCV is built in layers. At the top is the OS under which OpenCV operates.

• Next comes the language bindings and sample applications.

• Below that is the contributed code in opencv_contrib, which contains mostly higher-level functionality.

• After that is the core of OpenCV, and at the bottom are the various hardware optimizations in the hardware acceleration layer (HAL).

OpenCV contrib

• Face Recognition

• Object Tracking

• DNN – Deep Neural Networks – Using Caffe framework models

• SFM – Structure From Motion

• Structured Light

• Multi-camera Calibration

• Stereo Matching

• ...

Installing OpenCV 3.2

• Using prebuilt OpenCV binaries [Win]• tutorial from docs.opencv.org - it is made for v2.4 but works for all newer• In order to use them in Visual Studio include files and libraries need to be included/linked• libraries “as is”, if we want some additional function we need to rebuild sources

• NuGet• nuget.org• package manager for the Microsoft development platform• works only for VS14 and older• very simple and fast OpenCV installation, but not flexible • recommendation: install opencv3.1 package

• Building OpenCV from source• same tutorial from above• rather complicated on Windows, linux provides much easier interface• download OCV source code -> install Cmake• install additional libraries and download opencv-contrib• use cmake to configure and generate build, then VS to compile

Installing OpenCV

• Using prebuilt OpenCV binaries [Win]• Sourceforge page -> 3.2.0 -> -vc14.exe

• run the .exe, select path for extraction (suggested E:/MS1MV/opencv)

• set the environment variable: OPENCV_DIRE:\MS1MV\opencv\build\x64\vc14

• add bin folder to system path: %OPENCV_DIR%\bin

• lib – static libraries

• dll – dynamic link libraries – loaded only on demand during runtime

Building applications with OCV inside the Microsoft Visual Studio• To build an application with OpenCV you need to do two things:

• Tell to the compiler how the OpenCV library looks. You do this by showing it the header files.

• Tell to the linker from where to get the functions or data structures of OpenCV, when they are needed.

• VS -> New Project - > Win32 Console Application -> give name -> OK -> Next -> Empty Project -> Finish

• View -> Other Windows -> Property Manager

• open Property Manager view and select Debug|x64 -> Add New Project Property Sheet

Building applications with OCV inside the Microsoft Visual Studio• created Property Sheet -> Properties

• OpenCV include directory (*.h and *.hpp files)• C++ -> General -> Additional Include Directories =

$(OPENCV_DIR)\..\..\include (make sure to press Enter and Apply)

• Libraries – we want to link them• static (*.lib)

• Linker -> General -> Additional Library Directories -> $(OPENCV_DIR)\lib• Linker -> Input -> Additional Dependencies -> we need to write down all debug libraries

(when creating release property page, then release libraries).

• dynamic (*.dll)• program will look for the .dll’s during runtime and if we have correctly added their path

to path environment variable it should successfully find them

Run some code!

• Create new .cpp file in Source Files

• Copy the code

#include <opencv2/opencv.hpp> //Include file for every supported OpenCV function

int main(int argc, char** argv) {

cv::Mat img = cv::imread(argv[1], -1); // Image name is expected as command line argument

if (img.empty()) return -1; // always check if the image was successfully read!!!

// create display window

cv::namedWindow("Image", cv::WINDOW_AUTOSIZE);

cv::imshow("Image", img); // show image; the window is selected through is unique name

cv::waitKey(0); // to keep the window showing we use waitKey(ms delay), 0 is forever

cv::destroyWindow("Example1"); // after any button is pressed close the window and finish

return 0;

}

Documentation

• http://docs.opencv.org/master/

• tutorials• introduction: installation and hello world• core module: matrix manipulation, file I/O, random generator…• imgproc: smoothing, sharpening, edge detection, common image processing• highgui: manipulate windows displays and add trackbars and other UI• …

• cheatsheet (OCV 2.4!) http://docs.opencv.org/3.0-last-rst/opencv_cheatsheet.pdf

• High level overview of the library

• OpenCV 3.x Modules ppt

First program – Display a picture - DONE

• OpenCV provides utilities for reading from a wide array of image file types, as well as from video and cameras. These utilities are part of a toolkit called HighGUI, which is included in the OpenCV package. We used some of these utilities to create a simple program that opens an image and displays it on the screen.

• Display grayscale version of the image – cv::cvtColor (convert color)• cvtColor is not a part of highgui -> look for appropriate module and include it

• which color systems are supported?

• in OpenCV input images are in BGR color space!!!

Simple transformation

• Perform image blurring with Gaussian filter 5x5, sigma = 5.

• cv::GaussianBlur(cv::InputArray src, cv::OutputArray dst, Size ksize, double sigmaX, double sigmaY=0.0, int borderType = cv::BORDER_DEFAULT)

• Sobel edge detection on a grayscale image -> Save image (imwrite)• where are the edges?

• what’s the data type?

• how does imshow() work?

Input from a Camera -> cv::VideoCapture

• The first thing we need is the cv::VideoCapture object. This object contains the information needed for reading frames from a camera or video file.

• Depending on the source, we use one of three different calls to create a cv::VideoCapture object: • cv::VideoCapture::VideoCapture( const string& filename); // video filename• cv::VideoCapture::VideoCapture(int device); // camera (device) ID• cv::VideoCapture::VideoCapture(); // default constructor

• If the open is successful and we are able to start reading frames, the member function cv::VideoCapture::isOpened() will return true.

• A lot of people don’t always check these sorts of things, assuming that nothing will go wrong. Don’t do that here. The returned value of cv::VideoCapture::isOpened() will be false if for some reason the file could not be opened (e.g., if the file does not exist), but that is not the only possible cause. The constructed object will also not be ready to be used if the codec with which the video is compressed is not known. Because of the many issues surrounding codecs (legal as well as technical), this is not as rare of an occurrence as one might hope.

Reading the camera

• cv::VideoCapture::read(cv::OutputArray image);• cv::VideoCapture::operator>>(cv::Mat& image);• cv::VidedoCapture::grab(void) -> copies raw data from camera quickly to

computer, and then when we want to use it, we must decode it• cv::VideoCapture::retrieve(cv::OutputArray image, int channel=0)• read() preforms grabbing and retrieving together

• Where is grab-retrieve useful?• The most common situation arises when there are multiple cameras (e.g., with

stereo imaging). In this case, it is important to have frames that are separated in time by the minimum amount possible (ideally they would be simultaneous for stereo imaging). Therefore, it makes the most sense to first grab all the frames and then come back and decode them after you have them all safely in your buffers.

Video metadata

• cv::VideoCapture::get() and cv::VideoCapture::set()

• Video files contain not only the video frames themselves, but also important metadata, which can be essential for handling the files correctly. When a video file is opened, that information is copied into the cv::VideoCapture object’s internal data area. It is very common to want to read that information from the cv::VideoCapture object, and sometimes also useful to write to that data area ourselves.

• Some metadata:• video duration• frame size• bit rate mode• color cpace• chroma subsampling• …

Input from a Camera

• Open file read_camera.cpp and copy the code (make sure that you don’t have two main functions)

• Function expects camera filename as input, so make sure you have no command line arguments

• Run the code, see what happens!

• Instead of simple streaming, show edges! • we tried Sobel for image, now try Canny

Getting to know OpenCV data types

• OpenCV has many data types, which are designed to make the representation and handling of important computer vision concepts relatively easy and intuitive.

• At the same time, many algorithm developers require a set of relatively powerful primitives that can be generalized or extended for their particular needs.

• This library attempts to address both of these needs through the use of templates for fundamental data types, and specializations of those templates that make everyday operations easier.

OpenCV data types

1. Basic data types - are those that are assembled directly from C++ primitives (int, float, etc.). These types include simple vectors and matrices, as well as representations of simple geometric concepts like points, rectangles, sizes, and the like. cv::Matx

2. Helper objects - These objects represent more abstract concepts such as the range objects used for slicing, e.g. cv::Range

3. Large array types - These are objects whose fundamental purpose is to contain arrays or other assemblies of primitives or, more often, the basic data types. The star example of this category is the cv::Mat class, which is used to represent arbitrary-dimensional arrays containing arbitrary basic elements.

In addition to these types, OpenCV also makes heavy use of the Standard Template Library (STL). OpenCV particularly relies on the vector class, and many OpenCV library functions now have vector template objects in their argument lists.

cv::Matx – fixed matrix class

• matrix dimension known at compile time

• all memory for their data is allocated on the stack, which means that they allocate and clean up quickly

• Operations on them are fast, and there are specially optimizedimplementations for small matrices (2 × 2, 3 × 3, etc.)

cv::Vec<> - also fixed

• Used through aliases (typedefs) for common instantiations of the cv::Vec<> template.

• They have names like cv::Vec2i, cv::Vec3i, and cv::Vec4d (for a two-element integer vector, a three element integer vector, or a four-element double-precision floating-point vector, respectively).

• In general, anything of the form cv::Vec{2,3,4,6}{b,w,s,i,f,d} is valid for any combination of two to four dimensions and the six data types.

• Can be accessed by a vector index -> myvec[0], myvec[1],…• In the proper sense of C++ inheritance, it is correct to say that the fixed

vector template cv::Vec<> is a cv::Matx<> whose number of columns is one.

cv::Point<> - also fixed

• The point classes, which are containers for two or threevalues of one of the primitive types.

• cv::Point2i, cv::Point2f

• members are accessed by named variables -> mypoint.x, mypoint.y

• Similar to cv::Point<> is cv::Size which allows storing 2D object size and also has two members: width and height.

• Mostly used for defining mask/matrix size

cv::Range – helper object

• two elements: start and end

• Ranges are inclusive of their start value, but not inclusive of their end value

• cv::Range(int start, int end) -> [start, end)

cv::InputArray and cv::OutputArray

• Many OpenCV functions take arrays as arguments and return arrays as return values, but in OpenCV, there are many kinds of arrays. We have already seen that OpenCV supports• small array types

(cv::Scalar, cv::Vec, cv::Matx) and STL’s std::vector<> • large array types (cv::Mat and cv::SparseMat).

• In order to keep the interface from becoming onerously complicated (and repetitive), OpenCV defines the types cv::InputArray and cv::OutputArray.

• In effect, these types mean “any of the above” with respect to the many array forms supported by the library.

• The primary difference between cv::InputArray and cv::OutputArray is that the former is assumed to be const (i.e., read only).

cv::Mat

• Could be considered the epicenter of the entire C++ implementation of the OpenCV library.

• The overwhelming majority of functions in the OpenCV library are members of the cv::Mat class, take a cv::Mat as an argument, or return cv::Mat as a return value; quite a few are or do all three.

• Objects such as images are specialized uses of the cv::Mat class, but such specific use does not require a different class or type.

• The cv::Mat class is used to represent dense arrays of any number of dimensions. In this context, dense means that for every entry in the array, there is a data value stored in memory corresponding to that entry, even if that entry is zero.

• Most of the used types have their own Mat constructor. -> open docs

cv::Mat

• dims - number of dimensions

• rows & cols - number of rows and columns. If dims>2, (rows, cols) =(-1,-1)

• size – Size(cols, rows)

• *data - pointer to the data

• step – defines data layout of the matrix (regarded as an array)

• This latter member allows cv::Mat to behave very much like a smart pointer for the data contained in data. The data array is laid out such that the address of an element whose indices are (for 2D array) i and j:

&(mat_i,j) = mat.data + mat.step[0]*i + mat.step[1]*j

• This means that 2-dimensional matrices are stored row-by-row, 3-dimensional matrices are stored plane-by-plane, and so on.

• Each element of the data in a cv::Mat can itself be either a single number or multiple numbers.

Creating array/matrix

• You can create an array simply by instantiating a variable of type cv::Mat. An array created in this manner has no size and no data type.

• You can, however, later ask it to allocate data by using a member function such as create().

• One variation of create() takes as arguments a number of rows, a number of columns, and a type, and configures the array to represent a two-dimensional object.

• The type of an array determines what kind of elements it has.

• Valid types in this context specify both the fundamental type of element as well as the number of channels. All such types are defined in the library header, and have the form CV_{8U,16S,16U,32S,32F,64F}C{1,2,3}.

• For example, CV_32FC3 would imply a 32-bit floating-point three-channel array.

• If you prefer, you can also specify these things when you first allocate the matrix. There are many constructors for cv::Mat, one of which takes the same arguments as create()

THE MOST IMPORTANT PARAGRAPH IN THE BOOK

• It is critical to understand that the data in an array is not attached rigidly to the array object.

• The cv::Mat object is really a header for a data area, which — in principle — is an entirely separate thing.

• For example, it is possible to assign one matrix n to another matrix m (i.e., m=n). In this case several things happen:• the data pointer inside of m will be changed to point to the same data as n. • the data pointed to previously by the data element of m (if any) will be deallocated. • the reference counter for the data area that they both now share will be incremented.• the members of m that characterize its data (such as rows, cols, and flags) will be updated to

accurately describe the data now pointed to by data in m.

• This all results in a very convenient behavior, in which arrays can be assigned to one another, and the work necessary to do this takes place automatically behind the scenes to give the correct result.

Accessing Array Elements Individually

• By location -> member function at<>()• The way this function works is that you specialize the at<>() template to the

type of data that the matrix contains, then access that element using the row and column locations of the data you want.

• cv::DataType<>

Accessing Array Elements Individually

• Through iteration• use the iterator mechanism built into cv::Mat• cv::MatConstIterator<> -> The cv::Mat methods begin() and end() return objects of

this type.

• This method of iteration is convenient because the iterators are smart enough to handle the continuous packing and noncontinuous packing cases automatically, as ell as handling any number of dimensions in the array.

Accessing Array Elements by Block

• member functions of the cv::Mat class• row()

• col()

• rowRange() -> rowRange(i0, i1) == rowRange(cv::Range(i0,i1))

• colRange()

• m(cv::Range(i0,i1), cv::Range(j0,j1)) -> Array corresponding to the subrectangle of matrix m with one corner at i0, j0 and the opposite corner at (i1-1, j1-1)

• m( cv::Rect(i0,i1,w,h) ); Array corresponding to the subrectangle of matrix m with one corner at i0, j0 and the opposite corner at (i0+w-1, j0+h-1)

Back to Sobel

• Perform Sobel edge detection step by step by accessing each individual pixel and its neighborhood.

• S = [-1, -2, -1; 0, 0, 0;1, 2, 1]; //horizontal edges

Algebra and cv::Mat

• Operations available for matrix expressions• m0 + m1, m0 – m1;• m0 + s; m0 – s; s + m0, s – m1;• -m0;• s * m0; m0 * s;• m0.mul( m1 ); m0/m1; // per element• m0 * m1; // matrix multiplication• m0>m1; m0>=m1; m0==m1; m0<=m1; m0<m1; // per element comparison• m0&m1; m0|m1; m0^m1; ~m0;

m0&s; s&m0; m0|s; s|m0; m0^s; s^m0 ; //bitwise logical operations

More Things an Array Can Do

• m1 = m0.clone();

• m0.copyTo( m1 );

• m0.copyTo( m1, mask ); //only entries indicated in the array mask are copied

• m0.convertTo(m1, type, scale, offset); // Convert elements of m0 to type (e.g., CV_32F) and write to m1 after scaling by scale (default 1.0) and adding offset (default 0.0)

• m0.reshape( chan, rows);

• m0.empty();

Final goal – act like cats

1. Detect red circle in the image

2. Extract it from the background

3. extract its center position

4. Display the path of the circle through the given image set

Images and template code given:• track_red_circle.cpp

• \fotke

Hints and advices

• Use only one image for start!• decide upon color space you want to use (RGB, Lab, HSV)• debugging is not so simple, show your image between each image

processing step, use multiple windows• want to extract a range of values -> use cv::inRange(src, lowerb, upperb,

dst)• Image is of bad quality? Maybe use some blurring to make objects

uniform?• Object has holes or not-so-round shape? Morphological operations!• I detect several objects? bwlabel (connectedComponents)• Thresholding is boring? Try using Hough circle transform!

Time performance of accessing single elements• DEBUG RELEASE

Reference

[1] http://docs.opencv.org/master/– OpenCV 3.2.0-dev Documentation

[2] Adrian Kaehler and Gary Bradski. 2016. Learning OpenCV 3: Computer Vision in C++ with the OpenCV Library (1st ed.). O'Reilly Media, Inc..

[3] Kenneth Dawson-Howe. 2014. A Practical Introduction to Computer Vision with OpenCV (1st ed.). Wiley Publishing.