Video Stabilization Using RANSAC

In this work, to stabilize a video, a salient feature in a frame of the video is used as an anchor to cancel all changes in the relationship with the salient feature. This procedure must be implemented on all video with the knowledge that a salient features in the video frames. With this way, the deformations which is caused by camera movements are corrected. In our algorithm; once, we determine the affine image transformations between all adjacent frames of a video sequence with a random sample consensus (RANSAC) method is used to show correspondence between two images. Then, we deform video frames to achieve a stabilized video.And we used Matlab vision toolbox to implement this algorithm.

We handled this work under four step;

1. Reading frames 2. Finding salient points in frames and matching corresponded points3. Model fitting whit RANSAC 4. Estimating affine transform warping frames

Reading Frames

In this step we read the first and second frames of a video sequence as an intensity image. Because the color is not necessary for the stabilization algorithm, also grayscale values improve the algorithm speed.

dosyaIsmi='ornek.wmv'; hVideoSrc = vision.VideoFileReader(dosyaIsmi, ... 'ImageColorSpace', 'Intensity',... 'VideoOutputDataType', 'single'); kareA=step(hVideoSrc); kareB=step(hVideoSrc);

cvexShowImagePair(kareA, kareB, 'A karesi', 'B karesi');

The first and the second frames of video

We can see the movie between frames so..

figure; imshow(cat(3,kareA,kareB,kareB));

Above we show two images side by side, and producing an image composed of red-cyan, pixel by pixel in order to demonstrate the difference between them. Clearly, there is a. Large vertical and horizontal movement between two frames

Finding Salient Points In Frames And Matching Corresponded Points

In this step, to have the best chance we want the points around the image of great features such as corners. We used the CornerDetector object of which method is local intensity comparison (Rosen & Drummond).Also we can specify the method as one of Harris corner detection (Harris & Stephens) or Minimum eigenvalue (Shi & Tomasi), but we prefer the fastest of them, Local intensity comparison (Rosten & Drummond).

enFazlakosesayisi = 120;esikDeger = 1e-3;hCD = vision.CornerDetector( ... 'Method','Local intensity comparison (Rosen & Drummond)', ... 'MaximumCornerCount', enFazlakosesayisi, ... 'CornerThreshold', esikDeger, ... 'NeighborhoodSize', [11 11]);

Beyond, we restricted the maximum number corner to find with MaximumCornerCount Identified as 120.So we get rid of unnecessary workload.If we had used a video which has got lots of corners in frames, we must increase the maximum corner count to take a correct result from this stabilization algorithm.

koselerA = step(hCD, kareA);koselerB = step(hCD, kareB); %burada yaptığımızı izleyelimcvexShowImagePair(kareA, kareB, 'A daki köşeler', 'B deki köşeler', ... 'SingleColor', koselerA, koselerB);

the corners

off-topic in parentheses : If the cornerdetector parametries are changed

the other video frames for maximumu corner count defined as 500

the video frames for maximumu corner count defined as 10 you can see the decreasing of corner counts (compare with previous frames

In addition we declared CornerThreshold which specifies the minimum metric value that indicates a corner as a positive scalar number as 1e-1 and NeighborhoodSize which is the size of suppressed region around detected corner as 11 to 11 matrix range.

the frames for NeigborhoodSize=[5 5]

We used in this step of the algorithm future based matching. Therefore before matching we extracted the features from the points so we get rid of some corners which are not suitable. For each point, extracted a 11 by 11 block centered around. The corresponding cost is used between the points is the sum of squared differences (SSD) between their respective areas of the image.

[featureA, noktalarA] = extractFeatures(kareA, koselerA);[featureB, noktalarB] = extractFeatures(kareB, koselerB); eslenenciftler = matchFeatures(featureA, featureB);noktalarA = noktalarA(eslenenciftler(:, 1), :);noktalarB = noktalarB(eslenenciftler(:, 2), :);

cvexShowMatches(kareA, kareB, noktalarA, noktalarB, 'A', 'B');

Corresponded points frame A and frame B

Model Fitting With RANSAC And Estimating Geometric Transform

In the previous step we extracted the correspondence of the points but some of some the points is incorrect.So we must use RANSAC algorithm to find a robust estimate of the geometric transformation between two images.

The Ransac Algorithm

1. Select randomly the minimum number of points(S) required to determine the model parameters. 2. Solve for the parameters of the model. 3. Determine how many points from the set of all points fit with a predefined threshold d. 4. If the fraction of the number of inliers over the total number points in the set exceeds a predefined threshold T , re-estimate the model parameters using all the identified inliers and terminate. 5. Otherwise, repeat steps 1 through 4 (maximum of N times).

hGTE = vision.GeometricTransformEstimator;%geometrik transformu tahmin etmek için kullacağızhGT = vision.GeometricTransformer;hGTp = vision.GeometricTransformer;N = 5;% N Ransacta uygun modeli bulmak için deneme saysını 5 olarak belirledikTFORM = cell(1,N);cost = zeros(1,N);INLIER_INDEX = cell(1,N);

for j=1:N % affine transformunu tahmin et [TFORM{j},INLIER_INDEX{j}] = step(hGTE, noktalarB, noktalarA); % resmi düzeltelim kareBp = step(hGT, kareB, TFORM{j}); % hatayı hesaplayalım cost(j) = sum(sum(imabsdiff(kareBp, kareB))); end % en iyi sonucu alalım. [~,ix] = min(cost); kareBp= step(hGT, kareB, TFORM{ix}); H = [TFORM{ix} [0 0 1]'];

To implement RANSAC, we defined N as 5 to take optimum result (if we increase N the stabilization improves but the process time increase.)

Also we used to estimate geometric transform GeometricTransformEstimator object which estimates geometric transformation from matching point pairs and returns the transform in a matrix.

Inlier points

Estimating Affine Transform And Warping The Frame

In this step, we implemented the estimating of affine transform with cvexTformToSRT function and warped the frames.

HsRt = cvexTformToSRT(H); Hcumulative = HsRt * Hcumulative; kareBp = step(hGTp, kareB, Hcumulative);

a wrapped frame

Here , the all function which stabilize a video, videoDengeleme(dosyaIsmi) and explanation in Turkish;

function [] = videoDengeleme( dosyaIsmi )%Bu fonksiyon video stabilizasyonu yapmaktadır% Mücahit ALTINTAŞ 2012 %videoyu kullanacağımız hale getirelim%Stabilizasyon işleminde renklerle işimiz olmadığı için gray%değerlerle kullanıyoruz.hVideoSrc = vision.VideoFileReader(dosyaIsmi, ... 'ImageColorSpace', 'Intensity',... 'VideoOutputDataType', 'single'); %Burada köşe bulmak için vision toolbox unda bulunan CornerDetector %fonksiyonunun Local intesity comparison (Rosen&Drummond) metodunu kullandık%MaximumCornerCount yani karemizde en fazla bulmak itediğimiz köşe sayısını%120 olarak belirledik, CornerThreshold yani incelediğimiz noktanın köşe %olarak kabül edilmesi için (NeighborhoodSize kadar) etrafındaki intesisty %değerleriyle arasında olaması gereken farkı e-3 olarak belirledik ve %NeighborhoodSize yani her bir nokta için bakılacak matris aralığı 11 e 11%http://www.mathworks.com/help/toolbox/vision/ref/vision.cornerdetectorclass.htmlenFazlakosesayisi = 500;esikDeger = 1e-3;hCD = vision.CornerDetector( ... 'Method','Local intensity comparison (Rosen & Drummond)', ... 'MaximumCornerCount', enFazlakosesayisi, ... 'CornerThreshold', esikDeger, ... 'NeighborhoodSize', [11 11]); %RANSAChGTE = vision.GeometricTransformEstimator;%geometrik transformasyonu tahmin etmek için kullacağızhGT = vision.GeometricTransformer;hGTp = vision.GeometricTransformer;N = 5;% N Ransacta uygun modeli bulmak için deneme saysını 5 olarak belirledikTFORM = cell(1,N);cost = zeros(1,N);INLIER_INDEX = cell(1,N); hVPlayer = vision.VideoPlayer;kareB =step(hVideoSrc);kareBp = kareB;Hcumulative = eye(3); while ~isDone(hVideoSrc) % freamleri okuyalım kareA = kareB; kareAp = kareBp; kareB = step(hVideoSrc); koselerA = step(hCD, kareA); koselerB = step(hCD, kareB); %burada yaptığımızı izleyelim

%cvexShowImagePair(kareA, kareB, 'A daki köşeler', 'B deki köşeler', ... % 'SingleColor', koselerA, koselerB); %## %burada feature tabanlı ilişkili eşleme (feature-based corresponding %matching) gerçekleştireceğiz %## %biz yukarda köşeleri bulduk ama bu bulduğumuz köşelerden işimize yarayacak %olan feature (ayırt edici diyebiliriz) lar bizim işimize yarayacak.onun için %köşelerin içinden featurelarımızı çıkartıyoruz. bu işlem için extractFeature %foksiyonunu kullanacağız.Bu fonksiyonu default halde blockSize değeri 11 dir. %http://www.mathworks.com/help/toolbox/vision/ref/extractfeatures.html [featureA, noktalarA] = extractFeatures(kareA, koselerA); [featureB, noktalarB] = extractFeatures(kareB, koselerB); % MatchFeature fonksiyonuyla bulduğumuz ayırt edici feature ları sum of % square difference (SSD-kareleri farkının toplamı) yöntemiyle eşliyoruz %http://www.mathworks.com/help/toolbox/vision/ref/matchfeatures.html %Computer Vision Study Notes sayfa 33 eslenenciftler = matchFeatures(featureA, featureB); noktalarA = noktalarA(eslenenciftler(:, 1), :); noktalarB = noktalarB(eslenenciftler(:, 2), :); %yaptığımızı izleyelim %cvexShowMatches(kareA, kareB, noktalarA, noktalarB, 'A', 'B'); %## %burada eşlenen çiftlerin çoğu doğru olmakla birlikte doğru olmayan bazı %eşlemelerde var. bunlardan kurtulmak için RANSAC ı kullanacağız %## %Computer Vision Study Notes sayfa 44 %http://www.mathworks.com/help/toolbox/vision/ref/vision.geometrictransformestimatorclass.html %http://www.mathworks.com/help/toolbox/vision/ref/vision.geometrictransformerclass.html for j=1:N % affine transformunu tahmin et [TFORM{j},INLIER_INDEX{j}] = step(hGTE, noktalarB, noktalarA); % resmi düzeltelim kareBp = step(hGT, kareB, TFORM{j}); % hatayı hesaplayalım cost(j) = sum(sum(imabsdiff(kareBp, kareB))); end % en iyi sonucu alalım. [~,ix] = min(cost); kareBp= step(hGT, kareB, TFORM{ix}); H = [TFORM{ix} [0 0 1]']; HsRt = cvexTformToSRT(H); Hcumulative = HsRt * Hcumulative; kareBp = step(hGTp, kareB, Hcumulative); % videonun dengenmiş halini step(hVPlayer, cat(3,kareAp,kareBp,kareBp));end

release(hVideoSrc);release(hVPlayer); end%NOT:"http://www.mathworks.com" kaynak olarak kullanılmıştır

Print Scrn Image of this work

* the video 'bisikletli.avi' from http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/