Super-Resolution Based on Open Source Computer

Vision Library EmguCV

Lee Jee Shen

Faculty of Engineering and Technology

Multimedia University

Malacca, Malaysia

me@jeeshenlee.com

Authors Name/s per 2nd Affiliation (Author)

Dept. name of organization (Line 1 of Affiliation - optional)

Name of organization - acronyms acceptable (line 2)

City, Country (line 3)

name@xyz.com optional (line 4)

AbstractIn this paper, we present a simple implementation of

Super-Resolution technique using EmguCV computer vision

library. Instead of static images, we use real-time video inputs

from web camera as the input. Each Super-Resolution image

result are processed based on multiple frames of the video input.

The Super-Resolution technique applied our application is

implemented with the following sequence image acquisition,

image registration, sub-pixel image shift calculation, image

transformation and lastly image interpolation.

Index TermsSuper-Resolution, EmguCV, real-time video

input image processing

I. INTRODUCTION

Super-Resolution is an image processing technique that

takes multiple shifted images of a same scene and generate a

higher resolution image copy of the scene by considering lost

image information that are captured at the other frames of the

same scene due to vibration or image shift. The main concept

behind Super-Resolution is that CCD camera sensors are finite

and discrete while the light signal is continuous. In one image

frame taken by CCD camera, there are many lost information

of the scene that does not get picked up by the camera sensor

grid because some continuous light signal will not hit the CCD

sensor grid directly. Figure 1 illustrate the scenario when a full

hit of light signal on CCD camera sensor grid while the second

image (Fig.2) illustrate the scenario of none full hit

(information lost) when the light hit in the might of the sensor

grid.

EmguCV is an open source computer vision library written

on .NET platform. Its a translated copy of OpenCV with some additional feature specifically written for .NET programmer.

EmguCV act as a wrapper of OpenCV to access memory level

data which was considered as (unsafe) programming in the

.NET environment. The library is written using C#.NET

programming languages and it can be compiled into Window,

Linux, Max OS, Android compatible with the Mono compiler.

Mono Mono is a compiler written to compile C# language

standard into the target platform. On top of EmguCV, we are

using another open source computer vision library AForge

specifically to take web camera input.

Fig. 1. CCD sensor are recording the intensity of the light based on the amount of intensity hit the sensor grid.

Fig. 2. Extra information (the tip feature of the object) are better recorded.

II. SUPER-RESOLUTION TECHNIQUE

There are many Super-Resolution techniques proposed but

generally it involves the following steps image acquisition, image registration, sub-pixel shift estimation, and interpolation.

In our application, we started by first acquiring the first five

image frames from web camera and then feed them into the

Super-Resolution image processing function that we

implemented to get the first Super-Resolution of our video

stream. From then onward, we repeatedly update the Super-

Resolution frame based on the latest video frames acquired

from the web camera.

Fig. 3. Three image video frames (on the left) are taken in this example on the same scene. Based on the feature, sub-pixel shift of the scene are

detected on Frame 2 and Frame 3.

Fig. 4. Sub-pixel shift are estimated and corrected before merging into the final result to produce better estimation of the real scene (Super-

Resolution).

III. VIDEO FRAME ACQUISITION

Image acquisition from the web camera are done using the

AForge library. VideoCaptureDeviceForm class are used to

detect and selected the web camera. After the selection,

IVideoSource passed and initialised. The NewFrame event of

the IVideoSource are subscribed in the application to grab the

latest video frame from web camera and further triggering the

Super-Resolution processing. On top of this, we added a

feature to define Region of Interest (ROI) to select specific

region of an image input for processing. Region of Interested

are registered using System.Windows.Rectangle class with the

help of MouseDown and MouseUp event.

IV. IMAGE REGISTRATION

Image registration is the most important part of Super-

Resolution because its responsible for the feature detection and also the sub-pixel calculation that determine the quality for

the output. To make out Super-Resolution implementation

reusable in other application, we create a separate project and

wrote our SuperResolutionFilter class. In the class, we defined

the following object properties:

TABLE I. CLASS PROPERTY

Property Name Description

ScaleFactor Storing the scaling factor of the target output.

FramesFusionAccuracyThreshold

Minimum accuracy threshold that the shift result confidence must comply before

considering as a valid estimation

InputFrames List of input frames to store the latest frames from web camera.

SuperResolutionFrame Storing the latest Super-Resolution image

result.

Roi Storing the Region of Interest of the image.

ShiftResult Storing the X and Y shifted pixel in float data

type.

In order to find the sub-pixel of images with reference to

the first image, we first find a feature on an image to track. We

are using GoodFeaturesToTrack method of EmguCV to detect

features on the video inputs. Before the images are pass to

GoodFeaturesToTrack it was convert into the right image

format required by EmguCV which is Image. To

get better result, the resulted features detected are further

passed to FindCornerSubPix method of EmguCV to get sub-

pixel accurate corner of the image.

Based on the features found, a sub-pixel shift of important

feature are calculated using OpticalFlow.PyrLK function.

OpticalFlow.PyrLK calculates the optical flow for a sparse

features set (detected in the earlier steps) using iterative Lucas-

Kanade method in pyramids. To produce better result, we

added some additional code to calculate the confidence level of

each calculated shift. This confidence level value will be used

later in the merging steps to filtered inaccurate sub-pixel shift

estimation.

Based on the calculated sub-pixel shift results and

confidence level, the images are transforms using

WarpPerspective to correct the sub-pixel shift based on the

estimation. Lastly, the images list are merge using RunningAvg

of EmguCV to produce the Super-Resolution image. The

application then repeat the same process describe above on all

the incoming video frames except this time using the latest

Super-Resolution image calculated in the last iteration as

reference to produce better result.

V. EXPERIMENTAL RESULTS

To test the implemented Super-Resolution technique

implemented using EmguCV, we run our application with

many test conditions. Below are sample of the results recorded

during the test.

Fig. 5. The implemented Super-Resolution application using Visual Studio 2013 and EmguCV.

Fig. 6. On the left is the video stream from web camera and on the right is the result stream generated live.

Fig. 7. Improvement can be seen on the right-hand generated Super-Resolution image as compare to the original image taken from web

camera (on the right) when both images are zoom in for inspection.

Fig. 8. Bicubic interpolation out of the same scene.

Fig. 9. Super-Resolution image is clearer and with less noise as compare to bicubic interpolation (Figure 8) and Super-Resolution result can be

further improve by increasing the number of frames taken into

consideration. The top right edge of the paper are sharper compare to bicubic interpolation image.

VI. CONCLUSION

In this paper, a simple Super-Resolution algorithm is

implemented and presented. We see improvement in terms of

the quality of the Super-Resolution over the original images

taken from video camera. The algorithm implemented is fairly

simple but fast as its able to compute Super-Resolution video stream in real-time. With this technique, one can take higer

resolution picture with low resolution CCD.

ACKNOWLEDGMENT

The preferred spelling of the word acknowledgment in America is without an e after the g. Avoid the stilted expression, One of us (R. B. G.) thanks . . . Instead, try R. B. G. thanks. Put applicable sponsor acknowledgments here; DO NOT place them on the first page of your paper or as

a footnote.

REFERENCES

List and number all bibliographical references in 9-point

Times, single-spaced, at the end of your paper. When

referenced in the text, enclose the citation number in square

brackets, for example: [1]. Where appropriate, include the

name(s) of editors of referenced books. The template will

number citations consecutively within brackets [1]. The

sentence punctuation follows the bracket [2]. Refer simply to

the reference number, as in [3]do not use Ref. [3] or reference [3]. Do not use reference citations as nouns of a sentence (e.g., not: as the writer explains in [1]).

Unless there are six authors or more give all authors names and do not use et al.. Papers that have not been published, even if they have been submitted for publication, should be

cited as unpublished [4]. Papers that have been accepted for publication should be cited as in press [5]. Capitalize only the first word in a paper title, except for proper nouns and

element symbols.

For papers published in translation journals, please give the

English citation first, followed by the original foreign-language

citation [6]. [1] G. Eason, B. Noble, and I. N. Sneddon, On certain integrals of

Lipschitz-Hankel type involving products of Bessel functions, Phil. Trans. Roy. Soc. London, vol. A247, pp. 529551, April 1955. (references)

[2] J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.6873.

[3] I. S. Jacobs and C. P. Bean, Fine particles, thin films and exchange anisotropy, in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271350.

[4] K. Elissa, Title of paper if known, unpublished.

[5] R. Nicole, Title of paper with only first word capitalized, J. Name Stand. Abbrev., in press.

[6] Y. Yorozu, M. Hirano, K. Oka, and Y. Tagawa, Electron spectroscopy studies on magneto-optical media and plastic

substrate interface, IEEE Transl. J. Magn. Japan, vol. 2, pp. 740741, August 1987 [Digests 9th Annual Conf. Magnetics Japan, p. 301, 1982].

[7] M. Young, The Technical Writer's Handbook. Mill Valley, CA: University Science, 1989.