L2Bot using C# and Emgu

J. Ruszala

Visual Studio Express 2013 for

Windows Desktop

2

Visual Studio Express 2013 for Windows Desktop

• Visual Studio Express 2013 for Windows Desktop

• Download from: http://www.visualstudio.com/downloads/download-visual-studio-vs#d-express-windows-desktop

• Follow default installation options

3

L2Bot Button Control

4

L2Bot Button Control

• Objective: Create a Win Forms application to drive the L2Bot in any direction by using a graphic user interface (GUI)

5

L2Bot Button Control

• Add five buttons to the windows form to control the robot:

• Forward

• Backward

• Left

• Right

• Stop

• Giving the buttons meaningful names will help when referencing them in your C# file

• Double click each of the buttons to create an event handler

6

L2Bot Button Control

• The L2Bot motor controller is physically connected to the PC via a serial port. To access these serial port controls you must add the LoCoMoCo (Low Cost Motor Controller) class to your project.

7

L2Bot Button Control

• After adding the class to your project we are ready to start adding C# code

• Add a class LoCoMoCo variable

• Set that variable to a new instance of LoCoMoCo in the Form1 method

8

L2Bot Button Control

• For each of your button click event handlers add code to call the appropriate motor controller method

9

L2Bot Button Control

• Add override method to close the serial port connection when the program is closed

10

L2Bot Dead Reckoning Control

11

L2Bot Dead Reckoning Control

• Objective: Drive L2Bot to a new position purely based on direction and time

12

L2Bot Dead Reckoning Control

Start Button Event

Forward Event

• Command forward

• Sleep for time t

Spin Event

• Command L/R turn

• Sleep for time t

Forward Event

• Command forward

• Sleep for time t

Stop Event

• Command Stop

13

L2Bot Dead Reckoning Control

• Start by creating a new WinForms project and adding one large button to the form

• This button will start and stop the program

• Double click the button to create its event handler

14

L2Bot Dead Reckoning Control

• In the button click event handler you will add the dead reckoning controls to be sent to the motor controller

• To delay the transmission of motor control commands you can add sleep method calls that will pause your program for a specified number of milliseconds

15

L2Bot Dead Reckoning Control Using Timers

Start Button Event

Forward Event

• Command forward

• Set timer & start

Spin Event

• Command L/R turn

• Set timer & start

Forward Event

• Command forward

• Set timer & start

Stop Event

• Command Stop

• Stop timer

16

L2Bot Dead Reckoning Control Using Timers

• As with the simple dead reckoning program, you must first add the LoCoMoCo class to your project

• Then add the following class variables

• The Timer and status variables are needed to control the time between motor controller commands

17

L2Bot Dead Reckoning Control Using Timers

• Create a new instance of the LoCoMoCo and Timer class in the Form1 method

• Then create an event handler for the Timer instance and disable the timer until it is used

18

L2Bot Dead Reckoning Control Using Timers

• In the button click handler, we will add the following code.

• If running is false, begin the dead reckoning control

• If running is true, abort the current dead reckoning control

19

L2Bot Dead Reckoning Control Using Timers

• In the Timer tick event handler we will add the following code

• The switch statement will handle the control sequence discussed in the event flow chart

20

Emgu Setup

21

Downloading

• Download Emgu Version 2.9.0 beta from the following link

• http://sourceforge.net/projects/emgucv/files/emgucv/2.4.9-beta/

• Choose the correct version dependent on the operating system you will be using

• Once downloaded, run the install .exe

22

Emgu CV Path Reference

• Once Emgu CV is installed find the installation path to Emgu Version/bin/x64 folder

• Copy this path location to the clipboard

23

Emgu CV Path Reference

• From the Windows Desktop right click on the start menu and select “System”

24

• Then select “Advanced system settings” from the left menu

Emgu CV Path Reference

• Select “Environment Variables” from the System Properties window

25

Emgu CV Path Reference

• Select the “Path” list item from the “System variables” group box and click Edit…

26

Emgu CV Path Reference

• Scroll to the very end of the “Variable value:” text box

• Insert a semicolon and add the emgu/bin/x64 path to the end of the path

• Click OK

27

Emgu Setup in Visual Studio

• Create a new Visual C# Windows Forms Application

28

Emgu Setup in Visual Studio

• Emgu GUI components

• Import the OpenCV GUI components by following the procedure outlined here: http://www.emgu.com/wiki/index.php/Add_ImageBox_Control

• The ‘Emgu.CV.UI.dll’ file referenced can be found in the ‘C:\Emgu\emgucv-windows-universal-cuda 2.9.0.1922\bin’ folder

29

Emgu Setup in Visual Studio

• Once the GUI components have been added, drag and drop an ImageBox instance on to the current Windows Form

30

Emgu Setup in Visual Studio

• Reference the following .dll files in the project by right clicking the ‘References’ folder in the Solution Explorer to the right of the screen and selecting ‘Add Reference’

31

Emgu Setup in Visual Studio

• Using the ‘Browse’ tab, navigate to C:\Emgu\emgucv-windows-universal-cuda 2.9.0.1922\bin and select the following files:

32

• Emgu.CV

• Emgu.CV.UI

• Emgu.Util

Emgu Setup in Visual Studio

• The final step in the setup is to change the ‘Platform target’ to x64 if you are on a 64 bit OS

• From the ‘Project’ menu selectProjectName Properties…

33

Emgu Setup in Visual Studio

• Select x64 for the ‘Platform target’ in the Build tab

34

Emgu Basics

• Once the Emgu references have been setup in Visual Studio, you can begin writing your C# code

• First, include the referenced Emgu files in your Forms .cs file

• using Emgu.CV;

• using Emgu.Util;

• using Emgu.CV.Structure;

35

Emgu Basics

• Next, you will need to get an instance of the Capture class that allow you to capture images from your input device.

• Capture _capture = null;

• Add the Capture instance as a Class variable since we will use it in several of the Class methods

36

Emgu Basics

• In the form’s Load method you will have to:

• Create a new instance of the Capture method

• Create an event handler to receive image frames from the input device

• Start the Capture instance

private void Form1_Load(object sender, EventArgs e){

_capture = new Capture();_capture.ImageGrabbed += capture_ImageGrabbed;capture.Start();

}

37

Emgu Basics

• In the Capture event handler you need to:

• Retrieve the captured image frame

• Resize it to fit your imageBox element

• Display the image in the imageBox

void _capture_ImageGrabbed(object sender, EventArgs e)

{

Image<Bgr, Byte> frame =_capture.RetrieveBgrFrame().Resize(imageBox1.Width, imageBox1.Height,Emgu.CV.CvEnum.INTER.CV_INTER_LINEAR);imageBox1.Image = frame;

}

38

Emgu Basics

• As a last step, you need to override the onClosing method to stop the Capture instance. This will prevent an exception from being thrown when the program is closed.

protected override void OnClosing(CancelEventArgs e){

_capture.Stop();base.OnClosing(e);

}

39

Color Conversion

40

Color Conversion

• Grayscale

• In addition to retrieving a Bgr image in the ImageGrabbed event handler, you can also retrieve a grayscale image

• To do this we change the Image type from Bgr to Gray and replace the RetreiveBgrFrame method call with RetreiveGrayFrame

41

Color Conversion

• Binary – Pure Black and White

• To produce a pure black and white image, you first start with a Gray image and then call the ThresholdBinary method. The method accepts the min and max threshold values for the binary conversion

42

Pixel Counter

43

Pixel Counter

44

Objective: Create a Windows Forms application using Emgu that counts the number of white pixels in the image.

If > 50% set background RED

If <= 50% set background GREEN

Pixel Counter

• Create a new Win Forms application and repeat the steps outlined in the Emgu Basics section.

• Create one new class variable that will be used for the lower threshold in your binary image conversion method call

45

Pixel Counter

• In the ImageGrabbed method, retrieve a Gray frame and convert it to a binary image using the threshold variable you just created as the lower bound

• Display that image to the screen

46

Pixel Counter

• Iterate through the rows and columns

• Count the number of white pixels

47

Pixel Counter

• Set the background color of the form based on the total number of white pixels counted

48

Pix Count – Interactive Threshold

• The binary threshold variable can be made interactive by adding a simple key event handler

• In the Form1 method

• Set the KeyPreview attribute to true

• Create a new KeyEventHandler

49

Pix Count – Interactive Threshold

• In the KeyDown event handler method

• Determine if the event was caused by the up or down key

• Increase or decrease the value of the minimum binary threshold

50

L2Bot Stop at White

51

Stop at White

• Objective: Create a Windows Forms application using Emgu that drives the L2Bot forward until it encounters a white sheet of paper

• Hint: Start with the OpenCV_Pixel_Counter program

52

Stop at White

• Add the LoCoMoCo class to the OpenCV_Pixel_Counterprogram

• After counting the pixels, determine if they are greater than some threshold

• If greater, send a stop command to the motor controller

53

Stop at White

• Create a new instance of the LoCoMoCo class in the From1_Load method above the creation of your Emguinitialization code

• Command the robot to drive forward

54

Stop at White

• After evaluating the number of white pixels within the ImageGrabbed event handler, you will determine if the robot should stop or continue

55

Pixel_Counter Stop_At_White

L2Bot Line Following

56

L2Bot Line Following

• Objective: Create a Win Forms application that follows a white line using Emgu and the L2Bot

57

L2Bot Line Following

• Combining the Emgu image capture/manipulation applications and L2Bot control programs with some additional logic can produce a line following robot

• To start with, create an Emgu application (following the same steps you used in the past) and then add the LoCoMoCo class to the project. Alternatively, you may wish to start with the OpenCV_Pixel_Counter program as a base.

58

L2Bot Line Following

• To follow any line the computer must be able to:

• Distinguish the line from the surroundings

• Determine where the line is in relation to the robot

• Distinguishing the line from the surroundings

• Since the line the robot will follow is white and the floor color is dark, the line can easily be distinguished by converting the image to a pure black and white image

• image.ThresholdBinary(loThresh, hiThresh);

59

L2Bot Line Following

• Determining line location

• There are many ways to determine where the line is in relation to the robot. Some are very simple and some require some complex strategies and math

• The simplest method involves dividing the image up into threecolumns and then counting thenumber of white(line) pixels ineach column

60

L2Bot Line Following

• If the number of white pixels in column 1 is greater than the number of white pixels in columns 2 & 3 – Turn LEFT

61

If the number of white pixels in columns 2 is greater than the number of white pixels in columns 1 & 3 – Go FORWARD

If the number of white pixels in column 3 is greater than the number of white pixels in columns 1 & 2 – Turn RIGHT

L2Bot Line Following

• Once you have a Win Forms application that contains the essential files for both the motor controller and Emgu you can begin to add the line following logic

• From within the ImageGrabbed event handler:

• Retrieve a gray frame

• Convert the gray frame to a binary image

• Get the width and height of the captured image

• Create variables to hold the amount of white pixels in each column

• Iterate through all the image pixels to find white pixels and increment the corresponding column variable

• Command the L2Bot based on the column with the greatest number of white pixels

62

L2Bot Line Following

• Create variables to hold the amount of white pixels in each column

• Iterate through all the image pixels to find white pixels and increment the corresponding column variable

63

L2Bot Line Following

• Command the L2Bot based on the column with the greatest number of white pixels

64

Obtaining BGR Values From An

Image65

BGR Values

• Objective: Get the BGR (Blue, Green, Red) values of a pixel when selected using the mouse

66

BGR Values

• Begin by adding 6 labels to the form that contains your Emgu ImageBox

• Change the text attribute of the three labels in the first column to display the Red, Green, and Blue Labels

• Change the label name attribute of the 3 labels in the second column to reflect the containers the color values will be displayed in

67

BGR Values

• Create a class Image variable

• In the ImageGrabbed event handler, save the grabbed image to the class Image variable and display the image to the screen

68

• After your standard Emgu initialization code, add a mouse click listener for the ImageBox

• In the MouseClick event handler, retreive the color from the selected pixel and display the value to the screen

BGR Values

69

Creating a Black, White, and Red

Image70

Black, White, Red

• Objective: Display a black and white image but maintain any red pixels

71

Black, White, Red

• You can easily create the image by following these steps:

• Retrieve a gray frame

• Convert the gray frame to a binary image

• Retrieve a color frame

• Create a new color image that will be used to create the black, white, and red image

• Iterate through the color image pixels, if the pixel is red set the final images pixel color to red. Else, set the pixel to the corresponding binary image pixel value

72

Black, White, Red

• There are two ways to get/set pixel values in Emgu

• Image[Row, Column].Blue

• Image[Row, Column].Green

• Image[Row, Column].Red

• Pros – Easier to follow/use

• Cons – Slow to execute

• Image.Data[Row, Column, 0] => Blue Component

• Image.Data[Row, Column, 1] => Green Component

• Image.Data[Row, Column, 2] => Red Component

• Pros – Faster

• Cons – Harder to follow

73

Black, White, Red

• Retrieve a gray frame

• Convert the gray frame to a binary image

• Retrieve a color frame

• Create a new color image that will be used to create the black, white, and red image

74

Black, White, Red

75

• Iterate through the color image pixels, if the pixel is red set the final images pixel color to red. Else, set the pixel to the corresponding binary image pixel value

L2Bot Line Following Stop at

Color76

L2Bot Line Following w/ Cone

• Objective: Create a Win Forms application that follows a white line using Emgu and the L2Bot. The application must also stop the robot when a piece of colored paper is encountered and restart when the paper is removed.

77

L2Bot Line Following w/ Cone

• Combine what you have learned in the following programs to successfully complete the challenge:

• Stop at White

• L2Bot Line Following

• Black, White, and Red

78