Zebravision 6.0

Hans Khatri

September 2018

A Zebracorn Labs Paper

Contents

1 Introduction 3

2 Challenges 3

3 Basic Setup 3

4 Switch Detection 4

4.1 The Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4.2 Creating a “Tape” ObjectType . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4.3 Finding the Retro Reflective Tape . . . . . . . . . . . . . . . . . . . . . . . . 5

4.4 Narrowing Possible Tape Choices . . . . . . . . . . . . . . . . . . . . . . . . 6

5 Cube Detection 7

5.1 Hue, Saturation, and Value Filtering . . . . . . . . . . . . . . . . . . . . . . 7

5.2 Creating and Filtering Contours . . . . . . . . . . . . . . . . . . . . . . . . . 8

6 Data Extraction, Calculation, and Storage 8

6.1 Data Extraction: Subscribing to the ZED . . . . . . . . . . . . . . . . . . . . 8

6.2 Data Calculation: Screen To World . . . . . . . . . . . . . . . . . . . . . . . 9

6.3 Data Storage: ROS Messages . . . . . . . . . . . . . . . . . . . . . . . . . 9

7 Testing With ZMS Files 10

7.1 Creating a Reader for ZMS Files . . . . . . . . . . . . . . . . . . . . . . . . 10

7.2 Choosing the Correct Topics and Launch File . . . . . . . . . . . . . . . . . 11

8 Summary 12

2

1 Introduction

This year’s game, FIRST Power Up, produced unique challenges in terms of robot vision

and its necessary programming. The main goal of vision this year was to autonomously

detect parts of the field and individual game pieces (e.g. a vertical piece of tape that the

robot must detect or a cube that the robot must interact with) and make the data accessible

and easy to use. The way that the robot responded to sensor input will be detailed later in

the paper. Even though vision did not run on the robot during the competition, the steps

made towards game piece and field detection deserve recognition.1

2 Challenges

Integration of ROS2 into the Zebracorns code base proved to be tough but manage-

able. The Robot Operating System gives our code the robustness and ease of access

to change, add, and delete code with minimal hassle. To successfully implement the vi-

sion component for the game this year, specific objects on the field needed to be detected

and analyzed. The main target of detection was a Power Cube. The cube, a distinct shade

of yellow, produced a challenge in and of itself. We deduced that the vision team’s goal

in regards to this game piece was to autonomously detect the cube, visualize and depict

the cube in any position, and extract values that tell exactly where the cube is in relation

to the robot. For the tape attached on the side of the switch, the goal was to detect both

pieces of tape, interpret that as a goal, and extract values like depth and x-y misalignment

from the robot.

3 Basic Setup

Similar to previous years, the team used StereoLabs’ ZED Stereoscopic Camera to exe-

cute the vision system. The ZED has a dual-lens color camera, and is crucial for obtaining

raw RGB data as well as depth data. For testing, a case with LED rings was 3D printed to

fit around the ZED. These LED rings consist of multiple LEDs glued to two rings which are12016-17 Vison: Zebravison 5.02A collection of tools, libraries, and conventions that aim to simplify the task of creating complex and

robust robot behavior. About ROS

3

then attached around both the ZED’s cameras. The LEDs reflect off the tape that needs

to be detected and give the tape a distinct green color. The ZED is then connected to an

Nvidia Jetson TX2 using a standard USB connection.

The majority of the code base is written using C++ in conjunction with ROS. Also, most

images used were down sampled by scaling the raw 720p image to half its size in order

to reduce computational overhead.

4 Switch Detection

4.1 The Switch

Each switch consists of two plates, one on each end of its arm.

Figure 4.1: A full switch, as shown from the side of the field, consisting of two plates and

and an arm to create a lever.

4

Figure 4.2: The red alliance’s side of the switch. The arrow is pointing to the tape. To-

gether with the tape on the other side of the aluminum bar, they make a goal.

The tape is retro-reflective, meaning that it will reflect light back to its source (in this case

the LED rings attached to the ZED) with the least amount of scattering possible.

At this point the robot has no definition of the strips of tape and therefore, cannot distin-

guish them. Once the tape has a firm representation, it becomes possible to classify and

use.

4.2 Creating a “Tape” ObjectType

The first thing needed to successfully identify the tape as a game piece is to establish the

vertical strip located on the switch as an ObjectType. This stores the shape of the tape

in meters, each vertice being an element in a vector. This is good for determining the

relative dimensions of a specified entity in real world units.

4.3 Finding the Retro Reflective Tape

Thresholding an image is used to separate out regions of an image corresponding to

objects which we want to analyze. This separation is based on the variation of intensity

5

between the object pixels and the background pixels.

Figure 4.3: Left: Image Before Right: Image After

Using erosion and dilation on a thresholded image lets us eliminate any noise or holes in

the image, and just focus on the pixels that were not filtered out. Detecting the specific

color of the tape is now just a matter of adjusting and re-adjusting the values of green to

make sure the specified shade of green is exactly the green that is reflected off the tape

from the LED rings attached on the ZED.

4.4 Narrowing Possible Tape Choices

Detailed in a previous Zebravision paper, when ObjectTypes are used as input into a

series of checks (e.g. is the left piece of tape—or ObjectType in this case—longer than

the right piece?) they are able to serve as references and eliminate contours that do not

resemble the target pieces of tape. So in the aforementioned example, the program would

detect if the tapes are not equal length and move on. This year, the vertical orientation

allowed for a couple new checks like the distance between the right and left edges of the

tape. Through all these checks, only the two target tapes remain and are detected by the

ZED. These are what become a goal for the robot.

6

5 Cube Detection

5.1 Hue, Saturation, and Value Filtering

The two color spaces under consideration to detect the cube were the RGB color space

and the HSV color space. The RGB color space focuses specifically on filtering primary

colors, which is good for red, green, and blue values, but not so much for the neon yellow

of the cube. The HSV color space, however, separates color components from intensity,

for purposes such as lighting changes and shadow removal. The lightness and intensity

of a color, or the hue, can be much more exact and cover a wide range of lighting pos-

sibilities. This allows us to accurately suggest not only a range of hue, but also a range

of saturation and value. Filtering with the HSV color space means we can take into ac-

count the lighting, which varies from event to event as well as position on the field. All the

data, checks, and calculations from this section are stored in a custom node called the

cube_detection node.

Figure 5.1: Hue, Saturation, and Value displayed in a cylindrical shape.

7

5.2 Creating and Filtering Contours

The next step after finding anything detected by the camera with the correct yellow hue,

saturation, and value is to make sure that the only contours the program actually recog-

nizes are the cubes. To do this, we sorted contours using logic similar to that of tape

detection. Filtering out contours that did not resemble a cube in different specific aspects

allowed us to hone in on objects that are cube-like in shape, color, and size.

The checks used to determine the ideal cube include:

1. Creating an equation that relates the depth of the cube to the area of the cube. If

the ratio is skewed too high or too low, the check will fail.

2. Comparing the width and height of the contour to see if it is relatively cubical in

shape. If the height is too much larger than the width or vice versa, the check will

fail.

3. Making sure that the contour has four or more polygonal vertices—which a cube

should have—and if there are three or less, the check will fail.

4. Ensuring that all of the minuscule contours that the program finds will be filtered out

through a pixel error check. If a contour is under a certain pixel area, the check will

fail.

Once a contour passes all of these checks, it is drawn, a bounding rectangle is drawn

around it, and the distance from the camera is displayed (depth in meters).

6 Data Extraction, Calculation, and Storage

6.1 Data Extraction: Subscribing to the ZED

The ZED publishes around a dozen topics3 of data, ranging from ‘‘color rectified

image’’ to ‘‘registered color point cloud.’’ Meaning that at any time Specifically,

we are interested in two topics:3Topics are named buses over which nodes exchange messages. Publishing a topic means sending

messages to nodes for subscription.

8

1. /zed/rgb/image_rect_color

2. /zed/depth/depth_registered 4

These two topics publish the RGB image and the depth map5 image from the ZED, re-

spectively. To retrieve this data, we must make a ROS subscriber to subscribe to the

same topics that the ZED is publishing. By doing this, we are directly accessing the exact

topics and extracting the necessary data without receiving any irrelevant data. A more

in-depth explanation of ROS can be found in two other Zebravision papers.6

6.2 Data Calculation: Screen To World

To actually use the data that the ZED collects, we need to translate the on-screen x and

y coordinates as well as the calculated depth into real world units. To do this, we use a

ScreenToWorld() function (details for which can be found in the 2017 Zebravision paper)

to convert the specific points on the screen to distances. By doing some trigonometry, we

can turn those distances into four values that are useful for the robot:

1. The distance to the cube from the robot. This is to know how far the robot has to go

to reach the cube.

2. The sideways alignment, or misalignment, of the cube in relation to the robot.

3. The vertical alignment, or misalignment, of the cube in relation to the robot.

4. The angle that the robot must turn to align exactly with the robot.

6.3 Data Storage: ROS Messages

ROS has an interface to transmit data using a system of messages. The newly processed

data now needs to be in a usable format and in an accessible place to other running code4The name of the topics in the code are: /zed goal/left/image rect color and

/zed goal/depth/depth registered. The aforementioned topics are the standard topic nomencla-

ture for the ZED. Zed ROS Wrapper5The depth map image depicts the depth of the pixels on the screen as different shades of gray. So

something far away would be extremely dark gray–or black–and something close up would be extremely

light gray–or white.6Zebracorns’ ROS for the 2017-18 season: Zebros 1.0

9

or a connected laptop. This is where we begin using ROS messages. These messages

are able to keep track of the data we get from the ZED and store it in specific fields.

This message can then be accessed in the code by a ROS subscriber, or on a laptop by

running the rostopic echo command in the console. Viewing the data on a connected

computer allows for easy, live access to the data as the program is running.

In our message, we had three specific fields:

1. Header header: This was used to separate one message from another and is mostly

used as a time stamp.

2. geometry_msgs/Point32[] location:7 It has an x (the aforementioned #1 in 6.2),

y (the aforementioned #2 in 6.2), and z (the aforementioned #3 in 6.2).

3. float64 angle: The aforementioned #4 in 6.2, measured in radians.

7 Testing With ZMS Files

7.1 Creating a Reader for ZMS Files

ZMS files are a specific type of file custom made by the Zebracorns to store data from the

ZED, allowing for storage of compressed frames of RGB data and the respective depth

data. To convert the ZMS files that the ZED captures during runtime back into usable

data, there must be a node that can accept a ZMS file and turn it into readable RGB and

depth data to display. The vid_reader_node does exactly this.

Through a series of steps, the ZMS file is:

1. Read in through the parameter located in the launch file.

2. Loaded into a ZMSIn, the designated object for the data.7http://www.ros.org/reps/rep-0105.html A depiction of the x, y, and z directions in a specific

frame. This field stores the real world location of the cube in meters relative to the robot.

http://docs.ros.org/api/geometry msgs/html/msg/Point32.html Raw message definition.

10

The new ZMSIn Object however, cannot be published directly as a ROS message due

to the fact that it is a custom data type. So, to be able to publish and subscribe to the

input, the ZMSIn is converted to the more appropriate OpenCV format. This is done with

a package named cv bridge through a series of steps:

1. Creating a CvImage that will be the final product.

2. Assigning the extracted RGB and depth values from the ZMSIn to its respective

OpenCV values. (RGB encoding: BGR8 – Depth encoding: TYPE 32FC1)8

3. Checking for the cv bridge exception.

• This is extremely important as it catches for a segmentation fault and means

that the supplied image encoding is incompatible or of the wrong type.

7.2 Choosing the Correct Topics and Launch File

Finally, the new OpenCV video is ready to be accessed. By publishing to the same

topics, the cube_detection node subscribes to (/zed_goal/left/image_rect_color,

/zed_goal/depth/depth_registered), the video reader is able to replace video pub-

lished directly by the ZED with the OpenCV video. This makes it so that cube_detection

runs on the test video. Now, ZMS video from match data can be used for testing of the

cube_detection code to refine it even more.

The launch file9 for the vid_reader_node can be separate from the file that launches the

cube_detection node, or in conjunction. This launch file must have two parameters and

an argument to function correctly. The parameters are the variable requirements for the

launch file to execute. The argument is the actual variable value that the function needs.

The pub_rate parameter is to establish a time gap between each publish from the node

to the topics. It must also have a file_path parameter to correctly receive the full path of

the file that will be inputted into the program.8BGR8 is a string encoding that represents a color image with a blue-green-red color order, TYPE 32FC1

is a string encoding that represents an ”image” that stores depth values in a matrix.9XML files used with the ROS function ”roslaunch” that start up specified nodes and pre-set parameters.

11

8 Summary

Vision this year, even off the robot, was a significant accomplishment. Integrating ROS

into the code base of the whole project this season turned out to be quite a daunting task

and yet it was completed. Data extracted from the recording and calculations would have

proved most useful in many different areas, like pathing to the switch and driving to and

intaking cubes. The way that ROS has been used this season will serve as a stepping

stone for future years, allowing the vision team to delve deeper into new and exciting

technologies.

12