Development of A cheap chess robot: Planning and Perception

A Cheap Chess Robot: Perception

Billy Okal, Oliver Dunkley, Andreas Nuchter

Automation and Machine Vision LabJacobs University Bremen

May 13, 2011

Introduction Methodology Experiments Conclusion and Future Work

Outline

1 IntroductionMotivationPrevious/Related WorkResearch Agenda

2 MethodologyFinding the chessboardDetecting ChangesDetermining Moves

3 ExperimentsFrom scene to 2D viewDetect Move and Query Chess Engine

4 Conclusion and Future Work

Billy Okal, Oliver Dunkley, Andreas Nuchter A Cheap Chess Robot: Perception

Introduction Methodology Experiments Conclusion and Future WorkMotivation Previous/Related Work Research Agenda

Motivation

What is Robot Chess?

Robot playing a game of chess against another robot or humanopponent.

We focus on the case of a Robot vs a Human

What is Robot Chess Perception?

Robot Perception generally enables the robot to perceive thechanges in the environment using its sensors and interpret sensoryinput to make decisions about its next actions. In our case, thesensor is a monocular camera, the environment is a chessgame and actions include moving the chess pieces on thechessboard.



Motivation


Robot playing a game of chess against another robot or humanopponent. We focus on the case of a Robot vs a Human





Motivation




Robot Perception generally enables the robot to perceive thechanges in the environment using its sensors and interpret sensoryinput to make decisions about its next actions.

In our case, thesensor is a monocular camera, the environment is a chessgame and actions include moving the chess pieces on thechessboard.



Motivation







Previous/Related Work

We highlight a few interesting/related cases;

The AAAI small scale manipulation(table top chess) challengeduring AAAI2010 and AAAI2011

Modified Chessboard [http://chiara-robot.org/]

Carefully chosen colors forpieces and board

Non-standard chessboards

Non standard sizes

Systems that involved mounting the camera directly above thechessboard as in [Urting:2003] and [Groen:1992]

A fews humanoids like REEM-A(30 dof, stereo-cam) by PalRobotics. etc


http://chiara-robot.org/








Non standard sizes












Non standard sizes












Non standard sizes












Non standard sizes












Non standard sizes






Research Agenda

Common Feature of Previous Work

Constraints imposed on the chessboard and pieces to simplify theproblem.Which of these constraints can we relax/get rid of?

Our Attempt

Allow for camera to move relative to chessboard (includingmoving out of the scene)

Use a un-modified chessboard

Use standard chess piece set (Staunton standard tournamentpieces No 5 - from Amazon)

All the above using cheap camera (a USB webcam) and basichardware (4 dof + 1 for gripper).



Research Agenda


Constraints imposed on the chessboard and pieces to simplify theproblem.

Which of these constraints can we relax/get rid of?

Our Attempt







Research Agenda



Our Attempt







Research Agenda



Our Attempt






Introduction Methodology Experiments Conclusion and Future WorkFinding the chessboard Detecting Changes Determining Moves

Methodology

Finding the chessboard in the scene

Detecting Changes on the chessboard (moves)

Decoding the changes to identify user moves

Generating new moves and transform these into free spaceposes for planning




Algorithm 1: Detect Board in the Scene

Data: Raw Scene Image (img) - 800x600Result: 4 Corners of the chessboardbegin

img edges ←− EdgeDetect(img);all lines ←− LineDetect(img edges);MergeCloseLines(all lines);bounding lines ←− FindBoundingLines(all lines);corners ←− ComputerIntersections(bounding lines);return corners;

end







end







end



Project the chessboard to a 2D view

Homography/Projective Transformation

Given a 2D point ~p = (x , y) on the image plane, find a matrix Hsuch that the point ~p′ = (x ′, y ′) given by ~p′ = H~p lies on a 2Dplane P2 of choice.

In homogenous coordinates we have;

~p = (a, b, c), s.t x =a

c, y =

b

cand c 6= 0 (1)

~p′ = (a′, b′, c ′), s.t x ′ =a′

c ′, y ′ =

b′

c ′and c ′ 6= 0 (2)

a′

b′

c ′

=

h1 h2 h3

h4 h5 h6

h7 h8 h9

abc

(3)

[Hartley:2000]Billy Okal, Oliver Dunkley, Andreas Nuchter A Cheap Chess Robot: Perception



We plugin all the correspondences for the four corners and arrive atthe following system;

a1 b1 1 0 0 0 −a1a′1 −b1a

′1

a2 b2 1 0 0 0 −a2a′2 −b2a

′2

a3 b3 1 0 0 0 −a3a′3 −b3a

′3

a4 b4 1 0 0 0 −a4a′4 −b4a

′4

0 0 0 a1 b1 1 −a1b′1 −b1b

′1

0 0 0 a2 b2 1 −a2b′2 −b2b

′2

0 0 0 a3 b3 1 −a3b′3 −b3b

′3

0 0 0 a4 b4 1 −a4b′4 −b4b

′4

h1

h2

h3

h4

h5

h6

h7

h8

=

a′1a′2a′3a′4b′1b′2b′3b′4

(4)

Which then can be solved using a number of well establishedalgorithms.

In our implementation we make use of a OpenCVmethods cv::getPerspectiveTransform andcv::warpPerspective.[Hartley:2000]




We plugin all the correspondences for the four corners and arrive atthe following system;

a1 b1 1 0 0 0 −a1a′1 −b1a

′1

a2 b2 1 0 0 0 −a2a′2 −b2a

′2

a3 b3 1 0 0 0 −a3a′3 −b3a

′3

a4 b4 1 0 0 0 −a4a′4 −b4a

′4

0 0 0 a1 b1 1 −a1b′1 −b1b

′1

0 0 0 a2 b2 1 −a2b′2 −b2b

′2

0 0 0 a3 b3 1 −a3b′3 −b3b

′3

0 0 0 a4 b4 1 −a4b′4 −b4b

′4

h1

h2

h3

h4

h5

h6

h7

h8

=

a′1a′2a′3a′4b′1b′2b′3b′4

(4)

Which then can be solved using a number of well establishedalgorithms. In our implementation we make use of a OpenCVmethods cv::getPerspectiveTransform andcv::warpPerspective.[Hartley:2000]



Detecting Changes on the chessboard(moves)

Locate blobs/change spots

1 Difference image (Given two imagesf (x , y) at time t1 and f ′(x ′, y ′) attime t2, difference isg(x , y) = |f − f ′|, [Farahat:2003]

2 Remove noise by erosion and dilation3 Distance Transform (label pixels with

distance to background)4 Find the pixel whose label is largest5 Flood fill the pixel above6 Repeat steps 3,4,5 until threshold

Map change spots to board changes.Use image dimensions and know chesssquare sizes to place changes on boardsquares. Generate change strings.



















Determining chess moves from the changes

Convert chessboard changes into moves

Compare every pair of changes say a2b3 against an 8× 8 statematrix and return a move stringRepeat over the whole change string

Query chess engine for moves based on detected moves

Using standard chess engine communication protocol (UCI -Universal Chess Interface).Communication handles via ROS messages

Convert suggested moves into poses in free space

Poses are defines as a pair of Position p = (x , y , z)T andOrientation q = (x , y , z ,w)T to allow for design freedom inplanning.Using chessboard size, we interpret a move string say a2b3

into respective pose say p = (24, 14, 2)Tq = (0, 0, 0, 1)T






















Introduction Methodology Experiments Conclusion and Future WorkFrom scene to 2D view Detect Move and Query Chess Engine

Find Board and project it to 2D View

[[Movie]]


Introduction Methodology Experiments Conclusion and Future WorkFrom scene to 2D view Detect Move and Query Chess Engine

Determine Move and query Chess Engine



Conclusion and Future Work

Conclusions

We have demonstrated that the constraint of using a modifiedchessboard and/or pieces can be relaxed without compromisingperformanceWe demonstrated on very minimalistic hardware, use of betterhardware should produce even better performanceCamera need not be fixed

Future Work

Investigate how the task could be solved multiple cameras(stereo vision)Adding other types of sensors (e.g. Kinect)Add object recognition modulesImprove the blob detection module



Conclusion and Future Work

Conclusions

We have demonstrated that the constraint of using a modifiedchessboard and/or pieces can be relaxed without compromisingperformanceWe demonstrated on very minimalistic hardware, use of betterhardware should produce even better performanceCamera need not be fixed

Future Work

Investigate how the task could be solved multiple cameras(stereo vision)Adding other types of sensors (e.g. Kinect)Add object recognition modulesImprove the blob detection module



The End

Thank you for your attention

Questions/Comments?

Special ThanksDAAD/PROFIN,

Automation Group MembersFor further information check

B.Okal and O.Dunkley, ”A Cheap Chess Robot: Planning and Perception”,Bachelor Thesis 2011.

Software Tools

http://www.ros.org

http://opencv.willowgarage.com

http://openrave.programmingvision.com


http://www.ros.org

http://opencv.willowgarage.com

http://openrave.programmingvision.com

Education

Development of A cheap chess robot: Planning and Perception