Touch Seen Visual Board Game Platform Computer Vision Workshop, Fall 2010 Eldad Zinger, Oren Kishon, Ehud Fisher, Allon Freedman

Touch SeenVisual Board Game Platform

Computer Vision Workshop, Fall 2010Eldad Zinger, Oren Kishon, Ehud Fisher, Allon Freedman

Agenda• Overview• Architecture

– High Level– Server– Client– Camera Module

• Network Protocol• Game Plugin Platform• Client-Camera Interaction• Game move detection algorithm• Demonstration

2Thursday, April 20, 2023 Touch Seen

Overview


Touch Seen

• Application framework for playing board games over a computer network using a real physical board

• Moves are detected by a camera and transmitted over the network

• Provides a simple API for development of 3rd party games which can be uploaded to the platform in the form of plugins


Features

• Input using camera or any other standard input (keyboard, mouse etc.)

• Multiple clients connected simultaneously• Multiple games played simultaneously• All currently connected users displayed in

client• Option to challenge another user to a game• Textual chat between players


Features

• Open API for game development• Pluggable games• Simple text based protocol• Network based on ActiveMQ which supports

multi language clients


Architecture


High Level

8

Server

Client Client

Thursday, April 20, 2023 Touch Seen

Server

Two logical functions:• Messaging – distributing Move and Chat

messages between users• User & Game Management:

– User DB– User presence– Game state

Thursday, April 20, 2023 9Touch Seen

Server

• Messaging server based on Apache ActiveMQ• ActiveMQ – powerful open source Messaging

Server supporting JMS 1.1, J2EE 1.4, multiple cross language clients and protocols and morehttp://activemq.apache.org/

• An ActiveMQ server is run as an embedded broker within the Touch Seen Server


http://activemq.apache.org/

Server

JMS• A specification that describes a common way for Java

programs to create, send, receive and read distributed enterprise messages

• loosely coupled communication• Synchronous & Asynchronous messaging• Reliable delivery

– A message is guaranteed to be delivered once and only once• Outside the specification

– Security services– Management services


Server

JMS Messaging Domains• Point-to-Point (PTP)

Message Queues - each message has only one consumer

• Publish-Subscribe systems Message Topics - each message has multiple consumers


Server

JMS Message Consumption• Synchronous

– A subscriber or a receiver explicitly fetches the message from the destination by calling the receive method

– The receive method can block until a message arrives or can time out if a message does not arrive within a specified time limit

• Asynchronous– A client can register a message listener with a consumer– Whenever a message arrives at the destination, the JMS provider

delivers the message by calling the listener's onMessage() method

• All messages in Touch Seen use the asynchronous method


Server

JMS Messaging Domains – Queues

Queuesends

MsgMsg

acknowledges

consumes

MsgMsg

Client1 Client2


Server

JMS Messaging Domains – Topics

publishessubscribes

subscribes

MsgMsg

delivers

delivers

Client1

Client2

Client3

Topi

c


Server

JMS Model

ConnectionConnection

creates creates

creates

MsgMsg

receives fromsends to

ConnectionFactory

ConnectionFactory

MessageConsumerMessage

ConsumerSessionSessionMessageProducerMessageProducer

creates

Destination DestinationThursday, April 20, 2023 16Touch Seen

Server – Class Diagram

GamesUsers


ServerThreads• ServerController & NetworkController each operate

in separate threads• This ensures that server operations never block the

network thread so it can always be available for sending/receiving messages

Management• The server has no UI• Management is done using JMX (e.g. with jconsole)

utilizing both APIs provided by ActiveMQ and custom API’s designed specifically for this server


Client• Strict modular design based on MVC• Comprised of several independent modules• Each module has a controller class• ClientController acts as central controller for

entire application• All communication to and from a module is done

through its controller• The central controller connects to each of the

module controller (module controllers don’t interact directly)


Client

• The advantages of this design:– Modules are entirely decoupled from each other– Each module can be developed and tested

independently– Communication between modules is done in one

place – easy to maintain and debug– Modules can easily be replaced without affecting

the rest of the application – only the central controller “knows” all the modules


Client – Class Diagram


Threads

• The ClientController, NetworkController and CameraController all operate in separate threads

• This ensures that the Network and Camera threads are never blocked by other modules within the client

• All Objects in the View operate within Swing’s EventDispatcher thread

• The ClientController and Model also operate within the EventDispatcher thread to avoid synchronization problems with the View


23

Network Protocols


Logi

n Q

ueue

Login

usernamepassword

success

notify online

request players

players

Play

ers

Requ

est

Que

ue

Play

ers

Topi

c


Chal

leng

e Q

ueue

Game

challenge

accept

notify

challenge

accept

notify

Play

ers

Topi

c

New Game


Gam

e To

pic

Game

move

move

move

Game PlayingG

ame

Chat

Top

ic message

message

message


Gam

e To

pic

Game

quit

quit

quit

Game EndPl

ayer

s To

pic

notify online


Logo

ut Q

ueue

Logout

username

notify offline


29

Game Plugin Platform


Game SDK

• Games are developed using a provided SDK• Developers must implement only two abstract

classes:– GameInfo

• Flyweight which acts as a factory for creating instances of the game

• Provides common properties: name, icon etc.

– Game• Represents the game itself


Game SDK

• The server and client controllers interact with the game using the Game class

• It’s recommended (although optional) that this class include the game’s logic

• Game must provide a JPanel which acts as the game’s UI


Game Installation

• A game should be packaged as a .jar file and placed within the lib/games folder of the client and server

• A configuration file must be edited with the fully qualified name of the GameInfo class

• At startup, the client and server use a custom ClassLoader to load all the games dynamically

• A future release could include automatic deployment of new games to clients and installation/uninstallation without restart

• A simple (yet incomplete) example of Chess is included


33

Client-Camera Interaction


Client-Camera Interaction• The Touch Seen desktop client is written in Java• The camera module is based on OpenCV and written in C• Interaction between the two is done via a local TCP socket• The protocol is a simple text based protocol which simplifies debugging• When the user starts a game, the client starts the camera process and

waits for a socket connection for a specified timeout (externally configurable)

• If the initialization sequence completes successfully the camera can be used to play

• Alternative input devices (e.g. a mouse) can be used if errors occur during the initialization sequence or if moves aren’t detected successfully


35

Game Move Detection Algorithm


Detection algorithm

Image subtraction:


Detection algorithm (1)• Initialization:

– Detect chessboard– Find prospective transformation (3 x 3): rectify– Rotate coords to user’s point of view

• For each new image captured, do:– Apply prospective transformation (rectify)– Subtract image from previous image – to detect who

moved / captured by opponent player– Subtract image from empty board image – to detect who

actually cleared his location


Detection algorithm (2) Dealing with noise

• Noise: intensity only gets higher values• Filter1: image is a MIN of 20 frame captures• Filter2: Time-adaptive noise threshold:

– Capture two images each time, 1 sec apart– Subtract to get noise-only image– Calculate average value for square– Find max among squares (empirically ~5)– Thresh = max + const (=2, heuristics)


Detection algorithm (3) Square change detection

• Less sensitive change detection:– Among squares who passed threshold, take max.– <Used for move detection>

• More sensitive change detection:– For each square, sum value of pixels, regardless of

threshold. Take Max.– <Used for finding captured piece>


Detection algorithm (4) Evolution of development

• At first – simple subtraction algorithm (gray): hard-coded thresh, one background image

• Use RGB space – track each piece by its color. Each piece physically had a unique color

• Use HSV space – not good when dealing with B&W backgrounds

• Back to gray – smarter algorithm: min filter, multiple backgrounds, adaptive threshold


DemonstrationIn addition to the live demonstration we have provided the following documentation:•System Requirements Specification describing planned features and architecture•This presentation which provides the final design implemented•User Manual•Complete Javadoc for the Client, Server and Game Development SDK including UML diagrams (embedded within the Javadoc)•Fully documented code•Source code for Client, Server, SDK, Camera module and example game (Chess)•Installation instructions


Thank you…


Documents

Touch Seen Visual Board Game Platform Computer Vision Workshop, Fall 2010 Eldad Zinger, Oren Kishon, Ehud Fisher, Allon Freedman