Final Year Project 2004/2005Final Year Project 2004/2005LYU0402LYU0402

Augmented Reality Table for Augmented Reality Table for Interactive Card GamesInteractive Card Games

Supervisor: Prof Michael LyuSupervisor: Prof Michael Lyu

Presented by: Kevin Chow, Albert LamPresented by: Kevin Chow, Albert Lam

OverviewOverview

IntroductionIntroduction ObjectiveObjective Trading Card GameTrading Card Game Architecture of ARTArchitecture of ART Implementation in First TermImplementation in First Term ExperimentExperiment ConclusionConclusion Future WorkFuture Work

IntroductionIntroduction

What is What is Augmented RealityAugmented Reality??

• Mixed Reality combines the content from Mixed Reality combines the content from the real world with virtual imaginarythe real world with virtual imaginary

• Augmented Reality is a subset of this Augmented Reality is a subset of this where virtual content is overlaid into real where virtual content is overlaid into real objects of the worldobjects of the world

Introduction (Con’t)Introduction (Con’t)

• Paul Milgram characterized Mixed Paul Milgram characterized Mixed Reality interfaces on his “Reality-Reality interfaces on his “Reality-Virtuality Continuum” in 1994Virtuality Continuum” in 1994

Mixed Reality (MR)

VirtualEnvironment

Augmented Virtuality (AV)

AugmentedReality (AR)

RealEnvironment

Reality-Virtuality (RV) Continuum

Introduction (Con’t)Introduction (Con’t)

An AR system has the following three An AR system has the following three main characteristics:main characteristics:• Combines real and virtual objects in a Combines real and virtual objects in a

real environmentreal environment• Runs interactively, and in real timeRuns interactively, and in real time• Registers virtual objects onto the real Registers virtual objects onto the real

worldworld

Introduction (Con’t)Introduction (Con’t)

ART stands for ART stands for Augmented Reality TableAugmented Reality Table• User can play Card Games on the ART.User can play Card Games on the ART.

• ART can enhanceART can enhance the visual effect the visual effect during the playingduring the playing of the card game. of the card game.

ObjectiveObjective

Develop a generic Interactive ART Develop a generic Interactive ART Trading Card Game application. Trading Card Game application.

Enhance the traditional card games:Enhance the traditional card games:• Visual effectVisual effect• Sound effectSound effect• Complex calculationComplex calculation

Trading Card GameTrading Card Game

Trading Card GamesTrading Card Games• Two players play a match.Two players play a match.• Players play cards to fight with each Players play cards to fight with each

other. It includes summoning monster, other. It includes summoning monster, casting magic, or setting traps, etc.casting magic, or setting traps, etc.

• We choose We choose “YU-GI-OH”“YU-GI-OH” as our as our implementationimplementation

Trading Card Game (Con’t)Trading Card Game (Con’t)

Trading card game includes five kinds of Trading card game includes five kinds of actionsactions• RestoreRestore• Draw cardsDraw cards• Play cardsPlay cards• Attack/ChallengeAttack/Challenge• Discard cardsDiscard cards

An Example - YU-GI-OHAn Example - YU-GI-OH In YU-GI-OH, Different information are In YU-GI-OH, Different information are

stored on the card.stored on the card.

An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)

Game Mat of YU-GI-OHGame Mat of YU-GI-OH Different kinds of card must be put inside Different kinds of card must be put inside

particular card zonesparticular card zones

An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)

LP : 2000 LP : 2000

ATK : 500 ATK : 1200

1200 – 500= 700-700

LP : 1300

An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)

LP : 1300 LP : 2000

DEF : 1300 ATK : 1200

1300 – 1200= 100-100

LP : 1900

ArchitectureArchitecture

Hardware SetupHardware Setup• Plasma monitorPlasma monitor• Overhead cameraOverhead camera

System architectureSystem architecture• Augmented Reality Perception Augmented Reality Perception • Game CoreGame Core• DatabaseDatabase• Game Enhancement Game Enhancement

Hardware SetupHardware SetupOverhead mounted camera

Analyze the video captured from the camera, and generate output to the screen

The plasma monitor for playing trading card games

Plasma monitor act as a table

Hardware Setup (Con’t)Hardware Setup (Con’t)

ComputerComputer• processes perceiving imageprocesses perceiving image• generates visual and audio enhancementgenerates visual and audio enhancement• controls game flowcontrols game flow

Plasma MonitorPlasma Monitor..• acts as the Game Tableacts as the Game Table• displays computer-generated scenedisplays computer-generated scene

Overhead mounted cameraOverhead mounted camera• captures the cards and the screen of the captures the cards and the screen of the

plasma.plasma.• only input of the systemonly input of the system

System ArchitectureSystem Architecture

Game Core ModuleGame Core Module Perception ModulePerception Module Database ModuleDatabase Module Game Enhancement ModuleGame Enhancement Module

System Architecture (Con’t)System Architecture (Con’t)

ART Card GameGeneric Card Game Database

Perception

CommandDetector

Calibration

Card DetectorGame

Enhancement

Video Decoder

Card Locator

Calibration

Card Recognizer

3D Animation

Input Analyzer

Sound Effect

ART Card GameCore

Game RulesDatabase

DisplayCamera

Card Information

Input Information

Environment Information Output Generator

Sound

Card ImageDatabase

CardsDatabase

System Architecture (Con’t)System Architecture (Con’t)

Microsoft DirectX SDKMicrosoft DirectX SDK• A set of low-level APIs for high-A set of low-level APIs for high-

performance multimedia applicationsperformance multimedia applications• directly access the hardwaredirectly access the hardware• provides device independent through HALprovides device independent through HAL• We will use DirectShow, Direct3D and We will use DirectShow, Direct3D and

DirectSound in our implementationDirectSound in our implementation

ImplementationImplementation

Perception ModulePerception Module• CalibrationCalibration• Search window locatorSearch window locator• Card locatorCard locator• Card RecognizerCard Recognizer

Database ModuleDatabase Module• Identify card typeIdentify card type• Image retrievalImage retrieval• Image matchingImage matching

Perception ModulePerception Module

Read and process the raw video Read and process the raw video perceived from the cameraperceived from the camera

Detect input card and commandDetect input card and command Some assumptionsSome assumptions

• fixed camera, fixed tablefixed camera, fixed table• camera is approximately right above the camera is approximately right above the

tabletable• cards can only be placed in predefined cards can only be placed in predefined

region called card zonesregion called card zones

CalibrationCalibration

Why?Why?• Captured image Captured image

varies with varies with environmental environmental lighting conditionlighting condition

• Take advantage of Take advantage of the assumptions the assumptions mademade

Calibration (Con’t)Calibration (Con’t)

How?How?• Set the card zone positionSet the card zone position• Set the card area threshold for each Set the card area threshold for each

card zonecard zone• Calibrate the colorsCalibrate the colors• Calibrate card type colorsCalibrate card type colors

Search Windows LocatorSearch Windows Locator

Find region to search for cardsFind region to search for cards The screen is partitioned into some fixed The screen is partitioned into some fixed

small search windowssmall search windows Search windows which just stop changing Search windows which just stop changing

will be processedwill be processed

Search Windows Locator Search Windows Locator (Con’t)(Con’t)

Located search windows Located search windows that have changesthat have changes• Image differentImage different

• Pixel changed ifPixel changed if

• A Search window is saidA Search window is said to be changed to be changed ifif

Activate search window fromActivate search window from“change” to “unchange” state“change” to “unchange” state

),(),(),( yxIyxIyxI previouscurrentdiff

1),( yxI diff

2),( yxI diff

Card LocatorCard Locator

Locate four corner card positions Locate four corner card positions given a search windowgiven a search window• Apply Apply Canny AlgorithmCanny Algorithm• Extract ContoursExtract Contours• Search for contours whichSearch for contours which

contain 4 pointscontain 4 points contain nearly right contain nearly right

angle cornersangle corners have area within ahave area within a

certain thresholdcertain threshold

Card Locator (Con’t)Card Locator (Con’t)

Canny AlgorithmCanny Algorithm• Remove noise by Remove noise by

Gaussian FilterGaussian Filter• Compute edge strengthCompute edge strength

and edge direction by and edge direction by Sobel OperatorSobel Operator

• Apply Apply Nonmaximum Nonmaximum Suppression Suppression to trace to trace edges edges

• Use Use Hystersis Hystersis to eliminate streakingto eliminate streaking

Card RecognizerCard Recognizer

Card orientationCard orientation• Compare the lengths of the four edgesCompare the lengths of the four edges• Compare color difference between top Compare color difference between top

and bottom parts of the cardand bottom parts of the card

Card Recognizer (Con’t)Card Recognizer (Con’t)

Get Card ImageGet Card Image• Extract the undistorted card image from Extract the undistorted card image from

a distorted imagea distorted image

Card Recognizer (Con’t)Card Recognizer (Con’t)

• Pixel coordinate transformationPixel coordinate transformation Map a point (x,y) on the distorted image to a Map a point (x,y) on the distorted image to a

point (x’,y’) on the undistorted imagepoint (x’,y’) on the undistorted image

(x0,y0)

(x1,y1)

(x2,y2)

(x3,y3)

. (x,y)h

w

. (x’,y’)

3210

3210

))(1())(()1)(()1)(1(

))(1())(()1)(()1)(1(

yh

y

w

xy

h

y

w

xy

h

y

w

xy

h

y

w

xy

xh

y

w

xx

h

y

w

xx

h

y

w

xx

h

y

w

xx

Card Recognizer (Con’t)Card Recognizer (Con’t)

• Brightness interpolationBrightness interpolation Assign brightness value to theAssign brightness value to the

point (x,y) point (x,y) Apply Apply Bilinear InterpolationBilinear Interpolation

)(),(

)(),(

)1,1())((

)1,())(1(

),1()1)((

),()1)(1(),(

yfloorybyfloork

xfloorxaxfloorh

khIba

khIba

khIba

khIbayxfbilinear

Card Recognizer (Con’t)Card Recognizer (Con’t)

Identify the card uniquelyIdentify the card uniquely• Query the Image Database with the card Query the Image Database with the card

image for unique card Idimage for unique card Id• Query the Card Database with the card Query the Card Database with the card

Id for card information and detailsId for card information and details

Database ModuleDatabase Module

Contains all game informationContains all game information Identify card image by 3 steps:Identify card image by 3 steps:

• Identify card typeIdentify card type Classify card by card typeClassify card by card type

• Image retrievalImage retrieval Retrieve several similar candidate cardsRetrieve several similar candidate cards

• Image matchingImage matching Find the best match cardFind the best match card

Identify Card TypeIdentify Card Type

• Compare the background color of the Compare the background color of the card to the calibrated colorcard to the calibrated color

Image RetrievalImage Retrieval

Retrieve similar images of same card Retrieve similar images of same card type.type.

Use color-based retrieval method due Use color-based retrieval method due to low resolutionto low resolution

Color Histogram method select cards Color Histogram method select cards with largest color histogram intectionwith largest color histogram intection

Image Retrieval (Con’t)Image Retrieval (Con’t)

Color histogram intersectionColor histogram intersection

n

jD

n

jDQ

DQH

jHH

jIHjIH

IID

1

1

),(

)),(),,(min(

),(

Select the best matched Select the best matched card from the candidates card from the candidates

• Split the captured image Split the captured image into 4 channelsinto 4 channels

• Compare only inner imageCompare only inner image• Compare the different Compare the different

channels to the channels to the database separatelydatabase separately

Image MatchingImage Matching

Image MatchingImage Matching

• Find the pixel difference byFind the pixel difference by

• Reject the image if the pixel difference Reject the image if the pixel difference is larger than the thresholdis larger than the threshold

• Accept the image with minimum pixel Accept the image with minimum pixel differencedifference

w

i

h

jDQDQP jiIjiIIID

1 1

2)),(),((),(

ExperimentExperiment

Match 15 cards to a set of different Match 15 cards to a set of different cards with threshold set to 1500cards with threshold set to 1500

Here, we have chosen results of Here, we have chosen results of three cards for discussionthree cards for discussion

Experimental ResultExperimental Result

Overall Pixel Difference of Card 14

0

1000

2000

3000

4000

5000

6000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Card ID

Dif

fere

nce

H value R vaule G vaule B value

Experimental Result (Con’t)Experimental Result (Con’t)Overall Pixel Difference of Card 13

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Card ID

Dif

fere

nce

H value R vaule G vaule B value

Experimental Result (Con’t)Experimental Result (Con’t)Overall Pixel Difference of Card 9

0

1000

2000

3000

4000

5000

6000

7000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Card ID

Dif

fere

nce

H value R vaule G vaule B value

ConclusionConclusion

We have implemented a simplified We have implemented a simplified version of the ART systemversion of the ART system

Concerning low resolution of the Concerning low resolution of the image, we have develop an image, we have develop an algorithm to recognize card with high algorithm to recognize card with high accuracy but less efficiencyaccuracy but less efficiency

Future WorkFuture Work

Color CalibrationColor Calibration Develop an efficient card recognition Develop an efficient card recognition

algorithm while retaining high algorithm while retaining high accuracyaccuracy

Game EnhancementGame Enhancement Interactive input detectorInteractive input detector

End of PresentationEnd of Presentation

Thank you very muchThank you very much

Q & A SessionQ & A Session