Using Commonsense Reasoning in Video Games

There is a problem facing the video game industry: 3D environments are getting dramatically more realistic, but they are not getting any easier for

artists to create

The Dramatic Improvement in Video Game Realism

The Dramatic Improvement in Video Game Realism - Hardware

(2003)(1998)

In addition to looking more like jet engines, graphics cards are improving at an incredible pace

The Dramatic Improvement in Video Game Realism - Hardware

TNT

TNT2

GeFo

rce2

GeFo

rce3

GeFo

rce4

GeFo

rceFX

GeFo

rceFX

680

0

Mil.

of Trian

gles

/Sec

NVIDIA is working under the philosophy of "Moore's Law Cubed"

The Dramatic Improvement in Video Game Realism - Software

(1991) (1994) (1996)

(1999) (2004)

Wolfenstein 3D by id DOOM by id Quake by id

Quake III by id DOOM 3 by id

The environments are more realistic (texture mapping, anti-aliasing, pixel and vertex shading, improved shadow generation, increased memory bandwidth, increased fill rate…)

And they are also more complicated

The Dramatic Improvement in Video Game Realism

The Dramatic Improvement in Video Game Realism – Complexity of Environments

(lots of stuff)

Morrowind by Bethesda Softworks, PC and Xbox

Morrowind took 3 years to develop in part because the massive game environment was generated by hand

Result: Increased Development Time

The Complexity of Level Editors

Editors to create 3D environments represent some of the most complex software applications ever created

The Complexity of Level Editors: A Sea of Direct Manipulation

The Morrowind Construction Set

The Complexity of Level Editors

Unfortunately, editors are not improving at a pace to compensate for the increased complexity of 3D

environments

A CS100 Solution

Some games automatically create their own environments, but the result is usually as simplistic as randomly

generating a maze

A New Solution

We are trying to use Common Sense Reasoning to automatically generate rich and immersive 3D

environments

Rationale

• Save development time and money• Create dynamic and “endless” worlds• Expand the capabilities of Common Sense

Reasoning

Rationale

Project Goals

Project Goals

• Automatically place objects in a 3D world.• Develop a set of common sense predicates that

accurately describes object placement.

Human Computer Interaction

Um, I want a table with a vase on it

(X,Y,Z)?

There is a disconnect between how people express object placement and how computers do. Common Sense knowledge will allow computers to render

environments without requiring specific instructions for every action

System Architecture

Level Editor with Common Sense

Plug-in File

Render the environment in the Morrowind Engine

Algorithms to convert predicates to XYZ coordinatesand intelligently place objects in a 3D world

Knowledge about objects

Natural Language StatementsNatural Language Statements

XML file with a listof objects

XML file with RDF predicates about Objects

(automatically generate)

Common Sense Predicates

Common Sense Predicates

• Develop a set of predicates that can describe object placement.

• E.g. On top of, Around, Beside, etc…

• How people describe object locations:– “the chairs around the table”– “the vase on top of the table”

Common Sense Predicates

Common Sense Predicates

On Top of Below

Center of Corner of

Edge of Around

Inside Beside

In front of Behind

Against

Common Sense Predicates

Dealing with Multiple Predicates

• Vase goes on top of a table.• Vase goes in the center of a table.• Place the vase on top of and in the center of a table.

Common Sense Predicates

Dealing with Fuzzy Predicates

• What does “edge of” mean?• How close to the edge? It depends on the object• E.g. plates around edge of a table vs. frosting around

edge of a cake• Requires more common sense about object

properties, like don’t want plates to fall off.

Our Proof of Concept

The system makes mistakes, this is true with all applications that use common sense reasoning

However, we have a working proof of concept

Example Scene

These objects were automatically placed using Common Sense Reasoning

Future Work

We are now exploring to what extent these algorithms can scale to more complicated

environments

http://agents.media.mit.edu/projects/videogame