Immersive 3D Environment Using Kinect and Voice Commands

Utilizing Kinect control for a more immersive interaction

with 3D environments

Supervised byDr. Ammar Joukhadar Dr. Noor Shaker Eng. Mohammad Shaker

Designed, Implemented and Tested byMohammed Saed Haj Ali Marah Halawa Kinda Tarboush

F.I.T.E of Damascus, Syria – AI Department 2014

Content

• Motivation

• The Problem with Today’s Games

• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through

in-game Interactions• Hand Gesture • Voice Command

• Our Application

• Future Perspectives

• Demo

Content

• Motivation

• The Problem with Today’s Games

• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through

in-game Interactions• Hand Gesture • Voice Command

• Our Application

• Future Perspectives

• Demo

“Why using astylus when Godgave us tenfingers!” Steve Jobs when

introducingthe first iPhone, 2010

Good design looks great, yes,

Good design looks great, yes, but

smell great and sound great?

Good design looks great, yes, but

why shouldn't it also feel great,

smell great and sound great?

Good design looks great, yes, but

why shouldn't it also feel great,

Jinsop Lee

“Design for all 5 senses”

The Concept of

Content

• Motivation

• The Problem with Today’s Games

• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through

in-game Interactions• Hand Gesture • Voice Command

• Our Application

• Future Perspectives

• Demo

The player mostly has only one method to interact with the game (most games has the

same playing peripherals) like joystick or mouse

Most games don’t let the player to create their own content\ worlds and share it with others.

For most games there’s no direct interaction between the game and the player (only one way

from the player to the game)

For most games there’s no direct interaction between the game and the player (only one way

from the player to the game)

A Game For You to Use Your

Senses

Content

• Motivation

• The Problem with Today’s Games

• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through

in-game Interactions• Hand Gesture • Voice Command

• Our Application

• Future Perspectives

• Demo

Sketching a Level

Sketch Your Level

Using Kinect

Interaction with the environment

Hand Gesture

Interaction with the environment

Voice Command

Our Game

Put here a picture or a short video of the game. Just so that people know what are you talking about

The Immersion Process

Player Sketch

Generating basic Level

Canvas

Allow Edit and

Generation of Content

The Immersion Process

Player Sketch

Generating basic Level

Canvas

Allow Edit and

Generation of Content

The Immersion Process

Player Sketch

Generating basic Level

Canvas

Allow Edit and

Generation of Content

Sketching a Level

Put here a picture of a sketch and a picture of the height map of the same sketch

The Immersion Process

Player Sketch

Generating basic Level

Canvas

Allow Edit and

Generation of Content

Sketch to Level Conversion

Talk about the conversion from sketch to level and the cuboid algorithm you did (with picture)

The Immersion Process

Player Sketch

Generating basic Level

Canvas

Allow Edit and

Generation of Content

Hand Gesture

Interaction with

Dynamic Hand

Gesture

Interaction with

Static Hand

Gesture

Dynamic Hand Gesture

• Animated gestures

Static Hand Gesture

• Comparison with reference images• Using machine learning• Rule based system

SelectRotateDelete Move

The gestures that we will

use

Hand gesture in our application

Dynamic Hand

Gesture

Static Hand

Gesture

Hand gesture in our application

Dynamic Hand Gesture

Discrete Markov

Processes

Static Hand Gesture

Rule based system

Hand gesture in our application

Interaction via Voice Commands

Voice Command

• Audio to text conversion

Voice Command

• Speech Recognition Hidden Markov Model

Voice Command

• Words Meaning Conceptual Grammar

Content

• Motivation

• The Problem with Today’s Games

• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through

in-game Interactions• Hand Gesture • Voice Command

• Our Application

• Future Perspectives

• Demo

Our Application

Generate 3D environment and edit it using most of your senses!

Content

• Motivation

• The Problem with Today’s Games

• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through

in-game Interactions• Hand Gesture • Voice Command

• Our Application

• Future Perspectives

• Demo

Future Perspective

• Enhance the social aspect of the game• Multiple player playing at the same time at

the same level

• Assess the game immersions metrics

• Allow for more immersive and enhanced gestures

• Analyze the applicability of the approach on other hardware (like Oculus Rift)

• Analyze the applicability of the approach on other game genres

Content

• Motivation

• The Problem with Today’s Games

• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through

in-game Interactions• Hand Gesture • Voice Command

• Our Application

• Future Perspectives

• Demo

Thanx for listening!