HumanHuman InteractionInteraction Library Library

In partial fulfillments of the requirements for the In partial fulfillments of the requirements for the course of laboratory of computer graphics 2008course of laboratory of computer graphics 2008

Magliocchetti Daniele - 125162

ObjectivesObjectives• Build a new java library that allows the user to interact

with applications through non conventional devices, like:– Nintendo Wii® Controller (WiiMote)– Microphone– Web Cam

• Build the library as an API that can be easily reused• Integrate the library inside the WorldWind SDK

Capabilities Description: Finger TrackingCapabilities Description: Finger TrackingGesture recognition with the WiiMote IR camera and a set of IR emitters or reflective markers. The system should distinguish between two modes:

• Navigation Mode (up to 2 fingers):- Pan Action (1 finger) : default DRAG operation

- Zoom Action (2 fingers) : ZOOM IN ZOOM OUT

- Rotation Action (2 fingers) : CLOCKWISE COUNTERCLOCKWISE

- Look Action (2 fingers) : LOOK UP LOOK DOWN

• Edit Mode (3 or more fingers):- Move Cursor (1 or 3 fingers) :

- Single Click (1 or 3 fingers) :

- Double Click (1 or 3 fingers) :

- Drag Action (1 or 3 fingers) :

- Right Click (1 or 3 fingers) :

Speech Recognition of a small set of navigation and editing commands through the system microphone:

Speech RecognitionSpeech Recognition

Command Description Command DescriptionLeft Moves to the West direction Stop Stop all the movementsRight Moves to the East direction Edit Mode Switch to the edit modeLook Up Moves to the North direction Navigation Mode Switch to the navigation modeLook Down Moves to the South direction Single Click Notifies a single left click to WWJTurn Right Rotates the globe clockwise Double Click Notifies a double left click to WWJTurn Left Rotates the globe counterclockwise Right Click Notifies a single right click to WWJZoom In Zoom in Mouse Drag Notifies a drag to WWJZoom Out Zoom out

Head TrackingHead TrackingHead Tracking with a webcam that allow the user to modify the perspective of the application, like it is watching through a window:

Command DescriptionHead Right Moves the globe to the East directionHead Left Moves the globe to the West directionHead Up Moves the globe to the North directionHead Down Moves the globe to the South directionHead Near the Screen Zoom inHead Far from the Screen Zoom out

GesturesInteraction

Voice Interraction

Head MovementsInteraction

«extends»

Navigation Mode

Edit Mode«extends»

«extends»

CapabilitySelection

Use CaseUse Case

User

RequirementsRequirements ClassificationClassification

SystemSystem ArchitectureArchitecture• Based on the following patterns:

– Singleton: ensures that a class has only one instance and provides a global point of access to it

– Event-Listener:the system calls all the defined external listeners each time an event occurs. Allows a complete separation from third party applications, the ideal solution for an API.

• Organized in the following packages:- humanInteraction: with the main class of the system- humanInteraction.connection: WiiMote and network connection classes- humanInteraction.core.events: the event wrapper classes notified when an event

occurs- humanInteraction.core.listeners.ext: the listener interfaces that should be

implemented by the third party application to handle the incoming events from the system- humanInteraction.core.listeners.mote: the listener classes to handle the MoteJ

library- humanInteraction.core.queue: a set of utility classes for the concurrency- humanInteraction.core.fingerTracking: the classes for the gesture recognition

process- humanInteraction.core.headTracking: the classes for the head tracking process- humanInteraction.core.speechRecognition: the classes for the speech recognition

process

Bluetooth Dongle

Nintendo Wii ® Device

ConnectionManager

HumanInteraction Manager

HumanInteraction Library

Human Interaction Library Architecture

Sphinx4 java Library

Microphone Device

Speech Recognition Library

OpenCV Library

Webcam Device

Face Recognition Library

Face Recognition Client

Class /Component

ThreadLibrary Set

Device

Reference

Queue

Connection

Motej Library BlueCove Bluetooth

Stack

Wii Connection Libraries

Socket

Event Notification

WorldWind IntegrationWorldWind Integration• Extension of AWTInputHandler and OrbitViewInputBroker

into the following two classes:– WiiAWTInputHandler: implements all the event listeners of the human

interaction library in addition to the mouse and key event listeners;– WiiOrbitViewInputBroker: extended with some concurrency workarounds

• Each library event is mapped into WorldWind mouse or key events

• Addition two listeners classes and a layer to the WorldWind SDK (WiiMonitorLayer) to show on the screen the position of the fingers and the detected voice commands.

• Changes into the WWJ configuration file to load the layer and the WiiAWTInputHandler class

• Changes to the CGFrame class to load the library

Bugs and limitationsBugs and limitations• WiiMote discovery only short after the application start and if they are in connection

mode (button 1 and 2 pressed)• The library can handle only one WiiMote at a time• While dragging, sometimes the sdk generates some inconsistencies that lead to an

incorrect positioning of the globe (flipping and turning). We were not able to solve this problem, neither to detect it, even after a discussion on the WorldWind forum with the developers of the library.

• The Speech Recognition accuracy depends from the quality of the microphone and the noise of the environment. Each machine, microrphone or environment change, requires the reconfiguration of the hi.config.xml file;

• As a general rule for a good speech recognition, it’s preferable to speak slowly especially with multiwords (like “turn left”), and to pronounce the command “stop” with a long “o”.

• The library does not recognize combinations of buttons• The visibility of some WorldWind classes has been changed from protected to public• The head tracking client sets always the camera to the maximal resolution. With

webcams with a resolution greater than 640x480 pixel and slow cpus, the detection can be really slow. In this case, it’s preferable to set higher values for cvSizeHorizontal and cvSizeVertical inside the htSettings.cfg file.

• If there are multiple webcams, only the first installed will be used

ReferencesReferences• http://motej.sourceforge.net : The WiiMote java api official site• http://bluecove.sourceforge.net/ : The java bluetooth stack official site• http://www.wiili.org/index.php/Wiimote : A WiiMote hardware description• http://www.youtube.com/watch?v=0awjPUkBXOU : “Tracking Your Fingers with the

Wiimote” by Johnny Chung Lee• http://sourceforge.net/projects/opencv/ : The OpenCV official site• http://www.youtube.com/watch?v=Jd3-eiid-Uw : “Head tracking with the Wiimote” by

Johnny Chung Lee• http://cmusphinx.sourceforge.net/sphinx4/ : The Sphinx official site: “A speech

recognizer written entirely in the JavaTM programming language”• http://worldwind.arc.nasa.gov/java/index.html : The WorldWind official site

AcronymsAcronyms• WWJ: WorldWind Java SDK• BT: Bluetooth• IR: Infrared• JNI: The Java Native Interface• BNF: Backus-Naur Form