Using Tweek to Create Graphical User Interfaces in Virtual Reality Patrick Hartling IEEE VR 2003

Using Tweek to Create Using Tweek to Create Graphical User Interfaces inGraphical User Interfaces in

Virtual RealityVirtual Reality

Patrick HartlingPatrick Hartling

IEEE VR 2003IEEE VR 2003

VR Juggler — http://www.vrjuggler.org/

OverviewOverview

Problem description

Goals for Tweek software

Design overview

Implementation overview

Future work


Problem DescriptionProblem Description

Interaction in VR is primarily spatialNot all interactions map well to this method• Alphanumeric input• Fine-grained interaction• List selection

Per-application custom interfaces• Map abilities of specific devices to interaction• Typically use 3D input devices

Why abandon effective 2D interaction techniques in 3D space?


MotivationMotivation

Why have GUIs in virtual reality?• GUIs are everywhere in the computing world• Most computer users understand how to use

a GUI• Why not have GUIs in VR?


Issues with GUIs inIssues with GUIs inVirtual Reality (1/2)Virtual Reality (1/2)

Positives• Typically more flexible and more extensible

than other input devices–More input methods means less overloading of

button presses, gestures, etc.

• More portable between VR systems than hardware input devices

• Familiarity from desktop experience


Issues with GUIs inIssues with GUIs inVirtual Reality (2/2)Virtual Reality (2/2)

Negatives• No direct physical correspondence

– In VR, a user could grab an object with his/her hand or use a GUI

– No GUI in real world for moving the object

• Remote users may not see GUI [Park00]• Use of the interface may be difficult

– Collision detection– Placement

• Immersive interfaces have to re-invent GUI technology


Previous Work (1/3)Previous Work (1/3)

Many areas of previous work• Widget sets for VR• PDAs as input devices• Floating menus and heads-up displays

Virtual User Interface (VUI)• Originally developed at Iowa State University

[Heath98]• 3D immersive user interface• May be an open source release from Fuel Tech



Java Interface to Virtual Environment (JAIVE)• Used Java interface on handheld computer

[Hill00]• Communicated with C++ applications• Interface could be extended from C++ using

JAIVE protocol



PDA in a CAVE™ system• Kent Watsen, Naval Postgraduate School• Experimented with use of Palm PDA in CAVE

[Watsen99]

3DI Group• Doug Bowman, Virginia Tech• Investigating differences in system interaction

techniques between VE displays• Active on 3DUI mailing list


TweekTweek

Framework for the development of cross-platform, cross-language, scalable GUIs• GUIs communicate with remote applications• GUIs can be use on the desktop, on a PDA, or in an

immersive VR environment

Features• GUI panels are dynamically loaded components

– Panels use standard 2D GUI “widgets”

• Dynamic interface update and extension• Multi-language capabilities


GoalsGoals

Allow GUI scalability• Same GUI runs on the desktop, a PDA, and a

projection-based VR system• Must provide an easily extendable framework for

cross-platform, cross-device GUIs

Interact with VR applications written in any programming language• Fully separate the GUI from the application

Reuse existing GUI technology• We do not want to re-invent scrollbars, menus, etc.

Avoid cluttering immersive space


Same GUI, Multiple Uses (1/2)Same GUI, Multiple Uses (1/2)

PDA in Projection System• Offers force feedback• High-resolution display– Back-lit display is needed

• Widely available– A person with a PDA can walk into the VR system

and get the GUI dynamically


Same GUI, Multiple Uses (2/2)Same GUI, Multiple Uses (2/2)

Desktop simulation• Standard GUI interface used to control VR

software in simulator mode

Immersive 3D space• VRJ VNC incorporates 2D GUI into 3D

visualization–Modeled after 3Dwm (www.3dwm.org) use of VNC

and other VR VNC viewers


Design Overview (1/3)Design Overview (1/3)

Distributed Model/View/Controller

Cube rotation Cube translation

Network Communication

Controller

ModelView



Observer design pattern• Simplified MVC: subject, observer• VR application is the subject

– Maintains user-defined state information– Per-application customizations

• GUI is the observer– Provides a view of subject’s state– GUI interactions manipulate application state

• Benefits– Distinct separation of application and GUI– Allows multiple GUIs to observe the same application



Two-way communication between application and GUI• Application can send data to the GUI

GUI flexibility• A GUI can load any component it needs to

make interaction more effective• Not limited to direct manipulation of

application state


Implementation OverviewImplementation Overview

Language tools• C++ VR applications• Java-based GUI composed of JavaBeans– Beans can be “pushed” to the GUI while the user

is in the virtual space– Dynamic extension of interface

• CORBA communication between C++ and Java


Example Tweek InterfaceExample Tweek Interface

• Multi-purpose tool– Navigation–Map visualization– Data visualization– Information

storage to spreadsheet-friendly format


VRJ VNC: Embedding 2D GUIs VRJ VNC: Embedding 2D GUIs in 3D Spacein 3D Space

• Embed any 2D GUI in 3D world

• Uses VNC• Can be added

to existing VR Juggler applications


VRJ VNC: Presentations in VRVRJ VNC: Presentations in VR• Bring

PowerPoint presentation into the 3D environment


VRJ VNC: Embedding TweekVRJ VNC: Embedding Tweek

• Tweek Java GUI is just another window on a desktop

• Set VNC desktop size to give the desired embedded frame size

• Interaction with the GUI is translated to the desktop via VNC


Future WorkFuture Work

• Use Java Applets to allow Tweek to be used from a web browser–Make use of a familiar interface– Easy access to application GUI for remote

demonstrations


Online InformationOnline Information3DUI• http://www.mic.atr.co.jp/~poup/3dui.html

Tweek• http://www.vrjuggler.org/tweek/• Contains publications, guides, and

programmer references


ReferencesReferences

• [Park00] Park, K. et al. “Lessons Learned from Employing Multiple Perspectives in a Collaborative Virtual Environment for Visualizing Scientific Data”. In Proceedings of ACM Conference on Collaborative Virtual Environments, San Francisco, California, September 10–12, 2000.

• [Heath98] Heath, D. “Virtual User Interface (VUI): A Windowing System for VR”. In Proceedings of 2nd Immersive Projection Technology Workshop, Ames, Iowa, May 11–12, 1998.

• [Hill00] Hill, L., Usability of 2D Palmtop Interaction Device in Immersive Virtual Environment. Master’s thesis, Iowa State University, Ames, IA, 2000.

• [Watsen99] Watsen, K. “A Handheld Computer as an Interaction Device to a Virtual Environment”, In Proceedings of 3rd Immersive Projection Technology Workshop, Stuttgart, Germany, May 10–11, 1999.

Documents

Using Tweek to Create Graphical User Interfaces in Virtual Reality Patrick Hartling IEEE VR 2003