Virtual Worlds: Input

Virtual Environments

• Importance of immersion, presence, interaction, engagement, multisensory

• Elements of the world: graphics, representation, visual and other senses, interface, navigation, manipulation, story

• Interface: input, output, computer interface

Virtual Environments as a Medium

Think of the following mediums and how they communicate: medium, kind of virtual world, final composition, how experienced

Painting, dance, music, written word, play, movie, video game, virtual environments

Interaction in the Virtual World: Overview

• User interface• Manipulation

– Methods– Properties– Selection– Operations

• Navigation– Wayfinding– Travel

• Interaction and communication with others– Sharing– Collaboration

VR Worlds: Input Overview

• User input:– 3D– Mouse, keyboard– Position tracking – Body tracking– Other physical input, including speech

• World input

User Interface Metaphors

• Familiar user interfaces: mobile phones, TV controls, remote controls, cars, video games, microwaves, doorways, cursor icons, desktop, touchscreens, notions of right, left and double clicking, mouseover, tweeting, texting

• Do you use familiar ones in a VR world? Has limits and shortcuts

User Interface Interactions

• Need to map user input to interactions

• Users must already know how to use a tool or must be able to learn

• Users bring experience and cultural knowledge

• Users need familiarity but also want to extend what is normally possible in the real world (has limitations)

User Interface Interactions (con’t)

• Viewpoint navigation (or travel) and object manipulation (and selection) are the main types of interaction

• Differences between the two:– User’s conceptual model (what moves)– Extent of interaction space (navigation can be

large, object manipulation often small)– Perceptual cues (navigation generally visual,

object manipulation uses haptics)

3D User Interfaces

• Operate in 3D space

• Natural: measure of how closely the actions in the virtual environment correspond to the actions in the real world

• Examples: Wii Remote, Sony Move, Microsoft Kinect

Bowman, et all Questions about 3D User Interfaces and Naturalism

• Are 3D inherently more natural than traditional?

• Is naturalism the ultimate goal for designers?

• Does naturalism lead to better performance, increased engagement, or better learning?

• When naturalism isn’t possible should the designer go for traditional user interfaces or interfaces with some degree of naturalism?

Issues Involving Natural User Interfaces

• For the tasks of navigation, selection and manipulation of object, how well do 3D natural user interfaces perform vs. magic techniques or some combination of them?

• Issues involving turning, tracking, selecting, manipulating, steering

• Natural interfaces sometimes out-perform and are sometimes inferior- depends on task and context; they need high level of accuracy and must be familiar

• Definition of natural?

• Lack of guidelines and standards for gestural interfaces

• Problems raised by Norman and Nielsen of visibility, feedback, discoverability, consistency, scalability, reliability

Bowman et al table from Comm. of the ACM, 2012

Problems with Natural User Interfaces

• How natural are they?

• Who decides what the gestures or other actions will be?

• Cultural differences

Input: Position Trackers• Position tracking – gives location and orientation

of user and/or parts of user and feeds it to the computer - concerned with 6 degrees of freedom: x,y,z position and orientation, generally given by pitch, roll and yaw (angles with orthogonal axes)

• General issues of accuracy (walking in a landscape vs surgery), latency (conducting an orchestra vs walking), interference from surrounding objects (noise, occlusion, monitors, metal), encumbrance such as cables (dancing, large-scale movement), cost

Position Trackers: Main Types• Optical

• Electromagnetic

• Mechanical

• Tangible

• Neural – brain

• Others: 3D, Videometric, Ultrasonic, Inertial

Position Trackers: Optical• Visual information is used for tracking• Generally from video camera(s)• Generally need more than one unless don’t want

6 DOF; for example, the Kinect• Need good gesture recognition or visual

recognition techniques (algorithms)• Sometimes use body sensors (motion capture)• Main disadvantage is line of sight issues

Electromagnetic Position Trackers• Very common

• Need a transmitter (sends low-level magnetic fields from 3 orthogonal coils) and a receiver- can be in both configurations: transmitter on user or receiver on user

• Strength of receiving signal varies with orientation and position

• Some with cables and some wireless

Electromagnetic (con’t)

• Advantages are that line of sight does not have to be clear

• Disadvantages: can be interference from metal and monitors, must be within several feet of the transmitter

• Eg. Ascension Flock of Birds, Polhemus

Position Trackers: Mechanical

• Mechanical arms and booms that physically track movement

• Eg. of BOOM by Fakespace

• Advantages: accurate, no interference problems

• Disadvantages: Can’t move very far

Position Trackers: Tangible• Uses hands in a more natural way: always been

the case for tools from an axe to a toothbrush; relationship between design and interaction

• Physical models (for architecture, urban planning)• Clay• Sand• Blocks• Interactive lighting devices• Use of effectors, LEDs, motors, sensors, react to heat• Boxes for manipulating music

Position Trackers: Neural• Field of brain-machine interface (BMI)• Attempts to read brain signals to direct the

computer• Efforts to use EEGs, VEP (visually evoked

potential), motor imagery• Neuroprosthetic devices, brain-implantable chips

• Interfaces also benefit from research in neuroscience

Position Trackers: Others• Videometric: camera on person and tracks

surroundings via landmarks such as infrared light sources

• Ultrasonic: high-pitched sounds emitted at definite intervals – length of time – need multiple transmitters and receivers – subject to noise in the environment

• Inertial: gyrocsopes, accelerometers, tracking device attached to user with cable- generally only orientation, so used with other devices- inexpensive, good accuracy

• Neural: muscular (monitors electrical impulses in skin) – some experiments in reading brain wave

User Input: Body Tracking• Position of joints: Kinect

• Head tracking- usually through orientation tracking- used to know what to project visually

• Hand and fingers; position trackers, finger sensors (gloves), virtual scalpels

• Torso and feet

• Other body tracking such as heart rate, temperature

• Indirect tracking using props and platforms

Body Input: Eye Tracking

• Tracks movement of pupils – pretty accurate, requires that the head be still

• eg. of Jeanne Stern’s project

• Used for people with disabilities

• Problem of “Midas Touch”

User Input: Other Physical Devices• Physical controls: buttons, switches,

joysticks, mouse, steering wheels• Arduino devices• Props such as wands, 3D mouse, scalpel,

drill, realistic devices for particular application (balls, bodies, spiders), pressure devices, mobile phones

• Platforms: treadmills, locomotion, rings, kiosks, wheelchairs, cockpits, submarine control rooms, cars, workbenches,

• CAVEs, DiVE system at Duke

Other Physical Input: Speech

• Speech recognition: system will recognize commands, natural language interaction, Siri

• Microphone• Problem of accents and training: speaker

independence or dependence• How activated: button, command (talk), vision

(need eye tracking)• Sometimes increases bandwidth by using

separate processor

World Input to Virtual World• Persistent virtual worlds: exist over many

experiences, multiple users, Web communication, databases

• Importing data that changes (weather systems)• Importing real world data such as objects and

obstacles, vision, knowledge, digital images, satellite info

• Sometimes the info is brought in with transducers: translates data into electrical signals for the computer- devices such as microphones, weather stations, video, sensors, medical devices (for AR)

SourcesQuestioning Naturalism in 3D User Interfaces, Bowman,

McMahan, and Ragan, Comm. Of the ACM, 2012The Artificiality of Natural User Interfaces, Malizia and

Bellucci, Comm of the ACM, 2012Gestural Interfaces: A Step Backward In Usability, Norman

and Nielsen, Interactions, 2010Understanding Virtual Reality, Sherman & Craig, Morgan

Kaufman, 2003The Tangible User Interface and its Evolution, Ishi, Comm.

Of the ACM, 2008Neuroscience and the Future of Human-Computer

Interface, Minnery & Fine, Interactions, Mar-Apr 2009Building on Realism and Magic for Designing 3D Interaction

Techniques by Kulik, IEEE Computer Graphics and Applications, Nov/Dec 2009

Sources – con’tTangible Interaction=Form+Computing, Baskinger & Gross,

ACM Interactions, Jan-Feb 20103D input devices, Frohlich et al, CG&A, Mar-Apr 20063D User Interfaces: New Perspectives and Directions,

Bowman et al, Comp Graphics and Apps, Nov-Dec 2008

Usability of Multiple Degree-of-Freedom Input Devices and Virtual Reality: Displays for Interactive Visual Data Analysis, Moritz and Wischgoll, VRST (Virtual Reality Software and Technology) 2007