Touching More than 3 Dimensions Research Into Novel Interfaces – three dimensions

and above

Joaquim JorgeINESC-ID & IST / U Lisboa

Portugal

IP / Leiria Nov 2015

Joaquim JorgeInstituto Superior Técnico

Universidade de LisboaVisualization and Multimodal Interfaces

@ INESC-ID Lisboahttp://web.ist.utl.pt/jorgej

ACM/SIGGRAPH Small Conferences Committee Chair

Editor Computers & GraphicsResearch Interests: Calligraphic Interaction, Multimodal Interfaces, Graphical Modeling

IS TONY STARK HOME IS TONY STARK HOME ?

WE HAVE SEEN THE FUTURE…

Natural Interfaces ?

http://youtube.com/watch?v=oIOxT6PUgPA

Dassault Systèmes Vision

NATURAL INTERACTION Ivan Sutherland – 52 years ago

Ivan Sutherland SketchPad Video

youtube.com/watch?v=495nCzxM9PI

Beyond 2D+ Space Continuous Above+ Hands Bimanual Interactions+ Dimensions 3D UIs+ Expressiveness Sound, Kicking+ Proxemics Eery Space

INTERACT 2011Interaction Techniques Unifying Touch and Gesture On and Above a Digital Surface

The Continuous Interaction Space Video

http://youtu.be/A8kb7BV7j1Y

Exploring Bimanual Interaction on a TableTop

www.shapeshop3d.com

Pedro Lopes, Daniel Mendes, Bruno Araújo, Joaquim A. JorgeCombining bimanual manipulation andpen-based input for 3D modeling, SBIM 2011

Key Points

Table ArchitectureCombining Pen and GesturesAsymetric Guiard Model

Table Architecture

MOVIE

See Movie at: https://vimeo.com/152554942

EvaluationTest Procedure

Free drawing

EvaluationMore sketching, less camera

Evaluation

Camera operations: more fluid

(seconds)

44% faster

EvaluationHand Role Distribution

9% 91%

3% 80% 17%

EvaluationUsers prefer touch for all manipulations

Imagem e benefits of direct vs indirect

EvaluationPrefer Combined technique

ObservationsBimanual workspace and hand role

ObservationsNatural resting position of NDH

ObservationsPen-only often forces re-arrangement

ObservationsPen-only causes too much DH motion

ObservationsRotate gesture was frail, swapped pen to NDH

ObservationsCombined gestures were naturally used

Getting it together: Mockup Builder

SetupMultitouch 120HzTouch 16.25 DPI3D SVGA 120HzActive StereoKinect (30Hz)Gametrak (120Hz)

Gametrak

User ModelActive User KinectHead Position KinectHand Position (Right/Left) KinectFinger touches Table4 Fingers tracked in Space (Gametrak)

Finger X from Hand Y on or above the surface from user Z

BIMANUAL AsymmetricDH : SketchingNDH : Manipulation

Modes Hands (WIMP)

CONTINUOUS INTERACTION SPACE

NDH: Context Menus DH: Selects

ON & ABOVE THE SURFACE

Snapping

Discrete Constraint

Continuous Constraints

On the flyActive as long as needed

Preliminary Results

expert architect Rhino3D5’20”

2’45”

5’41”

3’46”

Expert User our modeler

Touching The 3D Dimension, Dagstuhl Feb. 2012

Modeling On and Above a Stereoscopic Multitouch Display, (3DCHI): May, 2012.

Mockup Builder: Direct 3D Modeling On and Above the Surface in a Continuous Interaction Space (GI'2012), May 28 - May 30, 2012.

Special Section C&G (January 2013)

ACM Spatial User Interfaces Symposia (SUI)

Adding Sound to Multimodal Interfaces

ITS 2011

Take-Home message

Touch signature

Impact Intensity

Interaction space

Tangibles

Take-Home message

Touch signature

Impact Intensity

Interaction space

Tangibles

Take-Home message

Touch signature

Impact Intensity

Interaction space

Tangibles

Take-Home message

Touch signature

Impact Intensity

Interaction space

Tangibles

Acoustics of Touch

Acoustics of Touch

Acoustic SensingSetup

Interaction Opportunities

https://vimeo.com/30997725

https://vimeo.com/30997725

See Video

Acoustic SensingConclusions

Affordable Setup

Acoustic SensingConclusions

Understand impact (amplitude)

Acoustic SensingConclusions

Recognize contact parts (timbre)

Acoustic SensingConclusions

Non-tracked Tangibles

Acoustic SensingConclusions

Multiplicity of Input

Let’s Kick ItCHI 2014

Exploring New Interaction Space

Large Scale Displays

Interacting with the bottom of the screen

Beyond Situated Displays

Eery Space

Remote Proxemics: Person to Device Interactions

Social Interactions

Proxemics Distances [Hall, 1966]

Eery Space

Providing Awareness: Floor Circles

Issues

Working / Virtual SpaceSimple Operators / DialoguesImplicit Mode ChangesContinuous TransitionAsymmetric Bimanual Modes

Take Home MessageExtending Touch With Other ModalitiesBimanual InteractionExplore Surface Above TableUse Sound in Novel WaysExplore New ModalitiesProxemics as tool for implicit interactions

Research Issues

Content Creation in VEsSketch Interpretation (3D) New ModalitiesSelf Disclosure (Not!)Understandability

Thank You!Joaquim Jorge

http://web.ist.utl.pt/jorgej

Bruno Rodrigues De AraújoMaurício de Sousa

Ricardo Jorge Jota CostaPedro Lopes

Daniel Mendes