Axis-based Multitouch Interfaces for 3D Manipulation

Technical Report

Abstract

We present intuitive multitouch interfaces for interactive 3D object manipulation. These interfaces are widgetlessand buttonless, allowing the user to directly manipulate 3D objects without operating complex widgets or per-forming mode switching or tool selection. Exploiting a real-time finger registration method and context-sensitiveoperation analysis, the interfaces directly map different multitouch gestures to different 3D manipulation tasks.User interaction is greatly simplified by using a single touch gesture that simultaneously specifies the transforma-tion type, orientation, and applies the transformation to the focus object. In addition, active snapping and axistransfer functionalities are supported. The former allows the user to snap together two objects by simply drawinga free touch path connecting them. The latter allows an object to be transformed along a predefined axis of anotherobject. Our pilot evaluation indicates that users can efficiently use our interfaces to construct simple 3D scenesafter only a short training period.

Categories and Subject Descriptors(according to ACM CCS): I.3.3 [Computer Graphics]: Interaction Techniques—H.5.2 [Information Interfaces and Presentation]: User Interfaces—

1. Introduction

Multitouch techniques are now commonly used in vari-ous applications. Many digital devices, from large desk-top computers to handheld mobile internet devices, are nowequipped with touchscreens or touchpads supporting mul-titouch operations. However, the use of multitouch input inreal-world applications have so far been restricted to enhanc-ing navigation and browsing functionalities, for example,browsing of images and maps on a virtual plane or navi-gating 3D spaces using multitouch supported rotation andzooming operations like in Google Earth [GOO].

3D modeling has made significant impact in the engi-neering, design and entertainment industries. However, theinterfaces provided in commercial modeling systems aremostly designed for use with keyboard and single-point-input (mouse or pen devices). Traditional 3D modeling soft-ware are notoriously difficult to use due to their complexuser interfaces that include a large set of mode-switchingbuttons and keyboard shortkeys. 3D manipulation is partic-ularly frequently used and are therefore assigned the mostaccessible buttons and keys. Ideally, the use of buttons andkeys should be avoided in touch-based systems.

While the traditional 3D manipulation widgets found incommercial modeling systems can be directly integrated into

touch-based interfaces, the design of these widgets is basedon a tool-switching metaphor which conflicts with the tool-free philosophy of multitouch environment. Further, smallwidgets in standard interfaces are difficult to operate due tothe relatively larger finger-tip blob.

There has been little research on using multitouch inputfor complex 3D editing such as manipulation of multipleobjects in 3D space. In fact, multitouch input contains richorientation and transformation information, allowing user toprovide multiple input data with only a single multitouchaction. This avoids tedious editing steps such as mode/toolswitching and item selection required in traditional modelingenvironments. In this paper we present a novel multitouchinterface for direct 3D object manipulation that is widget-less and buttonless, and supports imprecise touch-based in-put without sacrificing control power and usability.

Our system supports most manipulating capabilitiesfound in commercial 3D modeling interfaces. In addition tostandard translation, rotation and scaling operations, our sys-tem supports snapping of two arbitrarily located objects andmanipulation relative to an arbitrary reference frame. Theconvenient transfer of reference frames between objects isenabled by our axis transfer interface, which further sim-plifies relative transformation tasks among multiple objects.

2 Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation

Figure 1: Examples of models created using our multitouchinterface.

Figure1 shows two examples which created by our proposedinterfaces within a few of minutes.

2. Related Work

Since the early days of development of computer graphics,researchers have been investigating manipulation of 3D ob-jects in interactive environment. Early solutions usedindi-rect controls such as sliders to control the transformationof objects [ETW81]. With the increased computation power,real-time systems based on basic 3D widgets like skitter andjack became possible [Bie87].

In general, 3D widgets are any visual elements formanipulating objects in the virtual environment. Earlierresearch focused on general frameworks for 3D widgetdesign [CSH∗92, SZH94]. Later, interactive systems forspecific applications were developed [GP95] [CSB∗05][Han97]. Although widget implementations vary from appli-cations to applications, the most common manipulation toolsconsist of three-axis widgets that allow axis-based transla-tion, rotation and scaling, which are uniformly supportedby commercial 3D modeling software [AUTa,AUTb,BLE].While 3D manipulation widgets are now extensively used,they are in fact mainly designed for use with keyboard andsingle pointing devices (e.g. mouse or pen). There are othersystems developed for use with two pointing devices bi-manually for 3D manipulation [ZFS97,BK99], however theyhave not been popularized to the general market.

To support the increasingly popular sketch modelingparadigm, Schmidt el al. [SSB08] presented an interface for3D object manipulation using transient 3D widgets definedby sketched context-dependent strokes. The widgets are au-tomatically aligned to the object’s axes determined by theuser’s stroke, It provides widget interaction for coarse objectpositioning and orientating. Our interface design is inspiredby their approach and shares the same goal of minimizinginterface complexity and user interaction. We choose a wid-getless approach that exploits the rich information of multi-touch input, leading to a seamless interface that allows userto use single touch actions to specify the transformation axis,

transformation type, and at the same time apply the transfor-mation.

More recently, due to the rapid development of multi-touch display devices [DL01,Han05], researchers have beenexploring multitouch interfaces for virtual object manipula-tion. Direct manipulation techniques for 2D free-form trans-lation, rotation and scaling were first introduced [KCST05,HCV∗06, SVFR04]. Then several extensions from multi-touch 2D manipulation to 3D virtual spaces have beenproposed, such as the works of Hancock et al. [HCC07,HtCC09] and Reisman et al. [RDH09]. These works fo-cus on mapping direct touch inputs to 6DOF control (lo-cation and orientation) of 3D objects in the virtual space.These techniques provide intuitive control, especially free-form rotation, for 3D objects. This is highly suitable for ca-sual browsing applications that provide easy interaction foruser to browse and locate 3D objects. However, due to thelack of precise axis-based manipulation, they are not suit-able for modeling and composition tasks. In addition, thesetechniques mainly focus on single object manipulation andglobal operations such as changing of view-point, they donot support multiple object manipulations such as alignmentand object duplication. In contrast, our proposed interfaceenables precise axis-based transformations and integrateswith global view-point changing, objects snapping, focus-ing on efficient 3D composition as provided in traditional3D applications.

Recently, Au and Tai [AT10] presented a simple fingerregistration technique that can distinguish in real-time whichhand and fingers of the user are touching the touchscreen.Their method is able to distinguish different hands and fin-gers directly from the positions of the contact points when-ever the user places a hand in any orientation anywhere onthe touchscreen. We make use of this registration techniqueto design our interface for 3D object manipulation. It allowsthe design of rich multitouch interactions that associate dif-ferent operations to touch actions involving different palmand finger combinations, resulting in seamless interactivemanipulation without explicit mode and tool switching.

3. Design Overview

In general, any interactive editing process involves threemain kinds of operations – regions/objects focus selection,tool selection and manipulation. Take for example editing adocument with word processing software, the user first needsto select the correct page and location to be edited, s/he ed-its the content using a tool, such as highlighting the selectedtext by pressing a toolbar button. These three operations (fo-cus selection, tool selection and manipulation) consume themajority of the interaction time, therefore convenient and ef-ficient switching between operation modes and manipulationtools is the crucial part of intuitive interface design.

Interface design for 3D modeling is even more challeng-

Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation 3

Figure 2: 3D transformation widgets used in AutodeskMaya. (Left) Universal Manipulator. (Right) Move / rotate/ scale tool.

ing due to the need of 3D camera control and 3D manip-ulation with 2D pointing input. Most major commercial 3Dmodeling tools use 3D widgets to provide precise 3D manip-ulation. Generally, such widgets act as visual handles corre-sponding to specific axes for different operations, such asrotation about an axis or translation in a particular direc-tion. Almost all these 3D modeling tools are designed foruse with single pointing devices, thus the widgets they usealways contain numerous small elements, corresponding todifferent operations (e.g., the 3D transformation widgets ofMaya shown in Figure2). The small elements such as ar-rows, curves and cubes are designed for precise 3D manipu-lation, such as translation, rotation and scaling in the prede-fined axes of the object.

To enable direct and seamless manipulation, we chose todiscard all widgets in our user interface design. Our gen-eral goal is to design a "widgetless" 3D modeling environ-ment powered by multitouch input. As multitouch inputscontain rich orientation and transformation (motion paths oftouch points) information, they suit the needs of 3D edit-ing applications which require multi-dimension inputs. In-corporating the recently proposed finger and palm registra-tion method [AT10], we define rich and easy-to-use gesturesto replace the traditional complex widgets of non-multitouchsystems.

User indicates an object to manipulate by explicitly select-ing it with a single tap. After that, user can apply a touch ac-tion (single touch or multitouch) to manipulate the selectedobject. Rather than relying on explicit selection of the op-eration mode and editing tool by widgets and buttons, weadopt a context-sensitive operation analysis to determine theuser-desired operation to be performed based on the currentselected object, the user’s action and the registered palm andfingers. This allows designing an interface that directly mapsdifferent multitouch gestures to different types of 3D manip-ulations. User interaction can be greatly simplified by de-signing single touch actions that specify the transformationtype, the orientation, and at the same time apply the trans-formation to the focus object. Our interface supports basicmanipulation tasks including the global browsing operations

which control the camera or change the viewpoint and theaxis-based manipulations which transform objects to newpositions and orientations.

For most manipulation tasks, we choose to use the mo-tion paths and orientation information of the touching points,rather than their contact locations to determine user’s desiredoperations. Users are not required to precisely touch any spe-cific objects or UI elements to specify their intentions (recallthat no popup widgets are used). In any single manipulatingoperation, our interface constrains the object along a specificpredefined axis, this decreases the overall interface complex-ity. However, this does not reduce the degrees of freedom ofediting, as user can always define or reuse the axis set ofdifferent objects using the axis transfer operation. With thenon-locational input and constrained axis manipulation, ourinterface is tolerant to inaccurate input and provides a casualediting style which is suitable for non-professional users.

Although our multitouch interface avoids mode and toolswitching to change viewpoint and object transformations,in general it still requires multiple interaction steps to trans-form an object to a desired new position in the 3D virtualspace. For simple stacking and snapping manipulations, oursystem provides the active snapping operation that allowsuser to directly snap two 3D objects by drawing a free touchpath connecting the snapping objects. This avoids tediousoperations required with the standard transformation manip-ulations. In addition, our interface supports axis transfer op-eration that allows relative manipulation (e.g. rotate one ob-ject along an axis of another object) to be accomplished eas-ily. We design a simple two-hand interface for the axis trans-fer operations, with one hand selecting the reference object(the axes provider) and the other hand applying a basic ma-nipulation operation along a specific axis of the referenceobject, similar to a normal editing scenario.

Figure 3: Automatic finger registration. The palm and fin-gers are registered based on the relative positions and ori-entations of the finger tips [AT10].

4. Finger and Palm Registration

To enable richer control with multitouch input, our sys-tem makes use of a recently proposed automatic and real-time finger and palm registration method to determine whichpalms and fingers of a user are touching the screen dur-ing editing [AT10]. Whenever the system detects multiple

4 Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation

contact points within a short interval, the finger registrationprocedure is invoked to determine which hand of the useris touching the screen and which contact point belongs towhich finger. The registration process is based on the rela-tive positions and orientations of the finger tips when threeor more fingers are touching the screen in a natural pose. Theprocess is performed in real-time and it supports placing offingers in any arbitrary location and orientation (Figure3left).

Since the finger registration is only activated wheneverthree or more contact points are detected, it can be concur-rently used with other touch-based interfaces that do not re-quire finger registration, such as scaling with two fingers (seeSection5.3 axis-based manipulation) or manipulation withone finger (e.g. active snapping in Section5.4).

5. Multitouch Interface Design

A key characteristic of our interface design is enabling seam-less browsing in 3D space and 3D manipulation. We adopta context-sensitive approach to determine the user’s desiredoperation based the current selected object(s), the registeredpalm and fingers, and the motion and orientation of the con-tact points.

5.1. Global Browsing

Global rotation is one of the most frequently used manip-ulations in interactive 3D applications, since from time totime users need to check the current editing scene from dif-ferent viewing directions. Therefore we choose to use a sim-ple touch path (with one contact point) for the global ro-tation operation. Our UI adopts the virtual trackball inter-face [CMS88, HSH04] as the rotation controlling tool. Ob-jects and scene can be easily rotated about any axis with thevirtual trackball interface. In our design, the start and endpoints of a touch-and-drag gesture specify the orientationand magnitude of the rotation (see first row of Figure4).User activates the trackball interface by touching the screenusing one finger, dragging that finger to the desired end posi-tion and finally raising the finger to complete the operation.Smooth transformations are achieved while the end point ismoved by the user, providing instant and continuous feed-back. Note that we do not support global rotation about aspecific axis (e.g. the world x-, y- and z-axis), since pre-cise viewing direction is usually not needed for interactivemanipulation. In contrast, we provide axis-based transfor-mation operations for precise manipulation of individual ob-jects, but do not support free-form transformations of indi-vidual objects.

For global panning (panning the view point/camera par-allel to the screen), our UI uses the right palm translation(with 5 fingers touching the screen in a natural pose) to acti-vate the operation. It considers the screen as a large paper onthe table and the panning operation simulates shifting of the

Figure 4: Global browsing operations. (From top to bot-tom) Rotation with a single contact point; Panning with rightpalm translation; Zooming in and out with right palm radialmovements.

paper (see second row Figure4). In order to avoid accidentalactivation of global panning, we rely on the palm and fin-ger registration method in [AT10] to check if all the contactpoints are from the same palm.

For global zooming (moving the view point/camera for-ward and backward), our UI uses the fingers radial move-ment of the right palm to activate the operation. It simulatesthe pulling and pushing actions corresponding to the zoomin and zoom out operations (see last two rows of Figure4).Similarly, we use the finger registration to check whether thecontact points are from the same palm.

5.2. Object Selection

User needs to select the target object(s) in the scene beforeapplying any subsequent manipulation. Individual object isselected by tapping (i.e. a touch-and-raise operation) the ob-ject using one finger. The selected object is highlighted inred, indicating that it is the focus object to receive the sub-sequent operation (Figure5 left). Multiple object selection

Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation 5

Figure 5: Object selection. (left) A 1-finger tap to select anobject. (right) A 5-finger tap to include the new object underindex finger into current selection.

Figure 6: A double tap to select an object and center it atthe same time.

is also possible. To extend the current selection, user uses a5-finger tap with the new object to be included in the selec-tion under the index finger. The newly selected object thenchanges to blue, indicating that it is now included in the cur-rent selection (see Figure5 right), while the focus object re-mains red. We will present operations performed on multipleobjects in Sections5.4and 5.5.

Centering Object.To facilitate easy manipulation of the fo-cus object, we provide a fast way for the user to center iton the screen. The user can combine selection and centeringusing a double-tap. That is, in addition to the object being se-lected, the viewing point is panned automatically such thatthe selected object is centered on the screen (Figure6).

5.3. Axis-based Manipulation

We envision basic manipulation of 3D object achievablewith two-point gestures, since such gestures suffice for spec-ifying 2D position, orientation and transformation informa-tion in the screen space for editing purposes.

Since a 2D orientation in screen space cannot uniquelydefine a 3D axis or orientation, we predefine a set of can-didate axes for the focus object (or the focus group of ob-jects). The candidate axes usually form an orthogonal framebut are not necessarily so. For simplicity, we define them asthe face normals of the boundary box of the selected objects.Other possibilities are using the face normals of the convexhull of the objects or segmenting surface to obtain the patchorientations. User uses a 2-point touch to specify a 2D orien-tation which is then compared with the projected directionsof the candidate axes. The candidate axis with orientationmost similar to the specified 2D orientation is used for the

Figure 7: Choosing an axis for manipulation from the can-didate axes of the focus object. The axis with projected ori-entation most similar to the 2D orientation specified by a2-point touch (red circles) is selected (thick red line).

current manipulation (Figure7). From viewpoints where anaxis is nearly parallel to the viewing direction, we disallowselecting that axis since transformations along that axis areunstable due to a short distance in screen space correspond-ing to a large distance in 3D space.

Note that the axis selection process is activated by a 2-point touch only when there is a focus object. After an axisis selected, user can immediately apply manipulations, giv-ing a seamless and smooth manipulation integrating objectselection, axis selection and editing into one single action.

Recall that our system is widgetless, not requiring user totouch specific positions on the object or elements of widgetsto perform editing. Only the orientation and motion given bythe contact points are considered. This reduces the difficultyof editing using multitouch input since no complex widgetsneed to be handled and no precise tapping and selection arerequired.

Axis Translation. With the axis of an object selected, ifthe user moves the touching fingersalong the axis direction(i.e., translate the contact points parallel to the axis orienta-tion), we designate that gesture as axis translation. The sys-tem translates the object according to the moved distance ofthe contact points (Figure8 top). The amount of translationis according to the displacement of the contact points on thescreen, allowing the whole screen space to be the input re-gion and thus more easily achieve precise control.

Axis Rotation. Orienting 3D objects using a 2D input de-vice is difficult for non-experienced users. This is one ofthe reasons why 3D modeling systems have steep learningcurves. The virtual trackball technique can provide intuitivecontrol, however it is difficult to specify precise orientation.Users are significantly faster and more accurate when per-forming simple single-axis rotation [CMS88]. All commer-cial 3D modeling systems provide constrained rotation wid-gets to support single-axis rotations (see Figure2). In our UIdesign, axis rotation is activated when the two touching fin-gers are movedperpendicularlyto the selected axis (Figure8middle).

Axis Scaling. Like most 3D modeling systems, our UIsupports two different types of scaling manipulations. The

6 Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation

Figure 8: Axis-based manipulation. (from top to bottom)Translation: move two contact points along the chosen axis;Rotation: move two contact points perpendicular to the cho-sen axis; Scaling: Contract or expand the distance betweentwo contact points to scale object along the chosen axis.

first type is uniform scaling, which scales simultaneously inall three dimensions. Similar to global zooming, we designto use the radial movement of the left hand’s fingers to ac-tivate this operation, except that now the object to be scaledhas been selected. The second type is axis scaling. User con-tracts or expands the distance between two contact pointsalong the selected axis (Figure8 bottom) to scale the focusobject along the chosen axis. The amount of scaling (enlargeor shrink) is defined by the initial distance between the twocontact points and their displacements.

Buffer Zone. The user-desired manipulation operationsare determined based on the relative movement of the con-tact points, namely, 2D translation (for axis translation andaxis rotation) and 2D scaling along a line (for axis scaling).For robustness and avoidance of mis-interpreted operationsdue to unstable user touch input, our system ignores thosemovements of touch points within a given short distance.The transformation type of contact points and user desiredoperations are determined only when the contact points havedisplacements larger than the given threshold. This approachalso allows the user to cancel the current manipulation bymoving his/her fingers to the initial touch positions, causingthe contact points to be located within the buffer zone andthus no manipulation is applied. Note that this buffer zoneidea is similar to the magnitude filtering technique used for

Figure 9: (top) Automatic snapping between two snap-ping planes with similar orientations. (bottom) When mul-tiple possibilities are detected, snapping pairs are orderedaccording to their distances and conflicting pairs are dis-carded.

2D manipulation in [NBBW09]. The difference is that forsimplicity of control we only allow one of the operations(translation / rotation / scaling) to be applied in one user ac-tion, while [NBBW09] allows users to apply combination ofthese operations in a single action.

5.4. Object Snapping

Snapping is a very effective way of specifying accuratetransformations. Interactive snapping techniques are exten-sively used in 3D modeling systems, especially engineeringCAD systems, as they avoid tedious menu or dialog opera-tions. We also provide automatic snapping in our system toreduce user’s interaction. We predefine the snapping planesand edges for each input object, which are defined by thesurface normals of the object bounding box. In the case ofcomplex objects, the snapping planes can be found by ana-lyzing the surface normals of the object.

The snapping planes and edges define the snapping condi-tions during manipulation. The object being manipulated issnapped to another object if a pair of their snapping planesor edges are close to each other and have similar orienta-tions (Figure9 top). In the cases when two or more possiblesnapping pairs are detected, the snapping pairs are orderedaccording to their distances and the snapping pairs that con-flict with previously applied ones are ignored (Figure9 bot-tom).

Active Snapping. Automatic snapping is simple to use,however it still requires the user to place the object at anapproximate location and orientation to activate the snap-ping. It is useful and efficient to allow user to directly snapor stack objects together without first transforming the se-lected object. We call such a functionalityactive snapping,which requires only a single action to specify the snappingobjects and how they are to be snapped. User draws a free

Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation 7

Figure 10: Active snapping examples. The snapping planeswith normals parallel to the starting and ending orientationsof the touch path are selected. Note that the snapping planesneed not be visible.

touch path connecting two objects to be snapped. By project-ing the normals of the snapping planes of the focus object tothe screen space, user can easily select a snapping plane us-ing the drawn path. Specifically, the snapping plane of theobject with normal parallel to the starting orientation of thepath drawn by the user’s finger is selected (Figure??). Sim-ilarly, the target object and its snapping plane are selectedaccording to the ending location and orientation of the path(see two examples in Figure10). Since we are using the be-ginning and ending orientations of the touch path to specifythe snapping, user can snap objects even when the snappingfaces are not visible to the user. This avoids additional ro-tation of the view or objects needed by other interfaces thatrequire user to specify the points for snapping, like [SSB08].Overall, this operation avoids the otherwise tedious move-ment and rotation of the objects, giving a convenient inter-face for efficient multiple object editing.

5.5. Axis Transfer

A common limitation of most 3D editing systems is that,while it is easy to select the canonical axes of the editingobject, setting up arbitrary axes is often a difficult task. Itrequires setting up a pivot or frame object, which incurs ad-ditional manipulation. However, often the user’s desired axisis simply a candidate axis of another object. For example, inFigure11, to duplicate the leg of the toy dog model, the se-lected object group needs to be translated in the coordinatesystem of the body part. Thus what we need is transferingof axes between objects. Our solution to this task is a two-hand gesture, with one hand selecting the reference objectthat provides the new axes and the other hand manipulatingthe target object with respect to a specific axis of the refer-

Figure 11: Axis Transfer with a two-hand gesture. Dupli-cate and translate the selected dog leg in the coordinate sys-tem of the body. One hand selects the axis-provider referenceobject; the other hand manipulates the target object along aspecific axis of the reference object.

ence object (Figure11right). This solution naturally enablesthe user to easily construct new canonical axes for editing viaplacing a simple reference object as proxy.

5.6. Other Supporting Operations

Object Duplication. Object duplication is an important ba-sic operation in many 3D editing scenarios. Our multitouchmodeling system supports object duplication in two ways.We call the first wayactive duplication, which is similarto axis translation, except that instead of translating the se-lected object, here a new copy of the selected object is cre-ated then translated (Figure12 top). Active duplication isactivated with a 3-finger translation (with thumb, index andmiddle). The translation direction of the fingers decides thetranslation axis.

The second way is calledtransformed duplication,whichrequires two selected objects. It creates a new copy of thefocus object and transforms the new copy of focus objectto a new location based on the transformation between thetwo selected objects. This operation is supported by manycommercial modeling systems and is sometime referred toas "advance duplication". Our system provides this operationto facilitate construction of more complex scenes (Figure12bottom). Note that the selected objects need not be of thesame type since only the relative transformation between theobjects is needed for the duplication.

Atomic Operations. There are other supporting opera-tions that do not require any orientation and positional in-formation but are necessary for a complete interactive edit-ing system. We call them the atomic operations as theyare usually simply activated by selecting a toolbar buttonor menu item in traditional editing systems. Examples arethe undo/redo operations, and loading new objects or re-moving selected objects from the current scene. To integratethese operations into our multitouch system, we use the PalmMenu in [AT10] to provide a menu system that enables fastaccess and minimizes finger and eye movement. The menuis activated by a 5-finger left-hand tap and the popup but-tons are placed exactly under the finger contact points (Fig-

8 Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation

Figure 12: Object duplication. (Top) Active duplicationwith 3-finger translation: a new copy of selected object iscreated then translated. (bottom) Transformed duplication:the focus object (red) is duplicated and then translated basedon the transformation among objects in the selected group(red and blue objects).

ure.13). User can then tap one of the popup buttons to selecta desired command. This design is simple and intuitive sincedifferent commands are directly mapped to different fingers.Users even do not need to look at the popup buttons becausethe buttons are already located exactly at the finger contactpoints. This avoids the switching of focus between the objectto be manipulated and the menu itself.

6. Evaluation and Discussion

We conducted pilot experiments to investigate the effective-ness of our interface design. Six graduate students fromcomputer science department were involved in these infor-mal tests. All participants were experienced computer usersand about half of them regularly use multitouch applicationssuch as image and map browsing tools on mobile devices.None of them has extensive experience with graphics mod-eling software. Since one of our objectives is to provide in-tuitive multitouch interfaces for non-professional users, we

Figure 13: A five-finger tap activates the Palm Menu, whichhas menu buttons located exactly at the finger touch points.

Figure 14: Participants were asked to construct the castlefrom given basic blocks using our multitouch interface.

did not directly compare our system to commercial 3D mod-elling software, nor other traditional single-pointing-basedand widget-based interfaces. We record the feedback aboutthe controllability of our interface and the participants’ ac-ceptability of the interface.

Participants were asked to perform an assembly task ofconstructing a castle with different shapes of blocks (Fig-ure 14). There were no accuracy requirements; participantswere free to use the proposed manipulation gestures in anyorder to construct a castle as similar to the one shown as pos-sible. We then collected feedback from each participant afterhe/she completed the session.

The results of these pilot experiments indicate that our in-terface design is effective and has good controllability forbasic 3D manipulation. Each participant takes different timespans to familiarize with the manipulation operations, de-pending on their experience with 3D manipulation, fromseveral minutes to half an hour. All participants can success-fully construct the castle within 10 to 20 minutes. Partic-ipants who are regular users of multitouch mobile devicestake relatively less time to learn our interface and completethe assigned task. All the participants reported that after thelearning session, they can easily place the objects at the de-sired locations. The active snapping operation is particularlyconvenient for placing object at precise locations. Note that,with only standard transformations, due to the lack of par-allax in 2D display, user would need to change their view-ing angles to examine whether the objects are placed at thespecified locations. With the active snapping operation thatallows automatic alignment of objects in a single view, userscan avoid such examinations and adjustments, leading tofewer view-angle changings.

Some participants have experience playing 3D games andusing 3D browsing applications like Google Earth [GOO].Testing results show that they have better understanding ofthe basic 3D orientation concepts, such as view point, viewangle, transformation axis and transformation type, and theyare able to better control 3D objects. In comparison, other

Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation 9

participants who lack the understanding of visualization of3D space on 2D display reported frustration initially in con-trolling 3D objects that float in the virtual space. They needto spend more time familiarizing with the basic 3D orienta-tion concepts, starting from changing view angle with virtualtrackball to selecting suitable axis and transformation oper-ation.

Participants reported two minor issues after completingthe task. They cannot rotate objects with a rotation axisnearly parallel to the viewing direction (recalls that we dis-allow the selection of such an axis due to unstable controland difficulty in determining the projected axis orientation).Although using a global rotation to change the viewing an-gle can handle this issue, it introduces extra steps in the in-teractive editing process. Some participants suggest extend-ing the current interface to support screen-space rotation, forboth the camera (global rotation) and the focus object (localmanipulation).

The second issue is that participants sometime forgot tofirst select an object before applying the active snapping op-eration (thus they actually perform a global rotation withthe free-form path). This is the limitation of the context-sensitive interfaces since users need to remember which op-erations are supported under the current selection context.However, we found that once the dependence of the context-sensitive interactions were explained, users rarely make suchmistakes. In contrast, we noticed that users do not have thisproblem with the axis-based transformations, apparently be-cause the candidate axes of an object are displayed only afterthe object is selected. This extra visualization informationgives an implicit hint to user about the operation order. Notethat for the active snapping, there is no such auxiliary visu-alization to indicate that snapping can only be applied to thefocus object.

7. Conclusions

In this paper we proposed a novel multitouch interface for in-teractive manipulation of 3D objects. It provides widgetlesstranslation, rotation and scaling, passive and active snapping,and axis transfer functionalities. Multi-touch input containsorientation and transformation information which is natu-rally suitable for 3D object manipulation. Utilizing these in-formation, we designed a widgetless and context-aware in-terface that allows user to directly manipulate objects in thevirtual space, based only on multitouch input, without anymode switching or tool selection as with traditional inter-faces.

We implemented and evaluated our approach. Our ex-ploratory study shows that non-professional users are ableto manipulate 3D objects using our interface to build simplescenes after only a short training period. The active snap-ping technique provides a simple way to stack and align 3Dobjects without requiring multiple transformations and re-peated changing of view direction, thus further reduces user

interaction. With the axis transfer operation, users have theability and freedom to define arbitrary canonical axes, allow-ing object transformation according to any orientation.

There are a few possible directions for future study. One isto enrich the visualization and improve the interface designin order to provide more hints for context-sensitive opera-tions, thus reduces the occasions of incorrect operations. Itis also possible to integrate more kinds of manipulations intoour interface design, for example, supporting screen / worldspace transformation for individual objects, or allowing ob-jects to be moved along curves or surfaces, rather than onlyalong the predefined axes. Note that our current interfacedesign mainly focuses on single-hand operations, there arerooms for designing more complex operations using both-hand manipulations. Finally, as a pilot study, our results arepositive and encouraging. However, a more comprehensiveand formal evaluation is still needed for future development.

References

[AT10] AU O. K.-C., TAI C.-L.: Multitouch finger registrationand its applications. InOZCHI ’10 (2010). 2, 3, 4, 7

[AUTa] AUTODESK INC.: 3ds max. http://usa.autodesk.com/ . 2

[AUTb] AUTODESK INC.: Maya 2011. http://usa.autodesk.com/ . 2

[Bie87] BIER E. A.: Skitters and jacks: interactive 3d positioningtools. InI3D ’86 (1987), pp. 183–196.2

[BK99] BALAKRISHNAN R., KURTENBACH G.: Exploring bi-manual camera control and object manipulation in 3d graphicsinterfaces. InCHI ’99 (1999), pp. 56–62.2

[BLE] BLENDER FOUNDATION: Blender 2.49.http://www.blender.org/ . 2

[CMS88] CHEN M., MOUNTFORDS. J., SELLEN A.: A study ininteractive 3-d rotation using 2-d control devices. InSIGGRAPH’88 (1988), pp. 121–129.4, 5

[CSB∗05] COLEMAN P., SINGH K., BARRETT L., SU-DARSANAM N., GRIMM C.: 3d screen-space widgets for non-linear projection. InGRAPHITE ’05(2005), pp. 221–228.2

[CSH∗92] CONNER B. D., SNIBBE S. S., HERNDON K. P.,ROBBINS D. C., ZELEZNIK R. C., VAN DAM A.: Three-dimensional widgets. InI3D ’92 (1992), pp. 183–188.2

[DL01] DIETZ P., LEIGH D.: Diamondtouch: a multi-user touchtechnology. InUIST ’01(2001), pp. 219–226.2

[ETW81] EVANS K. B., TANNER P. P., WEIN M.: Tablet-basedvaluators that provide one, two, or three degrees of freedom. InSIGGRAPH ’81(1981), pp. 91–97.2

[GOO] GOOGLE INC.: Google earth. http://www.google.com/earth/ . 1, 8

[GP95] GRIMM C., PUGMIRE D.: Visual interfaces for solidsmodeling. InUIST ’95(1995), pp. 51–60.2

[Han97] HAND C.: A survey of 3d interaction techniques. InComputer Graphics Forum(1997), vol. 16, pp. 269–281.2

[Han05] HAN J. Y.: Low-cost multi-touch sensing through frus-trated total internal reflection. InUIST ’05(2005), pp. 115–118.2

10 Technical Report / Axis-based Multitouch Interfaces for 3D Manipulation

[HCC07] HANCOCK M., CARPENDALE S., COCKBURN A.:Shallow-depth 3d interaction: design and evaluation of one-, two-and three-touch techniques. InCHI ’07 (2007), pp. 1147–1156.2

[HCV∗06] HANCOCK M. S., CARPENDALE S., VERNIER F. D.,WIGDOR D., SHEN C.: Rotation and translation mechanisms fortabletop interaction. InTABLETOP ’06(2006), pp. 79–88.2

[HSH04] HENRIKSEN K., SPORRINGJ., HORNBÆK K.: Virtualtrackballs revisited.IEEE Trans. Vis. & Comput. Graph. 10, 2(2004), 206–216.4

[HtCC09] HANCOCK M., TEN CATE T., CARPENDALE S.:Sticky tools: Full 6DOF force-based interaction for multi-touchtables. InITS ’09(2009), pp. 145–152.2

[KCST05] KRUGER R., CARPENDALE S., SCOTT S. D., TANG

A.: Fluid integration of rotation and translation. InCHI ’05(2005), pp. 601–610.2

[NBBW09] NACENTA M. A., BAUDISCH P., BENKO H., WIL -SON A.: Separability of spatial manipulations in multi-touch in-terfaces. InGI ’09 (2009), pp. 175–182.6

[RDH09] REISMAN J. L., DAVIDSON P. L., HAN J. Y.: Ascreen-space formulation for 2d and 3d direct manipulation. InUIST ’09(2009), pp. 69–78.2

[SSB08] SCHMIDT R., SINGH K., BALAKRISHNAN R.: Sketch-ing and composing widgets for 3d manipulation.ComputerGraphics Forum 27, 2 (2008), 301–310.2, 7

[SVFR04] SHEN C., VERNIER F. D., FORLINES C., RINGEL

M.: Diamondspin: an extensible toolkit for around-the-table in-teraction. InCHI ’04 (2004), pp. 167–174.2

[SZH94] STEVENS M. P., ZELEZNIK R. C., HUGHES J. F.: Anarchitecture for an extensible 3d interface toolkit. InUIST ’94(1994), pp. 59–67.2

[ZFS97] ZELEZNIK R. C., FORSBERGA. S., STRAUSS P. S.:Two pointer input for 3d interaction. InI3D ’97 (1997), pp. 115–120. 2