6
Drawing with the Hand in Free Space Steven Schkolne Michael Pruett and Peter Schröder The gesture has a spontaneity, a freedom, an unfiltered physi- cality in its instantaneous choice. There is a depth of com- munication in this moment, the split-second of a photograph, the subtle timing of a comedian. These instants are not planned or contrived, but quickly communicated through a developed intuition. Mark-based traditional media, such as drawing and painting, engage this type of moment repeat- edly in a form that engages the body. I view drawing, espe- cially sketching, as a way of physically conceptualizing, a form of thought [1]. Yet lines and the paper they occupy are two-dimensional, not addressing the bodily space. My ex- cursions into 3d creation, assembling disparate objects, seem arduous: much planning for small moments of interaction and response. Computer modeling programs [2] require an indirect manipulation of form through mathematical quanti- ties, typically with a 2d interface. I find that these laborious processes do not support the spontaneity of creation and the physical, corporeal understanding that I am interested in. Without this immediacy, I feel the ability to freely explore and thoroughly engage 3d space lacking. The medium presented here, which I call surface draw- ing[3], is a response to these concerns. This method is an extension of line drawing to 3d space using the hand in place of a pen. When the hand is moved through space, its path takes form and hovers in the air as surface. This concept is realized with advanced computer interface devices and cus- tom software. Each hand motion is sensed by an instrumented glove, recorded by the computer, and displayed as a coher- ent stroke. An accumulation of these strokes form an object, in much the way 2d lines combine. The action of creating with the hand is somewhat like touching an imaginary ob- ject and having it materialize. This method of creation is a Fig 1. A thin strip of surface is drawn with the hand. Special viewing hardware makes the stroke appear to float above the table, as depicted in this composite image. in collaboration with Abstract: This article presents a new medium in which organic surfaces are drawn in 3d space with the hand. Additional tools move and deform the shape. Special interface hardware includes a head-tracked stereo- scopic display and sensors which track the body and handheld tools, allowing the artist to share the space of the artwork. This method provides a fluid, unstructured access to 3d, ideal for quick, spontaneous ideation and investigation of complex structure. Creating Organic 3d Shapes with Gesture in a Semi-Immersive Environment to be published in Leonardo, August 2002 submitted for publication June 2000 revised manuscript submitted April 2001

Drawing with the Hand in Free Space - multires.caltech.edumultires.caltech.edu/pubs/schkolne_leonardo.pdf · sketching. To draw fwr, shown in Fig-ure 3 [11], I first sketched some

  • Upload
    buidieu

  • View
    214

  • Download
    0

Embed Size (px)

Citation preview

Drawing with the Hand in Free Space

Steven Schkolne

Michael Pruett and Peter Schröder

The gesture has a spontaneity, a freedom, an unfiltered physi-cality in its instantaneous choice. There is a depth of com-munication in this moment, the split-second of a photograph,the subtle timing of a comedian. These instants are notplanned or contrived, but quickly communicated through adeveloped intuition. Mark-based traditional media, such asdrawing and painting, engage this type of moment repeat-edly in a form that engages the body. I view drawing, espe-cially sketching, as a way of physically conceptualizing, aform of thought [1]. Yet lines and the paper they occupy aretwo-dimensional, not addressing the bodily space. My ex-cursions into 3d creation, assembling disparate objects, seemarduous: much planning for small moments of interactionand response. Computer modeling programs [2] require anindirect manipulation of form through mathematical quanti-ties, typically with a 2d interface. I find that these laborious

processes do not support the spontaneity of creation and thephysical, corporeal understanding that I am interested in.Without this immediacy, I feel the ability to freely exploreand thoroughly engage 3d space lacking.

The medium presented here, which I call surface draw-ing[3], is a response to these concerns. This method is anextension of line drawing to 3d space using the hand in placeof a pen. When the hand is moved through space, its pathtakes form and hovers in the air as surface. This concept isrealized with advanced computer interface devices and cus-tom software. Each hand motion is sensed by an instrumentedglove, recorded by the computer, and displayed as a coher-ent stroke. An accumulation of these strokes form an object,in much the way 2d lines combine. The action of creatingwith the hand is somewhat like touching an imaginary ob-ject and having it materialize. This method of creation is a

Fig 1. A thin strip ofsurface is drawn withthe hand. Specialviewing hardwaremakes the strokeappear to float abovethe table, as depictedin this compositeimage.

in collaboration with

Abstract: This article presents anew medium in which organicsurfaces are drawn in 3d spacewith the hand. Additional toolsmove and deform the shape.Special interface hardwareincludes a head-tracked stereo-scopic display and sensors whichtrack the body and handheld tools,allowing the artist to share thespace of the artwork. This methodprovides a fluid, unstructuredaccess to 3d, ideal for quick,spontaneous ideation andinvestigation of complex structure.

Creating Organic 3d Shapes with Gesture in a Semi-Immersive Environment

to be published in Leonardo, August 2002submitted for publication June 2000revised manuscript submitted April 2001

recording of gesture, capturing aperformative body as object. The rela-tionship between artist and object istwo-way, with the object enveloping theartist, affecting the growth of form.

This method provides much of the re-lationship to 3d space that I seek. I havepursued this approach, both as an artistinvestigating visual space and as a com-puter scientist (in collaboration withPeter Schröder and Michael Pruett atCaltech) developing a system to sup-port this interaction. I am concernedwith making a system that is generalpurpose for the artistic community atlarge. To this end, I first investigatedthe ability to make representational sur-face drawings. More recently I createdabstract shapes that develop structuralrelationships which are unique to thismedium. In both of these investigationsthe focus has been more on geometrythan color. This is due to a favoritismof structure, and an anticipation of fu-ture work in texture and shading.

THE PHYSICAL INTERFACE

This concept was realized using theResponsive Workbench [4], a largetable that acts as a display surface.Wearing head-tracked stereoscopicshutterglasses [5], objects appear tofloat above this table within the user’sbody space. In this environment, a num-ber of physical tools are used to createand manipulate shapes. The user wearsa glove [6] on the dominant hand whichsenses the shape of the hand. Along witha motion tracker on the wrist, these handmotions are turned into shapes [7] by acomputer which displays them inrealtime. A stroke is started by closingthe thumb – pushing it against a sensoron the lower index finger. The color ofthe stroke can be changed by turning acolor wheel that rests on the tabletop.

A pair of kitchen tongs freely movesand rotates objects in 3d space. Themotion-tracked tongs have a pressuresensor which detects when they areclosed. A second pair of tongs has thesame functionality. When the two tongsare used together they can increase thesize of the object (by closing both tongsand moving them apart) or shrink it (bymoving them closer to one another).Scaling an object changes the size of

Fig. 2. Basic methods ofdeveloping surfaces:(a) The path of thehand in space iscaptured as a stroke. Asensor on the lowerindex finger detects thethumb’s closing, whichinitiates the drawing.(b) Sensed tongs areused to rotate thisfigure drawing. Twopairs of tongs usedtogether can scale anobject. (c) An ergo-nomic 3d eraser with abutton on it. The eraserremoves a small volumefrom an object. (d) Amagnet, held betweenthe fingers, attractssurface. This tool doesnot create newgeometry, but ratherproduces smallrefinements.photography by VanessaStump

to be published in Leonardo, August 2002

the hand relative to the object, and thusdetails can be added at any scale. Smallfeatures can also be created by drawingthinner strokes; This is accomplished bydrawing with the hand in a pointing po-sition [8].

Two additional tools modify shapes:an eraser and a magnet. The eraser ismolded out of yellow silicone, designedto fit easily in the hand. When its sen-sor is pressed with the thumb, a smallvolume is removed from the piece. Themagnet fits lightly in the hand, some-what like a toenail brush. This tool at-tracts a shape, pulling it slightly towardsthe hand. This method of overdrawing[9] does not create new geometry butrather slightly modifies it.

The pattern of these interactions isvery physical. All of the operations areaccomplished with motion, the act ofcreation is a performance whose recordis part of a developing interaction. Thethought that goes into an object’s cre-ation is channeled through motion andperception. Guided by the instant,forms emerge as a reaction to space.

This interest in performance, emotion,action, and the subtlety of the artist’shand is shared with abstract expression-ist painting [10]. In the behavior of themedium, some distinctions become ap-parent. As I have become adept withsurface drawing I have grown to movethe object constantly as I work. Thepiece spins quickly, and the process isas much about seeing as doing. This in-teraction is different from that of actionpainting, where the canvas is an objectof the mark. In surface drawing, marksre-enter and thus re-engage the spaceof the body, becoming both subject andobject. Instead of the static visual fieldof the painting, this interaction nurturesan intense form of vision which is de-manding and engrossing. Through thisprocess, an interactive contemplation ofform and relationship with structureemerges that is intricate, fluid, dynamic:as much about spatial awareness, ob-servation, and reaction as it is a placefor expressionist catharsis.

REPRESENTATIONAL SHAPES

My first experiences with this methodfocus on the creation of representational

objects. Constructing such shapes re-quires the invention of new process.Drawing precise objects, much like in2d painting, is aided by preliminarysketching. To draw fwr, shown in Fig-ure 3 [11], I first sketched some basicpetal positions. I analyzed this shapeand made marks approximating morefinal petal positions, which served asguidelines for a refined drawing of theshape. The petals are best drawn edge-first, as the edges are defining charac-teristics of this form.

I find this method similar in spirit tomy 2d representational process, fluid instructure, a reflection of thoughtthrough motion very much filled withthe instant. I often find myself squint-ing my eyes and getting a feel for wheresomething should go. In stark contrastto current 3d modeling software, I donot have to think about x-y-z coordi-nates, interpolating curves, or other lin-guistic handles. There is little enforcedstructure, whatever I describe is directlyand immediately represented. In con-trast to physical media, I do not have tobuild a structure to support the flower’spetals, or concern myself as to how theywill stay in place after being drawn.

It is common in line drawing to ren-der a live model in a short pose. I re-peated this exercise with one-minutegesture drawings, the resulting surface

drawings are shown in Figure 4. Un-like 2d drawing, perspective is notneeded. The drawing instead requiredfeeling the model’s position in my ownwork space. As with the flower, thisprocess entailed many motions rollingover and around the figure as it wasbeing created. These quick sketcheshave a pronounced roughness to them,a gestural quality that is inherent to theirprocess. The chunkiness of these fig-ures is somewhat characteristic of thegraphic signature of surface drawings.One can see the body, a crudeness ofmanipulation that resembles clay. Thesolidity of clay is replaced with the thin-ness, the air of foil.

ABSTRACT SHAPES

I am currently investigating surfacedrawings that adopt a more abstract rep-resentation. These objects, less focusedon replicating extant material forms,appear quite unlike physical sculpturesor other computer graphics shapes. Fig-ure 5 shows fthr, an example of thispurely constructive process. Theseforms grow incrementally in a spatialbalance. A salient characteristic of theseobjects is that they are hard to repre-sent on the printed page. The three sepa-rate views of fthr demonstrate a richthree-dimensional complexity. Viewingthe printed page, I find picturing the

Fig. 3. fwr, triangle mesh,2000. This shape wascreated from severalhand gestures: definingmotions which denotethe edge of the petals,followed by motions tofill in the interior.

to be published in Leonardo, August 2002

mental rotations between each pair tobe quite arduous [12]. Such an object isdifficult to conceptualize with two-di-mensional tools.

This enriched understanding of struc-ture is the greatest potential of thismethod. The comprehension builds asthe space serves as marking ground forshape. Elements are easily placed, re-lations develop amongst the marks. Thisconstructive process is not predeter-mined, but proceeds in a loose and fluidmanner, letting ideas grow. This physi-cal basis for abstract thought is a per-ceptual tool which enables conception.The sophistication and simplicity of theconstructive process facilitate explora-tion with a minimum of cognitive over-head.

Other objects in this series take ad-vantage of scale, containing miniaturedetails that are as rich as the larger formthey inhabit. Their nature is difficult torepresent on the page and in physicalreproduction alike: An interactive inves-tigation of the form is needed. Throughscaling and rotation, investigating theseshapes, the viewer is thrust into a com-plex spatial dialogue, the precise lan-guage of which lies undeveloped.

The objects in this series are an ex-ample of the freedom that I have beenseeking in three dimensions. This ab-straction is employed in a bodily spacewhich directly relates to the artist dur-ing construction, and interacts with the3d viewer’s body. The shapes have amateriality, a raw roughness of physi-cality which they embody despite theirinherently aphysical embodiment. Thestereotypical emotionally divorced,purely abstract and immaterial natureof the machine has merged with thestereotypically crude, lo-fi and Nean-derthal action of the body. This qualityis symptomatic of a larger philosophyof mine, that as technology progresseswe become more aware of our bodies,of our existence as perceptual entities,rather than fleeing into an abstract men-tal space that is devoid of body.

OTHER APPLICATIONS

This method is a general way of creat-ing 3d shapes, and as such has many

Fig. 4. These one-minutegestures were drawnfrom a live model. Theirrapid construction anddemonstrate theimmediacy of surfacedrawing. All three areuntitled triangle meshes,2000.

to be published in Leonardo, August 2002

applications outside of pure artistic ex-ploration. The system can be used forthe conceptual design of any 3d object:buildings, characters, cars, clothes, fur-niture, and roller coasters included.Once prototyped, shapes can be broughtinto standard 3d modeling applicationsfor lighting, refinement, rendering oranimation. The digital files can also beused as starting points for manufacture,or directly printed as objects using 3dlithography. Surface drawings are use-ful for conceptual design. For example,surface drawings can serve as concep-tual sketches for a project that will becompleted by a traditional method suchas object assembly.

Applications of surface drawing arebeing investigated in collaboration withDesignworks/USA [13], an industrialdesign firm. They are interested in mak-ing surface drawings for conceptualprototypes of products, which in theircase range from automobiles to cellu-lar phones. In the words of Senior De-signer Gary Fitzgerald:

We are interested in capturing the emo-tions that drive gestural descriptions ofform. We will be exploring the worldof subtleties and nuances that only thistype of rapid capture enables. This is astory telling experience illustrated asgeometry [14].

They have experimented with the setupat Caltech, and I have built a secondimplementation at Designworks to givetheir designers more access to the pro-cess.

I have also shown this system to au-diences in two recent exhibitions [15].The audience understands very quicklyhow this medium works: People begincreating shapes instantaneously. Gain-ing control of this medium is more dif-ficult, and requires experience. Gesturalskills translate well into creating shapes,although some artists begin by think-ing very two-dimensionally. It takespractice to get the most out of this me-dium. The learning process that usersgo through is not one in which theylearn how the software works, but ratheran understanding of body and space.

FUTURE WORK

The medium as it stands is primarilysuited to organic shapes. Perfectly flat

Fig. 5. fthr, trianglemesh, 2000. Three viewsof this static objectreveal a complex 3dstructure. This shape ispoorly represented bythe printed page. Forexample, the rotationsbetween the shapes aredifficult to visualize. Thesmall diagrams betweenthe images denote theapproximate rotationsthe shape goes throughbefore a stationaryviewer. The spatialrelationships inherent tothis form are not easilyconceptualized with two-dimensional tools. Theoriginal 3d model isavailable online [12].

30 180

30

to be published in Leonardo, August 2002

References and Notes

1. This viewpoint is informed by RudolfArnheim’s work (Visual Thinking Univ. ofCalifornia Press, Berkeley CA1969)

2. Such as Maya <http://www.aliaswavefront.com> and 3ds max<www.discreet.com>.

3. The name surface drawing refers to theextension of line drawing to surfaces. Theaction of this meduim is much like linedrawing, except that the marking tool produces2d surfaces instead of 1d lines.

planes, hard edges, and precise symme-tries are not supported by the currentinterface. The error in the tracking sys-tem provides a further obstacle to highprecision. Thus, for industrial concerns,the medium is not yet suitable forshapes beyond the prototype phase.

Other aspects of shape creation haveyet to be developed. For example, thereis no facility to control texture and light-ing. The larger variety of spatial ele-ments, such as volumes or time-vary-ing components, has not been explored.There are many different tools thatcould be built, diversifying and enrich-ing the constructive process.

Developments in these many areaswill enhance the physical, intuitive ac-cess to space which surface drawingprovides, enriching the spatial dialogueand enlarging the resulting understand-ing of structure. In the future I plan toexamine the vast landscape of possibil-ity which this 3d environment whereany type of action can be translated intoform provides. With improved sensing,more sophisticated display, and a widertoolset, the visual language encom-passed by these techniques will be rich,growing to become a vital componentof human communication.

ACKNOWLEDGMENTS

Special thanks to davidkremers, W.Everett Kane, Gary Fitzgerald, AlecBernstein, Aaron Rincover, and Al Barr.This work was supported in part by NSF(DMS-9872890, ACI-9721349); othersupport was provided byAlias|Wavefront, the Packard Founda-tion, Pixar, Intel, and DesignWorks/USA.

4. The Responsive Workbench was developedby Wolfgang Kruger and colleagues at GMD.See Kruger, W. and Fröhlich, B. “TheResponsive Workbench”, IEEE ComputerGraphics and Applications (May 1994) 12-15.

5. These glasses provide the illusion of depthby showing different images to the left andright eyes. A transparent LCD screen coverseach eye. The left screen turns black as animage for the right eye is displayed and viceversa.

6. The Cyberglove, by Virtual TechnologiesInc., Palo Alto CA <http://www.virtex.com>

7. The surfaces are represented as trianglemeshes, collections of colored triangles in 3dspace with connectivity information. Moretechnical details can be found in: StevenSchkolne, Surface Drawing: The PerceptualConstruction of Aesthetic Form. M.S. thesis,Computer Science, Caltech, 1999. <http://www.cs.caltech.edu/~ss/sdraw/pubs>

8. Computer users often ask for a feature suchas the ability to precisely adjust the strokewidth. The interface philosophy here is one ofminimalism. It is not necessary to create astroke-width knob if a similar effect can beacheived with the current tools (in this acase,scaling).

9. This mode was inspired Cohen et. al’s workon overdrawing curves. See Cohen, J.M.,Markosian, L., Zeleznik, R.C., Hughes, J.F.,and Barzel, R. “An interface for sketching 3Dcurves.” 1999 ACM Symposium on Interactive3D Graphics (1997). pp. 107-114.

10. Sandler, Irving, The Triumph of AmericanPainting; a History of Abstract Expressionism,Praeger Publishers 1970.

11. More images, along with further descriptionof this project can be found at <http://www.cs.caltech.edu/~ss/sdraw/>

12. A 3d version of fthr in VRML format isavailable online at <http://www.cs.caltech.edu/~ss/sdraw/gallery/fthr.wrl>

12. <http://www.designworksusa.com>

13. Personal communication, July 2000

14. Surface Drawing has been exhibited twice,first at SIGGRAPH 1999’s EmergingTechnologies forum, Los Angeles California,August 1999. The second exhibit was at the3rd Petrobras Mostra de Realidade Virtual, RioDe Janeiro, June 2000. Both exhibits involvedshowing audience members how to use surfacedrawing.

to be published in Leonardo, August 2002