N90-25528

THE HUMAN FACTORS OF WORKSTATION TELEPRESENCE

Thomas J. Smith"

Human Factors GroupBureau of Mines

5629 Minnehaha Ave. S.

Minneapolis, MN 55417

Karl U. SmithEmeritus Professor

Behavioral Cybernetics LaboratoryUniv. of Wisconsin - Madison

I001 Tower Blvd., Lake Wales, FL

ABSTRACTHuman-operated, remotely controlled robots

(telerobots) are projected to play a pivotal role inthe performance of assembly, maintenance, andservicing manipulation tasks, during constructionand operation of the U.S. space station in the nextdecade. To reap the anticipated benefits oftelerobotic systems---increased safety, efficiency,

and productivity of task performance in space,accompanied by reduced costs---it is essential that

the control requirements for telerobot operation arecompliant with control capabilities and limitationsof the human operator. The term workstationtelepresence has been introduced to describe suchhuman-telerobot compliance, which enables the human

operator to effectively project his/her body imageand behavioral skills to control of the telerobot

itself. This report addresses major human-factorsconsiderations for establishing high fidelityworkstation telepresence during human-telerobot

operation. Telerobot workstation telepresence isdefined by the proficiency and skill with which the

operator is able to control sensory feedback fromdirect interaction with the workstation itself, andfrom workstation-mediated interaction with the

telerobot. Numerous conditions influencing suchcontrol have been identified. This raises the

question as to what specific factors most criticallyinfluence the realization of high fidelity

workstation telepresence? The thesis advanced in

this report is that perturbations in sensoryfeedback represent a major source of variability inhuman performance during interactive telerobot

operation. Perturbed sensory Feedback research overthe past three decades has established that spatialtransformations or temporal delays in sensory

feedback engender substantial decrements ininteractive task performance, which training does

not completely overcome. Similar, more recentlaboratory studies with remote telerobotmanipulators have confirmed in part the earlier

findings. The goal of effective and safeinteractive telerobot operation therefore maybenefit from development of techniques which enablethe interactive computer to detect, and compensate

for, perturbations in sensory feedback beforepresentation of such feedback to the operator. Arecently developed social cybernetic model of human-

computer interaction can be used to guide thisapproach, based on computer-mediated tracking andcontrol of sensory feedback. The report will

conclude by indicating how the social cyberneticmodel can be employed for evaluating the variousmodes, patterns, and integrations of interpersonal,

team, and human-computer interactions which play acentral role in workstation telepresence............................................... .--_-

"This is not an official Bureau of Mines report.

The views expressed are not necessarily those of theU.S. Bureau of Mines.

INTRODUCTION

Automation and robotics (A&R) will play a

pivotal role in the development and operation ofpermanently-manned U.S. extraterrestrial outposts.

One such outpost currently under development by NASAis Space Station Freedom, an orbiting spacelaboratory targeted for assembly in the mid-1990s.Projected A&R applications For the U.S. spacestation encompass over twenty major projects,

including use of interactive teleoperated robots(telerobots) for assembly, servicing, andmaintenance functions []-3]. Telerobots are deviceswith mechanical arms for grasping and manipulationwhich are envisioned to play an integral role in

space station operation. The devices are projectedto have some of the same operational capabilities asan astronaut in a space suit. NASA and Bureau ofMines research also indicates that interactive A&R

systems will be employed in mining lunar resources,in support of a lunar base in the 21st century [4-

8]. Anticipated benefits of interactive A&Rapplications in space include a reduced crew size,increased crew productivity, consequent costsavings, and increased safety of the crew,concomitant with use of telerobots instead of crewmembers to conduct risky extravehicular (EVA)

maneuvers [1,2]. For these benefits to be realized,it is essential that the human-factors design and

operational characteristics of the human-computer/human-telerobot interface be compatible with the

behavioral-physiological performance capabilitiesand limitations of the human operator. In the case

of human operation of telerobots at remote sites

(teleoperation), such compliance across theinterface is described by the term telepresence,

defined by NASA as "a teleoperation situation inwhich the operator has sufficient cues to simulate

sensations that would be experienced in directmanual performance of the operations" [9].

This report provides a conceptual andtechnical analysis of the major human-factors issueswhich must be addressed in creating effective

workstation telepresence during telerobot operation.A major focus of the analysis concerns the

phenomenon of perturbed sensory feedback as adecisive influence on workstation telepresence. Theterm refers to the introduction of some sort of

perturbation, or transformation, in the spatial,temporal, or physical properties of one or moremodalities of sensory input to the human operatoracross the interface. Such perturbations alter the

normal closed-loop control mechanisms of cognitiveand motor perceptual behavior, and can thereby evokesubstantial decrements in human performance during

teleoperation.The principle hypothesis advanced in this

report is that perturbations in sensory feedbackconstitute a major obstacle to the creation of high

fidelity workstation telepresence. As a result of

235

https://ntrs.nasa.gov/search.jsp?R=19900016212 2018-05-13T10:18:14+00:00Z

human-factors limitations of interface design,perturbations in sensory feedback arise routinelyduring human interaction with telerobotic systems.

These limitations may be exacerbated by conditionsprevailing in space environments. To address this

problem, the following sections: (I) provide ahuman-factors analysis of telerobotic workstationtelepresence; (2) outline the adverse effects ofperturbed sensory feedback on interactive

performance for behavior generally, and fortelerobotic operation in particular; and (3) discusshow such effects can be managed operationally, basedon use of a social cybernetic model of workstation

telepresence to target research needs.

HUMAN-FACTORS ANALYSIS OF WORKSTATION TELEPRESENCE

During human performance with a telerobot viaa teleoperation workstation, sensory feedback

(visual, tactile, kinesthetic, and/or auditory) tothe operator is generated by two sources: directinteraction with the workstation, and workstation-

computer mediated interaction with the telerobot.To optimize workstation telepresence, the spatial,temporal, and physical properties of workstation and

telerobot sensory feedback, as determined by thehuman-factors design of the teleoperation system,must be compliant with the sensory feedback controlcapabilities of the human operator.

This concept is illustrated in Figure I, usingthe example of a movement-actuated (master-slave)telerobot manipulator. Effective control of themanipulator depends upon space, time, and force

(kinesthetic) compliance between operationalfeedback from the telerobot slave sector, and motorbehavioral control of this sensory feedback by theoperator. If human-factors design problems causespatiotemporal perturbations in sensory feedback,

then compliance between sensory feedback and its_ control is compromised and performance suffers.

Evidence from various sources suggests that becauseof human-factors design problems, both spatial and

temporal perturbations in sensory feedback are apervasive feature of interactive human-computer andtelerobotic systems [10-14].

Figure 2 expands upon Figure I by

schematically illustrating the major human-factorsdesign issues that must be addressed in creatingworkstation telepresence [11]. Figure 2A shows atelerobotic manipulator under direct control, ratherthan computer-mediated remote control, by the

operator. Integrated postural, transport,manipulative, and tremor movements are employed tocontrol the spatiotemporal properties of manipulatoroperation, in relation to its master-control,

actuator, and slave components (Figure 1). For thecontrol task, the operator relies upon three majorsources of sensory feedback: (1) reactive feedback

From movements of body segment and limb effectors;(2) instrumental feedback from effector interactionwith and movement of the manipulator master-controlcomponent; and (3) operational Feedback from action

of the manipulator slave component and its effect onthe environment.

Figure 2B illustrates how these relationshipsare altered in the case of teleoperation of a remotemanipulator. In this case, control of the

manipulator is mediated through a master-controlcomputer workstation, whose actual design for space

station telerobotic applications is still under

development [15,16]. Direct reactive, instrumental,and operational sources of sensory feedback aregenerated by operator interaction with the

workstation, rather than directly with thetelerobot. Through display of video images fromexternal cameras, plus tactile, force, and possiblyauditory sensors mounted on or near the robot, the

workstation provides indirect, computer-mediatedreactive, instrumental, and operational sensoryFeedback information from activity of the

telerobotic manipulator. Thus, during remotetelerobot manipulation, the operator mustsimultaneously control sensory feedback from twomajor modes of interaction, namely direct

interaction with the workstation plus workstation-mediated interaction with the telerobot manipulator.Collectively, this combination of direct plusindirect, computer-mediated feedback information maybe referred to as teleoresence sensory feedback.

Conclusions From Research on CyberneticAnthropomorphic Machines

One of the first comprehensive conceptual and

technical investigations of human-factorsengineering requirements for effective workstationtelepresence, as defined generally in Figures ] and2, was carried out during the fifties and sixties inthe context of an extensive General Electric

research and development program on cyberneticanthropomorphic (manlike) machines (CAMs) []I],

Figure I. Human factors in control-displayrelationships in a manual manipulator. Normalspatial and temporal visual feedback compliance isessential for optimal performance.

236

....tt

Operaeor

Masfer contro_

Master conh'ol ._ ..r [|

I I[ I• _ _ Operational FB N aclualor

t -............ ......-z___ ........... ...;.-" I I

Figure 2. Sources of visual feedback in direct versus remote operation of a manual manipulator.Reactive feedback is from movement of limb or segment effector. Instrumental feedback is from

effector-mediated movement of master-control component. Operational feedback is from action ofslave component. A. Direct operator interaction with telerobot; B. Computer-mediated operatorinteraction with telerobot.

under the direction of Mosher [17,18]. The broad

objective of the program was to develop mechanicaldevices which would serve as operator-controlled

extensions of the body to expand the strength andendurance capabilities of human performance. The

program dealt with four general types of CAMs: arm-claw manipulators, bipedal walking machines,

quadripedal walking machines, and ambulatoryexoskeletons which enclosed the operator within

body-like frames. These were true teleroboticdevices, in that they each comprised master control,actuator, and slave subsystems which provided

computer-mediated, servo-controlled spatial,temporal, and force feedback compliant with allmodes and patterns of motorsensory feedback from thehuman operator's articulated arm, hand-wrist, leg,

foot, and torso movements.To measure system relationships between

operator performance and machine feedback parameters(such as device position, servo gain, and force-feedback ratio), human-factors studies of the four

types of CAMs were carried out. These studies were

guided by the general block diagram of an operator-computer-telerobot system illustrated in Figure 3.Figure 3 is based on the original design for a CAMsystem [18]. However, the figure has been adapted[11] to show a more detailed breakdown of the

operator-workstation-telerobot interactionsschematically illustrated in Figure 2B, includingspecifically the sources of reactive, instrumental,and operational sensory feedback provided to the

operator from the master-control, actuator, andslave subsystems. The figure also has been modifiedto conform to current specifications [31,32,64] for

the Space Station Flight Telerobotic Servicer(SSFTS), to be discussed further in a subsequentsection.

Figure 3 emphasizes the complexity of sensoryfeedback relationships which must be built intotelerobotic systems to provide fully compliant force

and position feedback for effective telepresence.Force feedback to the operator is mediated by

kinesthetic (also termed proprioceptive)mechanoreceptors in the muscles, tendons, and

237

joints, plus tactile receptors in the skin.

Collectively, kinesthetic and tactile receptorsprovide muscle-tendon-joint sensibility, which

encompasses pressure, movement position and dynamics(occurrence, locus, direction, velocity, andacceleration of movement), temperature, and pain[]g, Chap. 14].

Mosher [17,18] established that four differentdimensions of force feedback are needed to control

body movement accurately, and should therefore be

incorporated into a telerobot control system: (])force of movement at different levels of exertion;(2) upper limit of force required; (3) prediction of

force needed to execute movements of a givenvelocity and power; and (4) detection of relativedisplacement between movement and its sensory

feedback. A series of system design and operationalfactors which influence the muscle-tendon-jointsensibility of the operator were identified,

including Force feedback ratios between operator androbot, drift on bias forces, friction thresholds,nonlinearity of force interactions, saturationfactors, and force-signal integrity as influenced bysystem kinematics. As part of this human-factors

system analysis, Mosher also demonstrated that forcefeedback related to the limits of exertion of

powered grips, lifts, and pulls must be built intotelerobots as a safety precaution to prevent their

destructive action, a Finding with implications forpresent-day robotic systems [20].

Although theoretical and experimentaldemonstration of the fact that force feedback is

needed to guide and control telerobotic machines wasfully developed and confirmed by the work of Mosher

(17,18), research conducted by the senior author ofthe present report established that effectiveworkstation telepresence could not be achievedwithout also introducing spatial compliance insensory feedback control of telerobotic action

[11,2],22]. Such compliance occurs only when theposition feedback system designated in Figure 3embodies several spatiotemporal parameters of

sensory feedback, namely position, direction,relative extent, range, velocity, acceleration, andform or pattern of telerobotic movement.

As described in the next section, the roles ofspatiotemporal sensory feedback factors in humanmovement control were established in the course of

an extensive program of experimental research on theeffects of spatially and temporally perturbed

sensory feedback on human behavior and performance[14, 23-29]. Findings from this research

...... ..............................................

VISUALSENSORY

RECEPTORS KINESTHETIC 1

TACTILE

REAC]IVEFCEDBACK

: !

: i. i

:. =, i

MUSCLE ACTIVITY

MOTOR CONTROL

{BODY POSTURE;TRANSPORT ANDMANIPULATIVE

MOVEMENTS OF

ARM AND HAND)

CENTRALNERVOUS

SYSTEM

COGNITION ,'PERCEPTION

WORKSTATION

COMPUTER

WORKSTATION

CONTROL PANEL

POSITIONING DEVICE

DYNAMICS

WORKSTATION

DISPLAY

=

FC#_CEFEEDBACK SERVOS _:

TELEROBOTIC

CAMERAS

EXTERNAL

CAMERAS

I

I

1i

...... .....TASK BOARD

HARDWARE/SENSOR

ONBOARD TILLUMINATION

POSFFIONFEEDBACK SERVOS

MANIPULATORPOSITION H MANIPULATORACTUATORS DYNAMICS

TASKEXECUTK)N

i

MANIPULATORSERVOSFORCE _ MANIPULATOROUTPUTFORCE I

Figure 3. Block diagram of telerobot control system, indicating interactive roles of the

operator, workstation, telerobot, and video cameras in system performance. Visual, position,and Force feedback to the operator combines reactive, instrumental, and operational feedback

generated by the master-control, actuator, and slave components of the system (Figure 2). TheFigure has been modified (from [I],]8]) to include vision system components For a space telerobot[32].

238

demonstrate that all body movements are guided andcontrolled primarily by spatial feedback factors,with different movement systems mediated bydifferent properties and dimensions of spatialfeedback control. In general, tactile-kinestheticmechanisms (for velocity, acceleration, and positioncontrol) are integrated with the visual andvestibular mechanisms (for control of position,direction, range, extent, and pattern of movement inspace) in mediating the overall organization,predictive guidance, and integration of everymovement pattern of the body.

Several characteristics of Mosher's originalCAMs point up the importance of full spatialcompliance for effective telepresence. For example,one problem that emerged in the design of thebipedal walking machines was inability of theoperator to determine whether the machine was goingdown an incline versus falling backwards (or goingup an incline versus falling forwards). To resolvethis problem, Smith [21] demonstrated, first, thatthe operator's normal posture had to be maintainedwith respect to leg motion, and second, that theoperator had to be able to sense any displacement ofeither leg of the machine with respect to theposition of the machine cab. It was concluded thatthese cab-leg displacements had to be sensed by theoperator via active force and position feedback.

After appropriate torso-leg/cab-leg servo feedbackmechanisms were incorporated into the design of thewalking machine, it was found that even a noviceoperator could effectively control the dynamicbalance of the machine.

A second example of the importance of spatialfactors in telepresence emerged with research onmanipulator CAMs. Tests showed that the operator ofa manipulator experienced extension of his/herdynamic body image as long as just the manipulatorwas moved, but that this sensation was lost when themanipulator claw was moved. This loss oftelepresencewas related to the fact that the graspactions of the claw were not spatially compliantwith the operator's hand movements, although theforce compliances were very adequate.

The two examples cited above demonstrate thatbody image is defined in large part by the spatialorganization of body movement, and that the skill,precision, and safety of telerobotic operation areenhanced when the human operator perceives robotmovements as an extension of his/her own movements.The general conclusion is that telerobotic designmust incorporate compliances of both spatial andforce factors betweenoperator movements and machinemovements before effective workstation telepresencecan be established.

Although this conclusion is based on CAMresearch dating back two to three decades[17,18,21,22], we are not aware of more recentevidence which contradicts the insights it providesinto the human factors of workstation telepresence,or that superior approaches to telerobotic designand control have been devised. Indeed,a recognized

expert in the field recently referred to Mosher'smachines as still among the most sophisticatedtelerobots yet developed [30].

EFFECTS OF PERTURBED SENSORY FEEDBACK ON INTERACTIVEPERFORMANCE

The remainder of this report summarizes

scientific evidence from perturbed sensory feedbackresearch which indicates why spatially compliantfeedback is critical to workstation telepresence,and explores the implications of this evidence foroptimizing the human-factors design of interactive,telerobotic systems. Scientific interest in thephenomenon of perturbed sensory feedback dates backover a century to the early work of Helmholtz [33],who studied the effects of and adjustment to spatialdisplacements in visual feedback. Since then, theeffects on human performance of both spatial

perturbations and temporal delays in visual andauditory feedback have been the subject of numerousinvestigations, whose findings are summarized in aseries of reviews published over the past threedecades []4, 23-29,34].

In this report, the conceptual approachadopted to interpreting the performance effects ofperturbations in sensory feedback is grounded inbehavioral cybernetic theory. The basic postulateof this theory is that behavior is a closed-loop,self-regulated process mediated by motor control ofsensory feedback (12,14,23-29]. From a behavioralcybernetic perspective, sensation and perception arefeedback integrated with the motor system throughmotor coordinate feedback control of receptorprocesses and environmental stimuli. During taskperformance, the individual uses movement-basedmechanisms to control spatial, temporal, or othertypes of perturbations in sensory feedback which mayarise as a result of the human-factorscharacteristics of task design.

Since Helmholtz first posed the question, animportant issue in psychology has been whether thespatial organization of behavior is innate orlearned through experience. Although experientiallearning appears to definitely contribute to thespatial guidance of behavior, experimental findingsindicate that complete adaptation to perturbationsin sensory feedback rarely occurs (]4,23-29,34].Instead, evidence indicates that performance andlearning remain suboptimal in the face of persistentexposure to a perturbation condition. This suggeststhat the nature and degree of learning andperformance proficiency achieved for a particulartask depend upon the nature and degree ofspatiotemporal compliance between sensory input andthe motor feedback mechanisms controlling that inputduring task execution.

Figure 4 provides a diagrammatic illustrationof the concept that workstation telepresence iscritically dependent upon spatial compliance betweensensory feedback and its motor control mechanisms,using the same master control-slave telerobot modelas in Figure I. In Figure 4, however, visualfeedback has been spatially perturbed by reversingthe lateral motion relationships between arm-controlled movement of the control component andderivative movement of the slave component.Consequently, the operator experiences aperturbation in the normal manner in which visualfeedback from the te!erobot manipulator iscontrolled by arm movement, telepresence iscompromised, and performance suffers.

In general, four major classes ofperturbations in sensory feedback may be identified,namely: (]) spatial transformations and/ordisplacements; (2) temporal delays; (3) changes inenergy properties; and (4) modality conflicts. Forexample, control of visual feedback from a video-

239

/

/ // I!i

r/.,">,/ li:

Co!lll_l I

Coi11pOil ez_t ._iO =_r_ _

Figure 4. Spatial perturbation (reversed visualfeedback) in visual-manual operation of a telerobot

manipulator. Spatial compliance between movementsof telerobot slave components, and control movements

of the operator, represents one of the key humanFactors of workstation telepresence.

Figure S also assumes that visual-manualperformance is based upon use of the motor system

(19 principle motor control factors specified) tofeedback control the visual system (11 principlevisual perception factors specified), involving 5distinct modes of sensory feedback control [14,23-

2g]. This control process potentially may beinfluenced by a broad range of space telerobotic

system design factors (18 principle designconsiderations specified), some of which may giverise to the perturbation conditions indicated in the

figure. The system design, visual perception, andmovement control factors specified in Figure 5 are

compiled from basic human-factors considerationsregarding telerobotic operation [11], plus recent

analyses by various scientific specialists in spacetelerobotlc research [32,38-4I].

Some appreciation of the scientific challenge

confronting systematic investigation of the humanfactors of workstation telepresence may be gained by

considering that there are over 2.3 million possiblecombinations of performance, design, and feedbackfactors (14x3x3xlgxI1xSx18) which potentially caninfluence operational variability in vlsual-manual

control of a space telerobot, based onspecifications in Figure 5 and summarized above.That it probably is not feasible to completelyevaluate even a reasonable subset of such possiblecombinations raises the question of research

priorities and emphasis in human-factors analysis ofperformance variability in telerobotic operation.The hypothesis adopted in this report is that

spatiotemporal perturbations in sensory feedbackconstitute a major source of variability in humanperformance with automated and telerobotic systems,and that this issue therefore merits attention byhuman-factors research aimed at optimizing

workstation telepresence. Evidence in support ofthis hypothesis is summarized in the next section.

displayed image during remote telerobotic operationmay be perturbed in each of these ways by: (1)

inversion, reversal, angular displacement,magnification, or miniaturization of the displayedimage; (2) feedback delay of the real-time display;(3) modulation of display quality due to glare,brightness, reflectance, contrast, and/or frequency

shift effects; or (4) auditory and/or kinestheticinterference with visual feedback control. In

extraterrestrial environments, microgravityrepresents a fifth class of sensory feedback

perturbation which may exacerbate the effects ofother types of perturbations [2,35-37].

Figure 5 expands upon Figure 4 with aschematic summary of the major sets of factors which

potentially influence visual-manual task performanceand workstation telepresence during operatorinteraction with a space telerobot such as theSSFTS. The diagram indicates that interaction of

the operator (to the right) with the teleroboticsystem (to the left) across the operator-systeminterface is influenced by one or more of 14

possible perturbed sensory feedback factorsspecified in the figure and cited above. Theoperator-system interface specified in Figure 5assumes the feedback relationships depicted in

Figures 2 and 3, namely 3 sources of feedback(reactive, instrumental, operational) from the 3telerobotic subsystems (master-control, actuator,

slave).

Experimental Observations On Perturbed SensoryFeedback Effects

Over the past three decades, the most

extensive program of experimental research onperturbed sensory feedback phenomena has beenconducted by K.U. Smith and colleagues [14,23-

29,34]. This work has focused primarily upondelineating the effects of spatial and temporalperturbations in sensory feedback on learning and

performance. For this purpose, computer- and video-based techniques are used to introduce controlledspatial transformations or displacements, and/ortemporal delays, in sensory feedback (usually visualor auditory) provided to subjects of their own task

performance. A wide range of different cognitiveand motor behavioral skills have been examined in

this manner, including machine performance, tool

using, musical performance, reading, writing and

drawing, tracking and steering, speech, memory,eyemovement control, visual perception, motorcoordination, postural control, behavioral-

physiological integration, and social interaction[28,2g,42,43].

With its use of computer and video-display

techniques, this research essentially represents the

application of interactive, human-computerlaboratory methodology to the study of howperturbations and distortions in sensory feedback

240

influence learning, performance, and workstationtelepresence. However, rather than arising as anincidental and unwelcome concomitant of human-

factors defects in interface design, suchperturbations are introduced deliberately by thecomputer under controlled laboratory conditions, sothat effects can be assessed in an objective andquantitative fashion. The research approachtherefore provides a methodological paradigm forsystematic study of the contribution of sensoryfeedback perturbations to variability in humaninteraction with automated and telerobotic systems,in relation to human-factors design features of theinterface [II,14].

The general conclusion from the extensive bodyof research conducted using this approach is thatthe performance and learning of every behavioraltask so far examined is degraded by perturbations insensory feedback [14,28,29]. Tables I and 2 providerepresentative data on the nature and extent ofthese effects, for spatial transformations andtemporal delays in sensory feedback respectively.All data are derived from controlled laboratorystudy of subject interactionwith computer-or video-based systems. In both tables, performance changesfor a series of tasks under specified perturbationconditions are summarized, along with informationregarding experimental conditions and referencecitations (first column).

Under spatially perturbed visual feedbackconditions, results for various measures of sevendistinct types of tasks are given in Table I (thirdcolumn), namely writing, drawing, reading, assembly,panel control, tool using, and tracking tasks. Thespatial perturbation conditions examined (secondcolumn) encompass inversion, reversal, combinedinversion and reversal, angular displacement, and

size reduction (miniaturization)of visual feedback.The measures used to assess performance (fourthcolumn) vary for different tasks, but generally fallinto the categories of movement characteristics(contact, travel, grasp, or assembly times), orperformance accuracy. Results (last column) areexpressed as percent change in performance, from thecontrol to the spatial perturbation condition, atthe start and at the end of a training period (fifthcolumn) which typically involved one or more trialsessions per day over a series of days. An asterlckis used in the last column to denote the statistical

significance of the performance change observed.Results for the feedback delay studies in

Table 2 are summarized in much the same manner.These studies examined writing, drawing, tracking

(manual, eye, and head movements), speech, memory,and interactive social tracking tasks (secondcolumn), with performance again assessed usingeither movement characteristics or accuracy measures(third column). Results (last three columns) arepresented as the change in performance, for delay ineither visual or auditory feedback, at delayintervals of 0.2 seconds, 0.4 seconds, and the

RCEPTIO_

Figure 5. Human factors which may influence workstation telepresence in space teleroboticoperation. Operator performance involves motor control (movement and sensory feedback controlfactors) of visual feedback (visual perception factors), which is influenced by both systemdesign and perturbed visual feedback factors.

241

maximum delay interval examined in the study,relative to performance under the no delaycondition. Performance change values for the delayconditions are expressed as multiples of the zero-delay levels. In all of the studies cited in Table2, analysis of variance was used to assess theoverall contribution of the delay condition to theperformance variability observed. In every case,the overall contribution was found to bestatistically significant, but tests of significanceat individual delay levels were not performed.

Results summarized in Tables ] and 2, plusfindings from an extensive body of related research,support the following conclusions regarding theeffects of spatial and temporal perturbations insensory feedback on interactive performance [i4,23-29].

1. Performance and learning of every behavioraltask so far examined ate degraded by suchperturbations. Consistent patterns and features ofbehavioral disturbance appear from study to study.Oscillatory instability in movement control becomesmore pronounced, accompanied by increasedvariability and extremes in movement velocities andaccelerations. Accuracy of movement guidance andtracking declines. Perception is disturbed and, insome cases, may disappear entirely; there is aconcomitant deterioration in learning. Subjectsreport feeling confused, uncertain, and/oruncomfortable about their behavior. Skilled

performers are particularly sensitive to theseeffects.

Table I. Changes Inlnteractive Performance and Learning Under Spatially PerturbedVisual Feedback

Reference-Sub.iects

[44!-12n

M

i!

||

[4s!- 4N

||

[241 - lo(p. 1631

I!

w!

||

PerturbationCondition z Task Measure

I .Hand- .Contact

" writing Time" .Travel" " Time

" .Drawin9 .Contact

" Triangles Time" .Travel" " Time" .Reading .Amount" Words Read

I .Reading .Time x• " ErrorR " Score

I-R .Assembly .Grasp" " DurationN I!

11 I!

" " .Assembly" " Duration" " .Nonloaded" " Travel

Duration

24] - 24 I

p..1711 RI-R" I

" R

" I-R" I

" R" I-R

" R

" I-R

[24] - 24 _ I(p. 172) R

W I-R" I" R" I-R

Days ofTraining Performance(Trials)3

0 (I) + 676 % *20 (20) + 153 *0 (I) + 410 *

20 (20) + 80 (1) + 493 *

20 (20) + 71 *0 (I) + 583 *

20 (20) + 330 (1) 81 *5 (5) 12 *

0 (I) + 253 *

5 i 5) + 210 1) + 210 *5 (5) + 6

0 (I) + 5 *4 (4) + 4 *

+ 24 *+ 6

0 (I) + 92 *4 (4) + 60 *0 (I) + 27 *4 (4) + 8

.Loaded Tra- 0 (I)vel Duration 4 (4)

.Hand- .Contact 2 ! 4) + 286 *writing Time + 154 *,' • " + 250 *" .Travel " + 240 *" Time " + 160 *" " " + 200 *

.Drawin9 .Contact " + 171 *Trlangles Time " + 80 *

" " " + ]05 *" .Travel " + 139 *" Time " + 72 *" " " + 94 *

.Haze .Accuracy 0 ! 1)Tracing . ,,

" " 3(91ml I!

72 *51 *3g *37 *15 *25 *

242

Table 1. (continued)

Days of

Reference- Perturbation Training Performance

Subjects i Condition 2 Task Measure T_Trials)3 _e(%__

[24] - 18 .Size .Panel .Contact 0 (I) + ]08 *(p. 218) Reduction Control Time 5 (40) + 104 *

" (to I/3 size) " .Travel 0 (I) + 232 *...... Time 5 (40) + 267 *

[24] - 24 I Tool Using:(p. 239) " .Dial .Manipulation 0 (I) + 273 *

.... Setting Time 7 (21) + I]7 *

.... .Turning " 0 (1) + 191 *

....... 7 (21) + 78 *.... .Pushing " 0 (1) + 200 *....... 7 (21) + 100 *

.... .Pressing " 0 (1) + 250 *

........ 7 (21) + 117 *

.... .Dial .Travel 0 (1) + 257 *

.... Setting Time 7 (21) + 29 *

.... .Turning " 0 (I) + 209 *

........ 7 (21) + 40 *

.... .Pushing " 0 (I) + 218 *

....... 7 (21) + 36 *

.... .Pressing " 0 (I) + 230 *

........ 7 (21) + 40 *

[46] - 24 AD:

" 20 degrees .Maze .Errors 0 ! 1) + 6

" 40 degrees T_acing " + 5" 60 degrees .... + 23 *

" 10 degrees " " 10 !10) - 2" 20 degrees .... 9" 30 degrees ', " " + 2" 40 degrees ..... + 7

" 50 degrees ...... + 63 *" 60 degrees ..... + I48 *" 70 degrees ...... + 287 *

................................................ .................... - ...... _ .................

*Change from nonperturbed visual feedback performance level is statistically significant

(p<.05).IReference in brackets, followed by number of subjects.

21 = Inversion; R = Reversal; I-R = Inversion-Reversal; AD = Angular Displacement (in

degrees).3Number of trials in parentheses.4Performance change, in percent, calculated as:[(perturbed viewing level - nonperturbed viewing level) x 100] / (nonperturbed viewing level)

2. For untrained subjects, spatial perturbations in of 43 cases in which training was provided for 2visual feedback have profound effects on task days or longer. A similar pattern has been observed

performance. In Table I for example, there are in experiments where subjects were exposedstatistically significant performance decrements for continuously to spatially perturbed visual feedback24 of 27 no training cases (fifth column, trial 1, for periods of time lasting days or weeks [24,34].

0 days training), and in 20 of these cases theperformance decrement exceeds 50 percent. 4. The training findings summarized in Point 3

suggest that human learning is refractory to

3. After training, subjects typically exhibit complete adaptation to spatial perturbations inimproved performance under spatially perturbed visual feedback. Neurobiological corroboration of

visual feedback conditions. In Table I for example, this conclusion is provided by recent researchthere are 27 instances in which pro- and post- [52,53] on motor learning in the primate vestibulo-

training performance levels can be compared, and ocular reflex (VOR). Under continuous exposure toperformance improved in 25 of these cases, size distortion (magnifying or miniaturizing

Nevertheless, with rare exceptions, under spatially spectacles), both the reflex gain and activity ofperturbed visual feedback subjects do not perform at brain stem neurons mediating the reflex adjustthe same level as under nonperturbed conditions, accordingly. However, the gain error increases fromeven after a training regimen lasting as long as 20 about zero, with no distortion, to 10-20 percentdays. In Table I for example, statistically under the distortion condition. This lack of

significant performance decrements persisted for 33 complete adaptation to size perturbation may explain

243

Table 2. Changes in Interactive Performance Under Feedback Delay I

Reference-

Subject@ z

[231- 2

II

It

[47]- 8

[48]- 8

[49]- 12

[50] - 8

[S]] - 2

Task Measure

.Writing .Contact TimeLetters

.Drawing "

.Star Tracing "

.Maze Tracing "

.Head Movement .TrackingTracking of ErrorVisual Target

Performance Change (xl)at Specified Delay 3

0.2 sec 0,4 @eC Max (sec)4

+ 2.4x (.52)

+ I.Ix (.52)

+ 4.6x (.52)+ 5.0x (.52)

+ ].3x + 1.6x + 2.3x (0.8)

[auditory delay interval s : O.I secl 0.2 secl]

.Speech .Speech Errors + 4.]x I +]1.]x I +14 x (0.3)

Tracking of Visual Target:

.Eye Movement .Tracking. + ].2x + ].2x + 2.0x (1.6)

.Head Movement Error + 1.2x + 1.2x + 5.0x (1.6)

.Head-Eye " 0 x + 1.2x + 1.9x (1.6)Movement

.Memory of .Recall + ].Sx + 1.9x + 3.3x (0.8)Visual Image Error

.Visual-Manual .TrackingTracking Error

-Individual: " + 1.]x + 1.5x + 2.6x (].5)-Social : " + ].5x + 1.8x + 3.0x (].5)

.............................................................................................

IAll studies examined visual feedback delays, except for Abbs and Smith [48], who examinedauditory feedback delay.

?Reference in brackets, followed by number of subjects.

3performance change, in multiples of one (indicated by small x), observed at delay intervalsof 0.2 sec, 0.4 sec, and maximum delay interval examined _. Performance change is calculatedas:

(Delayed Performance LeveIJ / (No Delay Performance Level)4Performance change, in multiples of ones, at maximum feedback delay interval examined in

study. The maximum delay interval examined, in seconds, is indicated in parentheses.SDelay intervals of 0.I sec, 0.2 sec, and 0.3 sec examined in this study.

the finding cited in Table 1 that performance byhuman subjects in an interactive panel control taskremained significantly impaired after 5 days of

training with use of miniaturizing spectacles thatprovided a two-thirds size reduction [24, p. 218].

5. Three experiments described in Table I [24, pp.171-172; 45] directly compared the effects ofinversion, reversal, and combined inversion-reversalof visual feedback. The results indicate that

inversion has the most adverse impact onperformance, followed in order by combinedinversion-reversal, and reversal alone. Other

research also supports this conclusion [25].

6. Under conditions of angular displacement of

visual feedback, as indicated in the last studycited in Table I [46], performance remainsrelatively unaffected until a displacement angle of

50 degrees is imposed. This finding has beeninterpreted as providing evidence for a breakdown

anqle of spatial displacement, beyond whichperformance becomes progressively more difficult.Typically, the breakdown angle falls in the range of40 to 50 degrees displacement. Results in Table I

indicate that even after IO days of training,performance under displacement conditions exceeding

the breakdown angle remains significantly impaired.

I. Relative to direct viewing conditions,performance on an assembly task is impaired undertelevision viewing of the assembly operation, evenwhen no deliberate spatial distortions are

introduced [24, pp. 161-163]. This may beattributed to inherent spatial distortions in visualfeedback introduced by the video camera and displaysystem.

8. Performance under spatially perturbed visualfeedback varies as a function of both age and gender[45,54].

9. Temporal perturbations (feedback delays) insensory feedback also result in consistent

decrements in performance, as illustrated by thefindings cited in Table 2. Performance decrements

at a delay interval of 0.2 sec occur for eight ofnine tasks listed in the table. At delay intervalsof 0.3 sec and greater, all 13 tasks listed in the

table manifest performance decrements. For speech,a fourfold increase in errors is observed at an

auditory feedback delay of O.I sec [48].

244

10. Trainingeffect dataarenot includedin Table2 becausethe studiesthat havebeendone(onbotheyemovementtrackingandmemory)indicatelittle orno performanceimprovementwith learning underfeedbackdelayconditions[47,55].11.Whenvisual feedbackis bothtemporallydelayedandspatially reversedin either aneye-or a head-movementtracking task, the delayexacerbatesthedecrementin trackingperformanceproducedby thereversalcondition[47,55].12.Socialtrackinginvolvesthemutualexchangeandcontrol of sensoryfeedbackamongtwo or moreindividuals during group activity [13,28,56].Salientfindingsregardingsocial trackingarethat:(a) relative to visual-manualsocial tracking,accuracyis higherfor tactile or auditorysocialtracking {57]; (b) social tracking accuracyiscomparableto individual tracking accuracyexceptunderfeedbackdelayconditions,whenthe latter issuperiorto theformerat all delaylevels [5]]; (c)learningof socialtrackingtasksis limited, highlyvariableandinconsistent,andunstable[58,59];and(d) thepreponderanceof performancevariability insocial trackingis attributable, not to a learningeffect, but to human-factorscharacteristicsof taskdesign{60].

SpaceTeleroboticTelepresence OperationalImplicationsof Perturbationsin SensoryFeedback

A majorassumptionof this report is that thefindingsandconclusionssummarizedin theprecedingsection have direct relevanceto understandingpossibleeffects of perturbedsensoryfeedbackoninteractive performance in extraterrestrialenvironments.Forexample,manyof thetaskslistedin Tables] and2 (reading, tracking, assembly,panelcontrol, tool using, speech,memory,socialtracking) are employedduring interactive humanperformancewith spacetelerobotic systems.Smithandcolleagues{61] haveaddressedthis question,andconcludethat perturbedsensoryfeedbackeffectscouldhavea potentially decisiveinfluenceon theperformanceof useful workwith man-in-the-loopspacetelerobotics. Someexperimentalevidencefromearth-based,human-factorsresearchon teleroboticsystemslends support to this conclusion,assummarizedbelow.

Establishmentof high fidelity workstationtelepresencefor spaceteleroboticoperatorswillrequire that inappropriateor hazardoussensoryfeedbackfromthe spaceenvironmentbe eliminated(i.e., direct sunlight, glare, cold, lackof air),whereasvisual, tactile, kinesthetic,andauditoryfeedbackessential to manipulativeperformanceisprovidedwhichconformsto thecontrol capabilitiesof the operator. Duringperformanceof a remotemanipulationtaskwith a telerobotin space,it maybe possible to view the remotescenedirectlythroughawindow.Formosttasks,however,a directview either will not be available, or will notprovidethe appropriatevisual cuesnecessaryforteleoperation(dueto distanceand/orinterpositionof other objects). For this reason, externalcameraswill providethe primarysourceof visualfeedbackaboutmanipulatorposition, orientation,andrate of movement[62].

Variousstudiesof viewingsystemrequirementsfor space telerobotic systemsindicate thatperturbationsin Visual feedbackmayrepresentapervasivefeatureof teleoperation. A numberofauthorsagree[38-41]that visualguidanceof spacestation telerobotic tasks will be influencedbyextreme illumination, contrast, glare, andreflectanceconditionsknownto exist in space. Inaddition,recentresearchbyR.Smithandcolleagues[32,39,62,63]hasaddressedthe questionof howspatial perturbationsin visual feedback,producedbycamera-inducedtransformationof themanipulatorreferenceplanerelative to that of the operator,could degradetask performanceof the telerobotoperatorandtherebyrisk damageor loss of themanipulatoror payload. In particular, spatialperturbationsof visual feedbackto the operatorfroma spacetelerobotcanbepredictedasa resultof: (]) variable orientations of surveillancecamerasonthe stationgridworkandmodules,andonthetelerobot; (2) lackof a fixedgroundreference;(3) abnormalor missingdepthcues;and(4) extremeviewingconditions,asnotedabove.

Decrementaleffectsof spatial transformationsin visual feedbackon remoteoperation of atelerobothavebeenconfirmedin arecentlaboratorystudy by Stuart and Smith[63], whoshowthattelerobotmanipulationtimesare: (1) significantlyslowerfor video-displayviewingof the worksite,relative to direct viewing;and(2) slowedmostbyinversion of visual feedback,with reversalandinversion-reversalconditionshavingless of aneffect. In particular, theseresearchersfoundthatperformancecompletiontimes in a manipulatorpositioning task wereincreasedby the followingamounts,relative to direct viewingof the task:(I) display viewing,2.0-fold increase(no otherspatial perturbation); (2) inverted-reversedviewing,8.5-fold increase;(3) reversedviewing,lO.2-foldincrease;and(4) invertedviewing,16.1-fold increase. As indicatedin TableI, thesefindings recapitulatethe earlier observationsofK.U.Smithandcolleagues[24], in whichthe sametechniquesof videocameradisplacementandrotationwereusedto spatially perturbvisual feedbackofthe taskprovidedto sub.iects.

Alsorecapitulatingearlier observations[23]are results fromrecentresearchonoperationof alaboratoryversionof a telerobot servicer []5],whichdemonstratethat delaysin visual feedbacksignificantly increasemanipulationperformancetimes. The earlier researchdocumentedthedetrimentaleffectsontrackingperformanceof delayintervalsrangingfromoneto threeseconds(in thesamerangeasthe round-triptransmissiondelayfora signalfromanearth-boundoperatorto anorbitingor moon-basedtelerobot), andwasamongthefirst topoint out that effective guidanceof a remotetelerobotwouldbeseriouslycompromisedby delaysin visual feedback[23, pp. 94-96].

A further important consideration inestablishingtelerobotworkstationtelepresenceisthat of sensoryfeedbackmodalityandits control.Forexample,in anexperimentalstudyof visual andtactile social tracking[25], it wasshownthat anobservercantrack anirregularlymovingobjectmoreaccurately by touching it (tactile-kinestheticfeedback)than by watchingit (visual feedback).The difference is attributable to the slowerfeedbackrelationships of the visual systemas

245

compared with those of the tactile-kinesthetic

system. There also is the question of modalitycompatibility between sensory feedback and sensoryfeedback control. For example, feedback of forceinformation from a telerobot force transducer to a

hand controller, rather than as visual or auditoryfeedback, is compatible with human kinesthetic

senses that are directly interpretable.An appreciation of the potential for

spatiotemporal perturbations in sensory feedback

during SSFTS operation may be gained from aconsideration of current system requirementspecifications [31], as embodied in the general

system diagram in Figure 3. Vision systemspecifications call for at least four telerobot-mounted video cameras, including one on themanipulator wrist, with separate positioning,orientation, zoom, focus, and aperture control

capabilities indicated for each camera. Variablecontrol of attitude and imaging parameters for boththe telerobot-mounted and external cameras, coupled

with possibilities for variability in task board andtelerobot manipulator geometry, introduce avirtually unlimited potential for reversal,inversion, angular displacement, magnification,

and/or miniaturization (Figure 5) of the displayimage presented as visual feedback to the humanoperator.

SSFTS manipulator specifications call For

kinesthetic feedback of force, position, and rateinformation from sensors at each joint [3],64]. Forpurposes of task execution, the specifications alsocall for an orientation accuracy of less than ].0inch in manipulator tool plate center position,

repeatable to an accuracy of less than 0.005 inch.Howe_2r, findings summarized in Tables I and 2, plusobservations from more recent perturbed sensoryfeedback research on telerobots [15,63], suggestthat such accuracy may be compromised by theoccurrence of any type of spatiotemporal

perturbation in visual feedback. Under earthlaboratory conditions, such perturbations causeerrors in movement trajectory guidance plusincreased oscillatory instability in staticpositioning control [23-2g,64]. In addition,experience from earlier work on Mosher's CAM systemsJill, summarized previously, suggests that it may be

necessary to supplement kinesthetic and visualfeedback with tactile feedback From manipulator endeffectors in order to achieve a high degree ofreproducible accuracy in control of manipulator

positioning.

OPTIMIZING WORKSTATION TELEPRESENCECYBERNETIC PERSPECTIVE

A SOCIAL

The analysis in the preceding sectionssuggests that effective workstation telepresencedepends on provision of compliant sensory Feedbackacross different modal]ties which can be readily

controlled by the operator. Because they cancompromise such compliance, perturbations in sensoryfeedback may significantly impair the operationaleffectiveness of space telerobotic systems, inrelation to visual-manual control of manipulator

functions. This raises the question as to whatstrategies might prove useful in reducing possibleadverse effects of sensory feedback perturbations on

human performance in space, in order to achieve high

fidelity workstation telepresence. Training is oneobvious choice. However, evidence cited in Tables

] and 2 suggests that even extended training willnot completely overcome these effects. It maytherefore prove desirable, or even necessary, to

develop techniques which enable the interactivecomputer to detect, and then correct, perturbations

in sensory feedback before presentation of suchfeedback to the human operator.

The idea of enlisting the workstation computeras an intelligent co-participant in optimizingworkstation telepresence is in line with proposalsby various authors that the design of spacetelerobotic systems should evolve from primaryreliance on operator control to reliance on both

operator and autonomous, computer-mediated (robotic)control of task execution [1-3,65-67]. To providea conceptual foundation for dealing with the humanfactors of workstation telepresence at various

levels of interactive complexity, we have developeda social cybernetic model of human-computerinteraction []2-14]. The model, shown in Figure 6A,assumes that behavioral cybernetic principles ofhuman social interaction [28,56,58-60] also can beapplied to an understanding of human-computerinteraction.

The term social tracking used in Figure 6Arefers to the feedback-controlled process by whichan individual engaged in social behavior follows or

tracks one or more social targets. During socialtracking, the activities of one person in a socialgroup effect behavioral-physiological changes in theother social partners, whose own activities in turnhave behavioral-physiological feedback effects onthe first. These effects arise as a consequence of

control by each participant of sensory feedbackgenerated by the other social tracking partners.During interpersonal social tracking for example,the movements of one individual generate sensoryFeedback that is controlled by tracking movements of

the social partner, whose actions generate compliantsensory feedback for the first individual, who inturn tracks this social feedback with further

movement, and so forth. The two social partnersthus become dynamically yoked or interlocked

behaviorally and physiologically, through mutualtracking and control of sensory feedback generatedby each other's social behavior. Through such

social tracking interlocks, participants in a socialgroup begin to operate as an integrated system, withdefinite systems feedback parameters and controlcharacteristics.

The premise of the model in Figure 6A is thatthe interactive computer can be considered as a

machine analog of a human social partner, and can beimbued through adaptive programming with

capabilities For tracking and controlling, acrossdifferent modal]ties, sensory feedback generatedboth by its own activities as well as those of itshuman partner. Unfortunately, because oftechnological and design limitations in such

capapabilities, today's interactive computer systemis a relatively limited and impoverished socialtracking target for the human operator. _roaalyspeaking, the creation of effective workstationtelepresence requires that such limitations beovercome.

A number of authors have suggested that thequality and nature of crew social interaction are animportant determinant of operational effectiveness

246

COMPUTER INTERACTION PER$ NIcOyOTE.I ,NTERACT,ONI O 1L. I _ S'e_,oryI

Ic uu1_l Oulpul Feedback Sensoryeedbad_ Feedback Con o Feeo'oad( oo,,o,J• T U

I °°'°°'°'"°' 1

SOCIAL CYBERNETIC

TRACKING MODEL

F'S "YUoTUADL

Figure 6. The social cybernetics of workstation telepresence. A. The social cybernetic modelof human-computer interaction, which assumes mutual social tracking by operator and machine ofsensory feedback generated by the other partner during interactive performance. B. Possiblesocialcybernetic interactionsduring SSFTSoperation. The diagram assumes that the SSFTS (whichmay be controlled by more than one operator) is conveyed to the task site by an OrbitalManeuvering Vehicle (OMV), controlled by a separate operator. The arrows symbolize possiblesensory feedback exchange and control relationships between operator and machine participantsin the control system. Also indicated are the most likely modes of social tracking for eachrelationship [13].

and safety in space [68]. Factoring possible human-computer interactions into the soclal trackingmatrix further complicates the social cybernetics offlight crew performance. Figure 6B illustratesthispoint with a diagram of some of the interpersonaland human-computer social interactionswhich may berequired for SSFTS operation. Current plans [1-3]call for the SSFTS to be conveyed to its tasklocation by a transport device called the OrbitalManeuvering Vehicle (OMV). It is envisioned thatthe OMV and the SSFTS each will be controlled by aseparate operator. In addition, currentspecifications [3]] call for monitoring oftelerobotlc activities by multiple workstations(such as on the space station and shuttle), with thecapability of handing control from one workstationto another. The various mutual social trackinginteractionssuggested by these plans are indicatedin Figure 6B. The diagram also indicates thespecific type of social tracking []3,56] which ismost likely to be employed for the relationshipspecified.

The general implication of the diagrams inFigure 6 Is that an analysis of the human factors ofworkstationtelepresencerequires consideration, notonly of the social tracking fidelity for aparticular operator-workstation-telerobot system,but also of the social tracking relationshipsbetween all of the players, human and computer,involved in system operation. That high fidelitysocial tracking requires effective control ofsensory feedback from two sources---generated byself-activity and by activity of the socialpartner(s)---suggests two broad themes for humanfactors and engineering research aimed at improvingworkstation telepresence [12,13].

First, computer capabilities for detecting andadaptively controlling sensory feedback from its ownsensors need to be enhanced. One useful outcome of

such research could be automated, computer-medlatedtransformation of spatially perturbed images fromcameras linked to the computer (see Figure 3).Secondly, computer capabilities for detecting andcontrolling sensory feedback from the human operatorneed to be enhanced. One useful outcome of suchresearch could be improved techniques for automateddetection of behavioral and physiologicalmanifestations of operator decremental performanceunder spatiotemporally perturbed sensory feedbackconditions, signalling need for computer-mediatedcorrection of the perturbation. The human factorsof workstation telepresence will benefit from thedevelopment of more robust strategies for automatedcontrol of sensory feedback, growing out of researchon the social cybernetics of human-computer systems,which enable improved system management ofspatiotemporal perturbations in sensory feedback.

SUMMARY

This report has presented a human-factorsanalysis of workstation telepresence which supportsthe following major conclusions:

I. Pioneering human-factors research on operationalrequirements for cybernetic anthropomorphicmachineshas established that spatial compliance betweenpostural, transport, and manipulative movements ofthe operator and those of the CAM is essential tothe creation of high fidelity workstationtelepresence.

247

2. An extensive body of behavioral cyberneticresearch evidence indicates that spatial andtemporal perturbations in visual and auditoryfeedback degrade learning and performance of everybehavioral task so far examined. The findings havedirect relevance to the human factors of human-computer interaction, in that the research itselfmade use of interactive computer- and video-basedmethods to deliberately introduce and evaluate theeffects of such perturbations.

3. Although training improvements occur, thisresearch has found no evidence to indicate thathumans can adapt perfectly to spatial and/ortemporal perturbations in sensory feedback which arenot compliant with established motor feedbackcontrol mechanisms.

4. Because of noncompliant human-factors designfeatures, sensory feedback perturbations ariseroutinely in the course of operational humaninteraction with telerobotic systems. Performancedecrements even occur when video display viewing ofa task is substituted for direct viewing.

5. A number of different modes and sources ofsensory feedback perturbations can be identifiedwhich may potentially influence interactive humanperformance during space telerobot operation.Results from recent laboratory research using remotetelerobot servicers show that manipulation tasks areadversely affected by both spatial perturbations andtemporal delays, confirming in part the findingsfrom earlier behavioral cybernetic research.

6. The human factors of workstation telepresenceencompass the distinct social dynamic attributes oftelerobotic systems, involving mutual sensoryfeedback exchange and control relationships amongmultiple human operators, computers, and telerobots.A social cybernetic model of human-computerinteraction is described which uses social trackingconcepts to characterize the sensory feedbackcontrol relationships among human and machineparticipants in complex, interactive systems.Current plans suggest that such social relationshipswill be an integral part of space stationteleoperations, where station-, shuttle-, andground-based systems may all be involved inmediating telerobotic tasks. One projection Fromthe social cybernetic model is that a researchemphasis on improving the capabilities of theinteractive computer for controlling sensoryFeedback, from its own performance as well as Fromthat of the human partner, will benefit thedevelopment of workstation telepresence.

REFERENCES

].

.

Purves, R.B., Lin, P.S., and Fisher, E.M.,Jr., "A Plan for Time-Phased Incorporation ofAutomation and Robotics on the U.S. SpaceStation," SECOND CONFSRENCE ON ARTIFICIALINTELLIGENCE FOR SPACE APPLICATIONS, NASAScientific and Technical InformationDivision,Washington, D.C., ]988, pp. 237-245.Varsi, G., "Telerobotics for the EfficientUtilization of Space," PROCEEDINGSOF THE 3gTHCONGRESS OF THE INTERNATIONAL ASTRONAUTICALFEDERATION, International AstronauticalFederation, Paris, 1988, IAF-88-023.

3. Varsi, G., and Herman, D.H., "Space Station asa Vital Focus for Advancing the Technologiesof Automation and Robotics," SECOND CONFERENCEON ARTIFICIAL INTELLIGENCE FOR SPACEAPPLICATIONS, NASA Scientific and TechnicalInformationDivision, Washington, D.C., 1988,pp. I-6.

4. Burke, J.D., "Lunar Resources: Evaluation andDevelopment for Use in Lunar Base Program,"PROCEEDINGS OF THE 3gTH CONGRESS OF THEINTERNATIONAL ASTRONAUTICAL FEDERATION,InternationalAstronautical Federation,Paris,1988, iAF-88-586.

5. Burke, J.D., "Precursors and Adjuncts of aLunar Base," PROCEEDINGS OF THE 3gTH CONGRESSOF THE INTERNATIONALASTRONAUTICAL FEDERATION,International Astronautical Federation,Paris,1988, IAF-88-616.

6. Podnieks, E.R., "Environmental Considerationsfor Lunar Base Engineering," ENGINEERING ANDCONSTRUCTION AND OPERATIONS IN SPACE,Aerospace Division/ASCE, Albuquerque, NM,]988, pp. 55-66.

7. Podnieks, E.R., and Roepke, W.W., "Mining forLunar Base Support," LUNAR BASES AND SPACEACTIVITIES OF THE 21ST CENTURY, W.W. Mendell(Ed.), Lunar and Planetary Institute, Houston,I985, pp, 445-452.

8. Podnieks, E.R., and Roepke, W.W., "MiningTechnology Requirements for Lunar ResourceDevelopment," PROCEEDINGS OF SYMPOSIUM '86 -THE FIRST LUNAR DEVELOPMENT SYMPOSIUM, LunarDevelopment Council, Pitman, NJ, ]987, pp.$86-38A to $86-38H.

9. NASA Advanced Technology Advisory Committee,"Advancing Automation and Robotics Technologyfor the U.S. Economy, Volumes I and II," NASATM-87566, NASA, Washington, D.C., 1985.

10. Shaw, J.A., "Reducing Operator Error inDistributed Control Systems", INTECH, March,]988, pp. 45-50.

11. Smith, K.U., and Smith, T.J., "Analysis of theHuman Factors in Automation," SUCCESS FACTORSFOR IMPLEMENTING CHANGE A MANUFACTURING

VIEWPOINT, K.M. Blache (Ed.), Society ofManufacturing Engineers, Dearborn, MI, 1988,pp. 259-338.

12. Smith, T.J., and Smith, K.U., "Motor FeedbackControl of Human Cognition- Implicationsforthe Cognitive Interface," SOCIAL, ERGONOMICAND STRESS ASPECTS OF WORK WITH COMPUTERS, G.Salvendy, S.L. Sauter, and J.J. Hurrell(Eds.), Elsevier, Amsterdam, 1987, pp. 239-254.

13. Smith, T.J., and Smith, K.U., "The SocialCybernetics of Human Interaction withAutomated Systems," ERGONOMICS OF HYBRIDAUTOMATED SYSTEMS I, W. Karwowski, H.R.Parsaei, and M.R. Wilhelm (Eds.), Elsevier,Amsterdam, 1988, pp. 691-711.

14. Smith, T.J., and Smith, K.U., "BehavioralCybernetic Basis of Cognitive Performance.Experimental and Theoretical Analysis,"ERGONOMICS (Special issue on 'MethodologicalIssues in Cognitive Ergonomics'), 1989, InPress.

15. Collins, S.L., and Purves, R.B., "RemoteServicing of Space Systems," PROCEEDINGS OFTHE SECOND CONFERENCE ON ARTIFICIALINTELLIGENCE FOR SPACE APPLICATIONS, NASA,

248

Washington, D.C., ]988, pp. 523-535.16. Keslowitz, S., "Open Control/Display System

for a Telerobotics Work Station," PROCEEDINGSOF THE GODDARD CONFERENCE ON SPACE

APPLICATIONS OF ARTIFICIAL INTELLIGENCE,NASA,Washington, D.C., 1987.

17. Mosher, R.S., "Industrial Manipulators,"SCIENTIFIC AMERICAN, Vol. 2II, No. 4, October,]964, pp. 88-98.

I8. Mosher, R.S., and Murphy, W., "Human ControlFactors in Walking Machines," PROCEEDINGS OFAMERICAN SOCIETY OF MECHANICAl EV_GINEERSHUMAN

FACTORS CONFERENCE, Chicago, 1965.19. Ruch, T.C., Patton, H.D., Woodbury, J.W., and

Towe, A.L., NEUROPHYSIOLOGY, Saunders,Philadelphia, ]965.

20. Rahimi, M., "Human Factors Engineering andSafety in Robotics and Automation," HUMANFACTORS SOCIETY BULLETIN, Vol. 30, No. 7,1987, pp. 3-5.

21. Smith, K.U., HUMAN FACTORS ANALYSIS OF THEMECHANISM OF WALKING WITH SPECIAL REFERENCE TO

DESIGN OF A PEDIPULATORVEHICLE, University ofWisconsin Behavioral Cybernetics Laboratory,Madison, WI, 1965.

22. Smith, K.U., SENSORY-FEEDBACK ANALYSIS OFMANLIKE MACHINE SYSTEMS: A NEW BEHAVIORALTHEORY FOR HUMAN ENGINEERING, University ofWisconsin Behavioral Cybernetics Laboratory,Madison, WI, 1965.

23. Smith, K.U., DELAYED SENSORY FEEDBACK ANDBEHAVIOR, Saunders, Philadelphia, 1962.

24. Smith, K.U., and Smith, W.M., PERCEPTION ANDMOTION: AN ANALYSIS OF SPACE-STRUCTUREDBEHAVIOR, Saunders, Philadelphia, 1962.

25. Smith, K.U., and Smith, M.F., CYBERNETICPRINCIPLES OF LEARNING AND EDUCATIONALDESIGN,Holt, Rinehart and Winston, New York, 1966.

26. Smith, K.U., "Cybernetic Psychology," THEPSYCHOMOTOR DOMAIN, R.N. Singer, Ed., Lea andFebiger, New York, 1972, pp. 285-348.

27. Smith, T.Jo, and Smith, K.U., "CyberneticFactors in Motor Performance and Development,"DIFFERING PERSPECTIVES IN MOTOR LEARNING,MEMORY, AND CONTROL, D. Goodman, R.B. Wilberg,and I.M. Franks, Eds., Elsevier, Amsterdam,1985, pp. 239-283.

28. Smith, T.J., and Smith, K.U., "Feedback-Control Mechanisms of Human Behavior,"HANDBOOK OF HUMAN FACTORS, G. Salvendy, Ed.,Wiley, New York, 1987, pp. 251-293.

29. Smith, T.J., and Smith, K.U., "The CyberneticBasis of Human Behavior and Performance,"CONTINUING THE CONVERSATION. A NEWSLETTER OFIDEAS IN CYBERNETICS,Winter, 1988, Number 15,pp. 1-28.

30. Raibert, M.H., "Machines That Run," ROBOTICSAND MOTOR CONTROL: COMMONALITIES ANDDIFFERENCES, Conference Paper No. 5, SimonFraser University, Burnaby, B.C., Canada, June3, 1987.

31. NASA Goddard Space Flight Center, "SpaceStation Flight Telerobotic ServicerRequirements," FTS REQUIREMENTS DOCUMENT FORPHASE C/D, NASA RFP SS-GSFC-O043, Nov. I,1988, Section 3.0.

32. Smith, R.L., "Human Visual Requirements forControl and Monitoring of a Space Telerobot,"ERGONOMICS OF HYBRID AUTOMATED SYSTEMS I,W. Karwowski, H.R. Parsaei, and M.R. Wilhelm

(Eds.), Elsevier, Amsterdam, 1988, pp. 233-240.

33. Helmholtz, H.L.F. von, HANDBUCH DERPHYSIOLOGISCHEN OPTIK, 3 Volumes, Voss,Hamburg and Leipzig, 1856-1866.

34. Welch, R.B., PERCEPTUAL MODIFICATION.ADAPTING TO ALTERED SENSORY ENVIRONMENTS,Academic Press, New York, 1978.

35. Howard, I.P., and Templeton, W.B., HUMANSPATIAL ORIENTATION, Wiley, New York, 1966.

36. Young, L.R., Oman, C.M., Watt, D.G.D., MoneyK.E., and Lichtenberg, B.K., "SpatialOrientation in Weightlessness andReadaptation to Earth's Gravity," LIFESCIENCES, Vol. 225, 1984, pp. 205-208.

37. Vasyutin, V.V., and Tishchenko, A.A., "SpaceColoristics," SCIENTIFIC AMERICAN, Vol. 26l,No. I, July 1989, pp. 84-90.

38. Huggins, C.T., Malone, T.B., and Shields,N.L., "Evaluation of Human Operator VisualPerformance Capability for TeleoperatorMissions," REMOTELY MANNED SYSTEMS, E. Heer(Ed.), California Institute of Technology,Pasadena, CA, 1973, pp. 337-350.

39. Chandlee, G.O., Smith, R.L., and Wheelwright,C.D., "Illumination Requirements for Operatinga Space Remote Manipulator," ERGONOMICS OFHYBRID AUTOMATED SYSTEMS I, W. Karwowski, H.R.Parsaei, and M.R. Wilhelm (Eds.), Elsevier,Amsterdam, ]988, pp. 241-248.

40. Krishen, K., "Robotic Vision Technology andAlgorithms for Space Applications,"PROCEEDINGS OF THE 39TH CONGRESS OF THEINTERNATIONAL ASTRONAUTICAL FEDERATION, PaperNo. IAF-88-028, International AstronauticalFederation, Paris, 1988.

41. Krishen, K., "Space Robotic Vision SystemTechnology," PROCEEDINGS OF ISA-88INTERNATIONAL CONFERENCE, Paper No. 88-1615,ISA, Research Triangle Park, NC, 1988, pp.1527-1537.

42. Smith, K.U., and Arndt, R., "ExperimentalHybrid-ComputerAutomation in Perceptual-Motorand Social Behavioral Research," JOURNAL OFMOTOR BEHAVIOR, Vol. I, No. I, 1969, pp. 11-27.

43. Smith, K.U., and Smith, T.J., "Systems Theoryof Therapeutic and Rehabilitative Learningwith Television," JOURNA_ERVCIIFS ANDMENTAL DISEASE, Vol. 148, 1969, pp. 386-429.

44. Smith, K.U., and Murphy, T.J., "SensoryFeedback Mechanisms of HandwritingMotions andTheir Neurogeometric Bases," NEW HORIZONS FORRESEARCH IN HANDWRITING, M. Herrick (Ed.),University of Wisconsin Press, Madison, WI,Chap. V, 1963, pp. III- 157.

45. Smith, K.U., Cambria, R., and Steffan, J.,"Sensory-Feedback Analysis of Reading,"JOURNAL OF APPLIED PSYCHOLOGY,Vol. 48, No. 5,1964, pp. 275-286.

46. Gould, J.D., and S_ith, K.U., "AngularDisplacement of Visual Feedback in Motion andLearning," PERCEPTUAL AND MOTOR SKILLS, Vol.17, 1963, pp. 699-710.

47. Smith, K.U., and Ramana, D.S.V.,"Characteristicsof Head Motion with VariableVisual Feedback Delay, Velocity andDisplacement," AMERICAN JOURNAL OF PHYSICALMEDICINE, Vol. 48, No. 6, 1969, pp. 289-300.

249

48. Abbs, J.H., and Smith, K.U., LateralityDifferences in the Auditory Feedback Controlof Speech," JOURNAL OF SPEECH AND HEARINGRESEARCH, Vol. 13, I970, pp. 298-303.

49. Smith, K.U., and Putz, V., "Dynamic MotorFactors in Determining Effects of RetlnalImage Feedback Reversal and Delay," AMERICANJOURNAL OF OPTOMETRY AND ARCHIVES OF AMERICAN

ACADEMY OF OPTOMETRY, Vol. 47, No. S, 1970,pp. 372-383.

50. Sussman, H.M., and Smith, K.U., "SensoryFeedback Persistence in Determining Memory,"JOURNAL OF APPLIED PSYCHOLOGY, Vol. 54, 1970,pp. 503-508.

S]. Rothe, M., SOCIAL TRACKING IN CHILDREN AS AFUNCTION OF AGE, Masters Dissertation,Department of Psychology, University ofWisconsin - Madison, Madison, WI, 1973.

52. Lisberger, S.G., "The Neural Basis forLearning of Simple Motor Skills," SCIENCE,Vol. 242, i988, pp. 728-73S.

53. Lisberger, S.G., and Pavelko, T.A., "BrainStem Neurons in Modified Pathways for MotorLearning in the Primate Vestibulo-OcularReFlex," SCIENCE, Vol. 242, 1988, pp. 771-773.

54. Smith, K.U., and Greene, P., "A CriticalPeriod in Maturation of Performance withSpace-Displaced Vision," PERCEPTUAL AND MOTORSKILLS, Vol. 17, 1963, pp. 627-639.

55. Smith, K.U., and Molitor, K., "Adaptation toCombined Reversal and Delay of Eyemovement-Retinal Feedback," JOURNAL OF MOTOR BEHAVIOR,Vol. ], No. 4, ]969, pp. 296-308.Smith, K.U., INDUSTRIAL SOCIAL CYBERNETICS,University of Wisconsin BehavioralCyberneticsLaboratory, Madison, WI, 1974.Cherry, C., and Sayers, B. McA., "Experimentson Total Inhibition of Stammering by ExternalControl and Some Clinical Results," JOURNAL OFPSYCHOSOMATIC RESEARCH, Vol. 1, 1956, pp. 233-246.

58. Sauter, S., and Smith, K.U., "Social Feedback:Quantitative Division of Labor in SocialInteractions,"JOURNAL OF CYBERNETICS,Vol. I,No. 2, 1971, pp. 80-93.

sg. Smith, K.U., and Kao, H., "Social Feedback:Determination of Social Learning," JOURNAL OFNERVOUS AND MENTAL DISEASE, Vol. 152, 1971,pp. 289-297.

60. Ting, T., Smith, M., and Smith, K.U., "SocialFeedback Factors in Rehabilitative Processesand Learning," AMERICAN JOURNAL OF PHYSICALMEDICINE, Vol. 51, 1972, pp. 86-101.

61. Smith, T.J., Smith, R.L., Stuart, M.A., Smith,S.T., and Smith, K.U. "InteractivePerformance in Space - The Role of PerturbedSensory Feedback," PROCEEDINGS OF THE 3RDINTERNATIONAL CONFERENCE ON HUMAN-COMPUTERINTERACTION, M. Smith (Ed.), Elsevier,Amsterdam, 1989, To be published.

62. Smith, R.L., THE EFFECTS OF TELEVISED ANGULARDISPLACEMENT OF VISUAL FEEDBACK ON A REMOTEMANIPULATION TASK, Unpubl|ShedMasters Thesis,Texas A&M University, College Station, TX,1986.

63. Stuart, M.A., and Smith, R.L., SPATIALLYDISPLACED VISUAL FEEDBACK AND PERFORMANCE OF

FTS-LIKE TASKS, NASA Technical Report JSC-23547, NASA, Houston, TX, 1989.

64. Lawrence, D.A., Chapel, J.D., and Depkovich,T.M., "Issues, Concerns, and InitialImplementation Results for Space BasedTelerobotic Control," PROCEEDINGS, 1987GODDARD CONFERENCE ON SPACE APPLICATIONS OFARTIFICIAL INTELLIGENCE (AI) AND ROBOTICS,NASA Goddard Space Flight Center, Greenbelt,MD, 1987, pp. 1-18.

65. Albus, J.S., Lumia, R., and McCain, H.,Hierarchical Control of Intelligent MachinesApplies to Space Station Telerobots," IEEETRANSACTIONS ON AEROSPACE AND ELECTRONSYSTEMS, Vol. 24, No. 5, ]988, pp. 535-541.

66. McCain, H.G., and Andary, J.F., "The FlightTelerobotic Servicer Project and SystemsOverview," PROCEEDINGS, 2]ST ANNUALELECTRONICS AND AEROSPACE CONFERENCE, IEEE,

Arlington, VA, 1988, pp. 97-102.67. Smith, J.H., Gyamfi, M., Volkmer, K., and

Zimmerman, W., "The Space Station AssemblyPhase: System Design Trade-Offs for theFlight Telerobotic Servicer," PROCEEDINGS,1988 GODDARD CONFERENCE ON SPACE APPLICATIONSOF ARTIFICIAL INTELLIGENCE (AI) AND ROBOTICS,NASA Goddard Space Flight Center, Greenbelt,MD, 1988.

68. Nicholas, J.M., "Interpersonal and group-behavior skills traintng for crews on spacestation," AVIATION, SPACE, AND ENVIRONMENTALMEDICINE, Vol. 60, No. 6, 1989, pp. 603-608.

250