jaa/Papers/dissertation.pdf · HUMAN MANA GEMENT OF A HIERAR CHICAL SYSTEM F OR THE CONTR OL OF MUL TIPLE MOBILE R OBOTS JULIE A AD AMS A DISSER T A TION in COMPUTER AND INF ORMA

HUMAN MANAGEMENT OF A HIERARCHICAL SYSTEM

FOR THE CONTROL OF MULTIPLE MOBILE ROBOTS

JULIE A� ADAMS

A DISSERTATION

in

COMPUTER AND INFORMATION SCIENCE

Presented to the Faculties of the University of Pennsylvania in Partial Ful�llment of

the Requirements for the Degree of Doctor of Philosophy�

��

Richard Paul

Supervisor of Dissertation

Peter Buneman

Graduate Group Chairperson

c� Copyright ��by

Julie A� Adams

To my grandmother� Elaine Ogle Adams

�February �� January ��

iii

Acknowledgments

First and foremost� I would like to thank my advisor� Dr� Richard Paul for his endless

guidance and assistance� This document would not exist without his support� I would

like to thank the members of my dissertation committee� In particular� Dr� Max Mintz

for his statistical assistance and Dr� Sharon Stans�eld and Sandia National Laboratories

for providing the grant to complete the human factors analysis� I thank the members of

the multiagent�s group which provided a test bed for this system� I also thank Dr� Lee

Scott Ehrhart who assisted with the human factors experimental design�

Since my move to Minneapolis I have received much support and assistance with the

completion of this document� I thank Dr� Maria Gini at the University of Minnesota�

my numerous colleagues at the Honeywell Technology Center� Jana Kosecka� Doreen

Jackson� Craig Reynolds� Pamela Lechleider� and Ulf Cahn von Seelen�

I would like to thank the past and present members of the GRASP laboratory for

their support and assistance� In particular� Craig Sayers� Richard Pito� and Marcos

Salganico� who convinced me to come to Penn� I thank Patti Bauer for being a good

friend� For their constant support� understanding and encouragement I thank� Mr� and

Mrs� Douglas Brown� Mr� and Mrs� Thomas Burbank� Elizabeth Echausse� Mr� and

Mrs� James Erlanger� Mr� and Mrs� Michael Oates and Mr� and Mrs� Everett Walsh�

The �Fool �nally �nished� Finally� I thank my family and all others who have been a

part of my life during my tenure as a graduate student�

This work was partially supported by� ARO Grants DAAL��C�� ARPA

Grants N��J�� ARPA�NSF Grant IRI�� NSF Grants STC SBR��

and CISE�CDA�� and Sandia National Laboratories AN��

iv

Abstract

HUMAN MANAGEMENT OF A HIERARCHICAL CONTROL SYSTEM

FOR MULTIPLE MOBILE ROBOTS

Julie A� Adams

Richard P� Paul

We have developed a hierarchy of human supervisory mediation levels into a robotic

system� The purpose of this work was to create a semi�autonomous system which should

enable completion of all feasible tasks� We propose the mediation hierarchy permits

increased interaction between the human supervisor and all levels of a robotic system�

This interaction permits the supervisor to maintain the system in a stable state� We

have incorporated these hierarchical levels into MASC� a Multiple Agent Supervisory

Control system� MASC provides the human supervisor with a three dimensional graph�

ical human�machine interface� Our current test bed is the University of Pennsylvania�s

General Robotics and Active Sensory Perception GRASP Laboratory�s multiagent sys�

tem� This system includes two mobile observation robots and two mobile manipulatory

agents� We designed human factors experiments to test the MASC system�s usability

as well as the mediation hierarchy theory� Novice users were recruited to participate in

the experiments which included three di�erent tasks� the single agent� the two agent

and the four agent tasks� This thesis reviews the design of the experiments� analysis

of the experimental results and a discussion of the results� The results are discussed in

three perspectives� the MASC interface� the mediation hierarchy and the multiagent�s

system� In general� we found novice users were capable of executing the assigned tasks�

We also substantiated the higher level interactions as well as the need for the lower level

interactions of the mediation hierarchy� This thesis also presents a detailed review of all

related research� as well as the contributions and future directions of this research�

v

Contents

Acknowledgments � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � iv

Abstract � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � v

� Introduction �

�� Problem Statement � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �

�� Scope and Outline of Document � � � � � � � � � � � � � � � � � � � � � � � � �

�� Literature Review � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �

�� Control Methods � � � � � � � � � � � � � � � � � � � � � � � � � � � � �

�� General Human�Machine Interface Review � � � � � � � � � � � � � � ��

�� Graphical User Interfaces � � � � � � � � � � � � � � � � � � � � � � � ��

�� Human Factors Considerations � � � � � � � � � � � � � � � � � � � � ��

�� Human Factors Analysis Design � � � � � � � � � � � � � � � � � � � � ��

�� Human�machine System Mediation�Intervention � � � � � � � � � � ��

� Multiple Agent Supervisory Control System �MASC� and Application

Description ��

�� Multiagents Project � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Multiagents Architecture � � � � � � � � � � � � � � � � � � � � � � � ��

�� Multiagents Experiments � � � � � � � � � � � � � � � � � � � � � � � ��

�� MASC System Overview � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� MASC System Layout � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� MASC System Control Buttons � � � � � � � � � � � � � � � � � � � � ��

�� MASC System Modes � � � � � � � � � � � � � � � � � � � � � � � � � ��

vi

� Mediation Hierarchy �

�� Motivation for Development � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Mediation Hierarchy Description � � � � � � � � � � � � � � � � � � � � � � � ��

�� Level Descriptions � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Task Level � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Regulation Level � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Processing Level � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Data Level � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

� Multiagents Process Integrations into the Mediation Hierarchy ��

�� Task Level � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Regulation Level � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Control Interaction � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Request Interaction � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Speci�cation Interaction � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Processing Level � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

Human Factors Experimental Design ��

�� Purpose � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Tasks � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Method � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Subjects � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Apparatus � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Procedure � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

� Human Factors Experimental Results �

�� Pre�Experimental Questionnaire � � � � � � � � � � � � � � � � � � � � � � � ��

�� Number of Commands � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Sensing Modalities � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Number of Errors � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Task Completions � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Task Completion Times � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

vii

�� Perceived Workload Measures � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Multiple Data Comparisons � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Perceived Workload Measures � � � � � � � � � � � � � � � � � � � � � ��

�� Number of Commands � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Number of Errors � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Completion Times � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Post�task Questionnaire � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Post�Experimental Questionnaire � � � � � � � � � � � � � � � � � � � � � � � ��

�� Subjects� Written Comments � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Ravden and Johnson�s Evaluation Check list � � � � � � � � � � � � � � � � � ��

�� Visual Clarity � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Consistency � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Compatibility � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Informative Feedback � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Explicitness � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Appropriate Functionality � � � � � � � � � � � � � � � � � � � � � � � ��

�� Flexibility and Control � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Error Prevention and Correction � � � � � � � � � � � � � � � � � � � ��

�� User Guidance and Support � � � � � � � � � � � � � � � � � � � � � � ��

��System Usability Problems � � � � � � � � � � � � � � � � � � � � � � ��

� Discussion ��

�� MASC System Discussion � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� General Discussion � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Mediation Hierarchy Discussion � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Multiagents System Discussion � � � � � � � � � � � � � � � � � � � � � � � � ��

� Summary ��

�� Contributions � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Future Work � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� MASC Interface Future Work � � � � � � � � � � � � � � � � � � � � � ��

viii

�� Mediation Hierarchy Future Work � � � � � � � � � � � � � � � � � � ��

�� Human Factors Analysis � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Conclusions � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A Further Graphical Presentation of Human Factors Experimental Re

sults ��

A�� Number of Commands � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� Number of Errors � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� Completion Times � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� Perceived Workload Measures � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� Combined Analysis � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� Perceived Workload Measures � � � � � � � � � � � � � � � � � � � � � ��

A�� Perceived Workload Measures Versus the Number of Errors � � � � ��

A�� Perceived Workload Measures Versus the Completion Times � � � � ��

A�� Number of Commands � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� Number of Commands Versus the Number of Errors � � � � � � � � ��

A�� Number of Commands Versus Completion Times � � � � � � � � � � ��

A�� Number of Errors � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� Number of Errors Versus Completion Times � � � � � � � � � � � � � ��

B Human Factors Experiment Consent Form ��

C Pre Experimental Questionnaire ��

C�� Quesstionnaire � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

C�� Graphical Presentation of Subject�s Responses � � � � � � � � � � � � � � � ��

D NASA TLX Questionnaire ��

E Post Task Questionnaire ��

E�� Questionnaire � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

E�� Graphical Presentation of Subject�s Responses � � � � � � � � � � � � � � � ��

ix

F Post Experimental Questionnaire ��

F�� Quesstionnaire � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� Graphical Presentation of Results � � � � � � � � � � � � � � � � � � � � � � � ��

Bibliography �

x

List of Tables

�� Data collection methods� � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The break down of all commands created by task for all data� � � � � � � � ��

�� Number of commands for single and four agent tasks� � � � � � � � � � � � ��

�� Total number of commands for the single agent task between sessions� � � ��

�� The display methods and the percentage of time they were used� � � � � � ��

�� The average number of displays employed by task and session for all trials ��

�� Errors subjects received by frequency of occurrence� � � � � � � � � � � � � ��

�� Break down of task completions results by task and session� � � � � � � � � ��

�� Task completion times between the single and four agent tasks � � � � � � ��

�� Perceived workload measures between the single and four agent tasks for

all data� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Total number of commands versus perceived workload measures for all data��

�� Total number of commands versus perceived workload measures for session

two� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Total number of errors versus perceived workload measures for all data� � ��

�� Task completion times versus perceived workload measures for all data� � ��

�� Task completion times versus total number of commands for all data� � � ��

�� Total number of errors versus task completion times for all data� � � � � � ��

A�� All commands created by task as a percentage of total commands� � � � � ��

A�� The number of commands between the single and two agent tasks� � � � � ��

A�� The number of commands between the two and four agent tasks� � � � � � ��

A�� The number of commands for two agent task between sessions� � � � � � � ��

A�� The number of commands the four agent task between sessions� � � � � � � ��

xi

A�� The number of errors received between single and two agent tasks� � � � � ��

A�� The number of errors received between the single and four agent tasks� � � ��

A�� The number of errors received between the two and four agent tasks� � � � ��

A�� The task completion times between the single and two agent tasks� � � � � ��

A�� The task completion times between the two and four agent tasks� � � � � ��

A�� The task completion times for the single agent task between sessions� � � � ��

A�� The task completion times for the two agent task between sessions� � � � � ��

A�� The task completion times for the four agent task between sessions� � � � ��

A�� The perceived workload measures between the single and two agent tasks� ��

A�� The perceived workload measures between the two and four agent tasks� � ��

A�� The perceived workload measures for the single agent task between sessions��

A�� The perceived workload measures for the two agent task between sessions� ��

A�� The perceived workload measures for the four agent task between sessions� ��

A�� The number of commands versus perceived workload measures for session

one� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The number of commands versus perceived workload measures for the

single agent task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The number of commands versus perceived workload measures for the two

agent task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The number of commands versus perceived workload measures for the four

agent task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The task completion times versus perceived workload measures for session

one� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� the task completion times versus perceived workload measures for session

two� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The task completion times versus perceived workload measures for the


A�� The task completion times versus perceived workload measures for the two

agent task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The task completion times versus perceived workload measures for the

four agent task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

xii

A�� The number of errors versus number of commands for all data� � � � � � � ��

A�� The number of commands versus task completion times for session one� � ��

A�� The number of commands versus task completion times for session two� � ��

A�� The number of commands versus task completion times for the single agent

task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The number of commands versus task completion times for the two agent

task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The number of commands versus task completion times for the four agent

task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

xiii

List of Figures

�� The horizontal control structure as described in �Kelley� ��

�� Brooks� levels of competence� � � � � � � � � � � � � � � � � � � � � � � � � � �

�� Bellingham and Consi�s state con�gured layered control architecture� � � � �

�� Taipale and Hirai�s control level scheme of one robot� � � � � � � � � � � � � �

�� The �ve supervisory functions as nested control loops as presented by

Sheridan� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �

�� Rasmussen�s abstraction hierarchy� � � � � � � � � � � � � � � � � � � � � � � ��

�� The supervision model of multiple computers and tasks� � � � � � � � � � � ��

�� The multiple and mirrored loops supervisory control model� � � � � � � � � ��

�� Nakamura�s structure of a human�supervised control system� � � � � � � � ��

�� The dual design approach to human�machine interface development� � � � ��

�� The inverted �U hypothesis for performance vs� mental workload� � � � � ��

�� The adaptive task allocation human�computer interface� � � � � � � � � � � ��

�� The determinants of usability� � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The Observation Agents � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The Manipulatory Agents� ZebraBot left and PumaBot right� � � � � � ��

�� The experimental set�up � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The MASC system interface� � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The MASC system robot control buttons� � � � � � � � � � � � � � � � � � � ��

�� The MASC system mode control buttons� � � � � � � � � � � � � � � � � � � ��

�� The �phantom agent during teleoperation� � � � � � � � � � � � � � � � � � ��

�� The Path Planning Methods � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Hierarchical levels of human interaction� � � � � � � � � � � � � � � � � � � � ��

xiv

�� The interaction of the regulation level� � � � � � � � � � � � � � � � � � � � � ��

�� The error message displayed when the supervisor chooses an inactive agent� ��

�� The display of the raw ultrasonic and infrared sensors� � � � � � � � � � � � ��

�� The features detected by the ultrasonic process� � � � � � � � � � � � � � � � ��

�� The localization data display determined by the ultrasound process� � � � ��

�� Visually guided obstacle avoidance process� state diagram� � � � � � � � � � ��

�� The free space map displays� � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The error message generated when improperly adding way points� � � � � ��

�� The error message generated for the local path planner singularity case� � ��

�� Clustering variables modi�cation window� � � � � � � � � � � � � � � � � � � ��

�� Localization variable modi�cation window� � � � � � � � � � � � � � � � � � � ��

�� The single agent task � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The two agent task � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The four agent task � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� The generalized GRASP Laboratory�s �oor plan� � � � � � � � � � � � � � � ��

�� The means for all commands by task and session� � � � � � � � � � � � � � � ��

�� Number of commands between single and four agent tasks� � � � � � � � � ��

�� The number of displays employed for the single agent task by session� � � ��

�� The number of displays employed for the two agent task by session� � � � ��

�� The number of displays employed for the four agent task by session� � � � ��

�� The number of errors by task and session for all trials� � � � � � � � � � � � ��

�� The mean completion times by task and session in seconds� � � � � � � � � ��

�� Task completion time for the single agent task� � � � � � � � � � � � � � � � ��

�� The perceived workload means by task and session for all trials� � � � � � � ��

�� Perceived workload measures for the single agent task� � � � � � � � � � � � ��

�� Perceived workload measures versus number of commands for session one� ��

�� Perceived workload versus completion times for the two agent task� � � � � ��

�� The number of errors versus the number of commands for all data� � � � � ��

�� Number of commands versus completion times for the four agent task� � � ��

xv

A�� The mean number of locomotion commands by task and session� � � � � � ��

A�� The mean number of agent mode commands by task and session� � � � � � ��

A�� The mean number of agent switch commands by task and session� � � � � ��

A�� The number of commands for the single versus the two agent task� � � � � ��

A�� The number of commands for the two versus four agent tasks� � � � � � � � ��

A�� The number of commands for the single agent task between sessions� � � � ��

A�� The number of commands for the two agent task by session� � � � � � � � � ��

A�� The number of commands for the four agent by session� � � � � � � � � � � ��

A�� The number of errors for the single versus two agent tasks� � � � � � � � � ��

A�� The number of errors for the single versus four agent tasks� � � � � � � � � ��

A�� The number of errors for the two versus four agent tasks� � � � � � � � � � ��

A�� The completion times between the single and two agents tasks� � � � � � � ��

A�� The Completion times between the two and four agents tasks� � � � � � � � ��

A�� The Completion times between the single and four agents tasks� � � � � � � ��

A�� The completion times for the two agent task between sessions� � � � � � � ��

A�� The completion times for the four agent task between sessions� � � � � � � ��

A�� The perceived workload measures between the single and two agent tasks� ��

A�� The perceived workload measures between the two and four agent tasks� � ��

A�� The perceived workload measures between the single and four agent tasks� ��

A�� The perceived workload measures for the two agent tasks between sessions� ��

A�� The perceived workload measure for the four agent tasks between sessions� ��

A�� The number of commands versus the perceived workload measures for all

data� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The number of commands versus perceived workload measures for session

two� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The number of commands versus perceived workload measures for the


A�� The number of commands versus perceived workload for the two agent task��

A�� The number of commands versus perceived workload for the four agent task��

A�� The number of errors versus perceived workload measures for all data� � � ��

xvi

A�� The task completion times versus perceived workload measures for the all

data� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The completion times versus perceived workload measures for session one� ��

A�� The completion times versus perceived workload measures for session two� ��

A�� The completion times versus perceived workload measures for the single

agent task� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The completion times versus perceived workload for the four agent task� � ��

A�� The number of commands versus completion times for all data� � � � � � � ��

A�� The number of commands versus completion times for session one� � � � � ��

A�� The number of commands versus completion times for all data in session

two� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

A�� The number of commands versus completion times for the single agent task��

A�� The number of commands versus completion times for the two agent task� ��

A�� The completion times versus the number of errors for all data� � � � � � � ��

C�� On average� how much time do you use a computer daily� � � � � � � � ��

C�� How often do you play computer games� � � � � � � � � � � � � � � � � � ��

C�� What level of computer expertise to you possess� � � � � � � � � � � � � ��

C�� What is your experience level using direct manipulation interfaces� � � ��

C�� What is your experience level using computer graphics� � � � � � � � � � ��

C�� What is your experience level using a three�dimensional graphical user

interface� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

C�� What is your experience level using a direct manipulation three�dimensional

graphical user interface� � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

C�� What is your experience level working with robots� � � � � � � � � � � � ��

C�� What is your experience level with mobile robots� � � � � � � � � � � � � ��

E�� This task was� � � Impossible� � � Easy� �� NA by task � � � � � � � ��

E�� This task was� � � Impossible� � � Easy� �� NA by Sessions � � � � � ��

E�� This task was� � � Confusing� � � Clear� �� NA by task � � � � � � � ��

E�� This task was� � � Confusing� � � Clear� �� NA by Sessions � � � � � ��

E�� This task was� � � Dull� � � Stimulating� �� NA by task � � � � � � ��

xvii

E�� This task was� � � Dull� � � Stimulating� �� NA by Sessions � � � � ��

E�� This task was� � � Frustrating� � � Satisfying� �� NA by task � � � � ��

E�� This task was� � � Frustrating� � � Satisfying� �� NA by Sessions � ��

E�� I felt in control of the system during this task� by task � � � � � � � � � ��

E�� I felt in control of the system during this task� by Sessions � � � � � � ��

E�� I was able to interpret and understand the data readings during this

task� by task � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

E�� I was able to interpret and understand the data readings during this

task� by Sessions � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

E�� I was able to correct my errors during this task� by task � � � � � � � � ��

E�� I was able to correct my errors during this task� by Sessions � � � � � � ��

E�� I felt able to complete the task� by task � � � � � � � � � � � � � � � � � ��

E�� I felt able to complete the task� by Sessions � � � � � � � � � � � � � � � ��

E�� I felt in control of the individual agents� by task � � � � � � � � � � � � ��

E�� I felt in control of the individual agents� by Sessions � � � � � � � � � � ��

E�� The system�s capabilities for this task were� � � Inadequate power� �

� Adequate power� �� NA by task � � � � � � � � � � � � � � � � � � � � � ��

E�� The system�s capabilities for this task were� � � Inadequate power� �

� Adequate power� �� NA by Sessions � � � � � � � � � � � � � � � � � � � ��

E�� The system�s capabilities for this task were� � � Rigid� � � Flexible�

�� NA by task � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

E�� The system�s capabilities for this task were� � � Rigid� � � Flexible�

�� NA by Sessions � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� Overall reaction to the system� � � Terrible� � � Wonderful� �� NA ��

F�� Overall reaction to the system� � � Frustrating� � � Satisfying� �� NA��

F�� Overall reaction to the system� � � Dull� � � Stimulating� �� NA � � ��

F�� Overall reaction to the system� � � Di�cult� � � Easy� �� NA � � � ��

F�� Overall reaction to the system� � � Inadequate power� � � Adequate

power� �� NA � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� Overall reaction to the system� � � Rigid� � � Flexible� �� NA � � � ��

xviii

F�� Is screen window layout helpful� � � � � � � � � � � � � � � � � � � � � � ��

F�� Is one main window helpful� � � � � � � � � � � � � � � � � � � � � � � � ��

F�� The ability to turn on and o� data displays is� � � Frustrating� � �

Satisfying� �� NA � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� The ability to turn on and o� data displays is� � � Di�cult� � � Easy�

�� NA � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� The ability to turn on and o� data displays is� � � Rigid� � � Flexible�

�� NA � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� The use of command buttons is� � � Illogical� � � Logical� �� NA � � ��

F�� The use of command buttons is� � � Frustrating� � � Satisfying� ��

NA � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� The use of command buttons is� � � Di�cult� � � Easy� �� NA � � � ��

F�� The use of the phantom agent during teleoperation is� � � Unhelpful�

� � Helpful� �� NA � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� The use of the phantom agent during teleoperation is� � � Illogical� �

� Logical� �� NA � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� The use of the phantom agent during teleoperation is� � � Frustrating�

� � Satisfying� �� NA � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� The amount of information which can be displayed on the screen is� � ��

F�� The arrangement of information on the screen is� � � � � � � � � � � � � ��

F�� The use of terms throughout the system is� � � � � � � � � � � � � � � � ��

F�� Control button labels are� � � Inconsistent� � � Consistent� �� NA � ��

F�� Control button labels are� � � Confusing� � � Clear� �� NA � � � � � ��

F�� Error messages which appear on the screen are� � � � � � � � � � � � � � ��

F�� Position of error messages on the screen are� � � � � � � � � � � � � � � � ��

F�� Error messages are� � � Confusing� � � Clear� �� NA � � � � � � � � ��

F�� Error messages are� � � Unhelpful� � � Helpful� �� NA � � � � � � � ��

F�� Error messages are� � � Hard to read� � � Easy to read� �� NA � � � ��

F�� Error messages clarify the problem� � � � � � � � � � � � � � � � � � � � � ��

F�� Phrasing of error messages is� � � Unpleasant� � � Pleasant� �� NA ��

F�� Phrasing of error messages is� � � Confusing� � � Clear� �� NA � � � ��

xix

F�� Phrasing of error messages is� � � Unhelpful� � � Helpful� �� NA � � ��

F�� Instructions for correcting errors are� � � � � � � � � � � � � � � � � � � � ��

F�� Learning to operate the system was� � � � � � � � � � � � � � � � � � � � ��

F�� Getting started with the system was� � � � � � � � � � � � � � � � � � � � ��

F�� Time to learn the system was� � � � � � � � � � � � � � � � � � � � � � � � ��

F�� Remembering names and uses if command buttons was� � � � � � � � � ��

F�� Remembering speci�c rules about entering data was� � � � � � � � � � � ��

F�� Tasks can be performed in a straight forward manner� � � � � � � � � � ��

F�� Number of steps per task were� � � � � � � � � � � � � � � � � � � � � � � ��

F�� Steps to complete a task follow a logical sequence� � � � � � � � � � � � ��

F�� System speed was� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� Response time for most operations is� � � � � � � � � � � � � � � � � � � ��

F�� Rate information is displayed is� � � � � � � � � � � � � � � � � � � � � � ��

F�� System failures occur� � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

F�� Novices can accomplish tasks after proper training� � � � � � � � � � � � ��

xx

Chapter �

Introduction

This chapter states our problem and then the outline for the remainder of the document�

We also present our literature review of the various topics which are related to this

dissertation�

�� Problem Statement

In view of the fact that autonomous systems are likely to fail while executing assigned

tasks� one would prefer to develop some mechanism to avoid this failure� Teleoperated

systems are one such mechanism which can perhaps avoid this failure but have associated

problems� Teleoperated systems require the human operator to create the necessary

commands to carry out the task� While it is true a few teleoperated systems incorporate

a certain level of autonomy� this level is still quite low� Such systems are an improvement

over fully teleoperated systems but do not provide a signi�cant level of autonomy� A

system which permits a high level of autonomy and teleoperation when necessary� can

be termed a semi�autonomous system� Such a system is permitted to autonomously

attempt to complete tasks while utilizing the human operator as an assistant when a

problem situation arises� Thus the human operator should act as a �supervisor of the

system� only interacting as needed to assist with task completion� Such a system would

expand the capabilities of traditional autonomous and teleoperated systems�

This type of system requires a human�machine interface to permit the human op�

erator to monitor and assist it� Many human�machine interfaces have been devel�

oped for applications in the areas of nuclear power plants� aviation� and telerobotics

�

�Christensen� �� Hancke and Braune� �� Sheridan� �� however� these systems

are generally not considered autonomous with the operator providing a �supervisory

role� Typically� the human operator controls the entire task execution�

Thus the development of a system which permits the human operator� when necessary�

to interact with all system levels to assist with process errors is necessary� This interaction

should encompass all areas of a semi�autonomous system from the processes which would

be considered fully�autonomous to those considered telerobotic�

Our system� the Multiple Agent Supervisory Control MASC system� permits such

monitoring and interaction capabilities� In MASC the agents work autonomously until

the human supervisor is requested to take control or a problem is detected by the super�

visor� Our design strategy is to develop a general system which is applicable to various

robotic systems� We combine the advantages of autonomous systems with the human�s

ability to control a system through a human�machine interface� MASC provides the

supervisor with tools to interact with all processing levels of the robotic system� These

interactions may correct corrupted data or process decisions which would typically cause

an autonomous system to enter an unstable state� We desire to create a more comprehen�

sive semi�autonomous system based on this interaction which will successfully complete

the execution of task assignments�

We have de�ned four hierarchical levels of supervisory interaction with the various

levels of a robotic system� The mediation hierarchy permits the supervisor to specify

task assignments� teleoperate agents� display sensory data� override process conclusions

and recon�gure the system during sundry sensory and agent failures�

�� Scope and Outline of Document

This dissertation describes the mediation hierarchy� the process integrations and the hu�

man factors experiments and results which we will use to verify our hypothesis� The

remainder of this chapter will provide a review of the related research topics� Chapter

Two brie�y describes the test bed for the interface as well as the MASC interface aes�

thetics� Chapter Three provides the motivation for the development of this theory� as

well as a formal de�nition of the mediation hierarchy and its levels� A description of

�

the process implementations integrated into the MASC interface is presented in Chapter

Four� Chapter Five describes the experimental design and set up of our human factors

testing on the MASC system� The results of this testing are described in Chapter Six

and the discussion of these results is in Chapter Seven� Finally� Chapter Eight states the

future work topics� contributions and conclusions of this research�

�� Literature Review

�� Control Methods

Kelley �Kelley� �� states�

Control involves a choice or selection among possible future states� the chosen

state comprising the chooser�s goal� This choice is implicit in every control

activity� whether action to achieve it is carried out by living individuals� by

an automatic device or control system� or by some complex arrangement of

men and equipment� And the choice itself is always made by man�

When exercising control over an object we are changing the course of events for that ob�

ject� Many �choices were necessary when developing our system� One of these �choices

was the method of control we would employ through our human�machine interface� Dur�

ing the course of determining this �choice we reviewed behavior based and supervisory

control methods� This subsection is a review of these control methods and an explanation

of the �choice of supervisory control�

Behavior Based Control

Prior to Brooks� development� in �Brooks� �� Brooks� �� of the subsumption ar�

chitecture most control architectures were organized as horizontal subsystems� broken

into sub�subsystems� sub�sub�subsystems� etc� such that systems were built as a chain of

causes and e�ects� An example of this as demonstrated by Kelley in �Kelley� ��

�� consider the control of an environmental variable X� where X is brought

about by Y� and Y is brought about by Z� The energy of control is applied

to Z in the innermost loop� Z is varied to bring about a desired change in Y�

which will in turn bring about the desired change in X in the outer loop�

�

Yd

SelectorZd

Selector∆Z

SelectorControlJunction

ZEffector

YEffector

XEffector

Xd Yd Zd ∆Zd

X Y Z

Z Y X

Sensors

PowerSource

Figure �� The horizontal control structure as described in �Kelley� ��

This example is shown in Figure �� Generally� the inner loops� denoted by Z and Y� are

stronger and of a higher frequency than the outer loops� denoted by X and Y� These inner

loops may also create rate changes or accelerations in the variables of their respective

outer loops�

reason about behavior of objects

plan changes to the world

identify objects

monitor changes

build maps

explore

wander

avoid objects

Sensors Actuators

Figure �� Brooks� levels of competence�

As living beings control their surrounding environments the same applies to control

systems �Kelley� �� The outer loops represent a being�s increased control over its

environment� but as each outer loop is developed� it becomes increasingly dependent

upon the previously developed inner loops� Therefore� when the inner loops fail� the

outer loops also fail and are unable to recover� This as well as the ability to distribute

a representation amongst individual behaviors and the ability to create reasoning from

�

various behaviors led Brooks �Brooks� �� Brooks� �� to develop the subsumption

architecture� His concept was to construct a system bottom�up which could exhibit intel�

ligence and navigate in an unstructured environment� Brooks decomposed the problem

into levels of competence� as shown in Figure �� Brooks believes this architecture per�

mits a complete control system to be built and tested and then allows for the addition

of higher level control systems�

State Table

Behavior Library Active Layered Control Structure

Figure �� Bellingham and Consi�s state con�gured layered control architecture�

Bellingham and Consi expanded the basic subsumption architecture into the state

con�gured layered control �Bellingham and Consi� �� This control method was cre�

ated to address the complexities which develop when employing the subsumption archi�

tecture� While employing Bellingham and Consi�s method� only the layers relevant to

the current portion of the mission are active� thus reducing the complexity� This method

is displayed in Figure �� The �Behavior Library is composed of all possible inac�

tive behaviors while the �Active Layered Control Structure houses only those behaviors

pertinent to the current goal or sub�goal� A �State Table is employed as an external

structure to ensure behaviors are activated at the proper moment and with the correct

priority� The objective is to minimize the number of behaviors active at any given time

frame�

Taipale and Hirai� in �Taipale and Hirai� �� also extend Brooks� subsumption ar�

chitecture to a multiple robots domain� Their master robot can subsume the behavior of

the slave robots� Figure �� displays this control scheme� They employ master�slave con�

�

Messages from other robots

Messages fromslaves

Messages from master

Mastersubsumption signal

Master levelbehavior

Co−operative behavior

Subsumption signal

Subsumption signal

Messages to master

Messages toslaves

Messages to other robots

Control and subsumption signals to lower level behaviors

Figure �� Taipale and Hirai�s control level scheme of one robot�

trol such that the master�s signals subsume the slave�s normal behavior� If a robot is the

master� it employs the upper �Master level behavior for control while the slave actions

are controlled by the �Co�operative behavior � If the master fails then the �Master level

behavior of a slave assumes control� The agent which requires assistance becomes the

master� While this approach extends the layered control to include master�slave control

it also raises issues when determining which robot should be the master� and permits

the possibility of deadlock when help is needed by numerous masters and there are an

insu�cient number of slaves to assist�

There are many problems associated with the subsumption architecture�s use de�

scribed in �Bogoni� �� and similarly in �Hartley and Pipitone� �� Bogoni observes�

it is not clear that a strict hierarchical relation between the various system modules is

su�cient or possible� It is possible that mutual exclusion may be necessary such that only

one behavior is active at a time� Bogoni notes that Brooks� model employs the world as a

means of communication which may create numerous problems when dealing with reac�

tive behaviors that are initiated by �preconditions matched in the environment � With

complex systems it is possible many of these behaviors may be initiated at any single time

instance� Also as Bogoni observes� for complex systems with numerous behaviors� �the

original schema of control and passing of simple message scheme is not su�cient when

�

attempting to carry out a more interesting task� There are also problems associated

with the need for redundant code when addressing the parameterization of the behav�

iors and instances where a behavior is a subset of another behavior at a di�erent level�

Many others� �Fleury et al�� Hartley and Pipitone� �� Martinengo et al��

Mataric� �� Stein� �� have developed systems based upon behavior based control�

While some of these systems have attempted to implement versions of this architecture�

they were unable to solve all the associated problems� We believe� as does Sheridan

�Sheridan� �� that a more robust system can be created employing human supervi�

sory interaction�

Supervisory Control

Almost �� years prior to Sheridan�s supervisory control de�nition in �Sheridan� ��

Kelley �Kelley� �� concluded that automatic control devices could not �approach the

�exibility and versatility of man� He explained that a human could act as an adaptive

controller� since the human was able to foresee the possible system alternatives and

then invoke the proper procedures to obtain the desired goal� He believed that the

human�s ability to �understand and evaluate complex criteria and to appropriately

modify the control behavior were a virtue for a human operator�s existence� In essence�

this description is a high level idealism of supervisory control as de�ned by Sheridan�

Sheridan de�nes supervisory control in two �senses �Sheridan� ��

� The stricter sense� one or more human operators are intermittently pro�

gramming and continually receiving information from a computer that

itself closes an autonomous control loop through arti�cial e�ectors and

sensors to the controlled process or task environment�

� the less strict sense� one or more human operators are continually pro�

gramming and receiving information from a computer that interconnects

through arti�cial e�ectors and sensors to the controlled process or task

environment�

�As de�ned in �Stramler Jr�� adaptive control is a form of automated control equipped with aselfcontained decisionmaking capability for modifying its own operation based on previous experience�

�

intervene

monitor

teach

plan

learn

Figure �� The �ve supervisory functions as nested control loops as presented by Sheri�dan�

The �ve basic human supervisory functions� as displayed in Figure �� include�

� the task planning which entails learning about the process and how it is carried

out� setting goals and objectives which the computer can understand and then

formulating the plan to move from the initial state to the goal state�

� teaching programming the computer by translating goals and objectives into de�

tailed instructions such that the computer can automatically perform portions of

the task�

� the monitoring the autonomous execution either via direct viewing or remote sens�

ing instruments to ensure proper performance�

� the ability to intervene through updating instructions or direct manual control if a

problem exists during execution� and

� the ability to learn from the experience by reviewing recorded data and models and

then applying what was learned to the above phases in the future�

Baron cites the following as general characteristics of applications for human super�

visory control in �Baron� ��

�� large scale� technological systems involving high economic value and�

often� signi�cant risk�

�

�� Complex and dynamic processes with many outputs to be �controlled

and many potential inputs for achieving that control�

�� A structure in which there are many sub�systems� with the coupling

between variables in di�erent sub�systems much looser than that among

variables in a given sub�system�

�� A signi�cant degree of automation� both in system monitoring and con�

trol�

�� Relatively slow response of the variables to be controlled with rapidly

changing variables controlled automatically�

�� Event driven demands�

�� The need to interact and coordinate with other operators and�or exter�

nal entities�

�� The requirement to follow speci�c procedures in de�ned situations avail�

able in procedures �manuals or residing in the operator�s memory�

While our situation does not involve all of these characteristics� it is composed of many

of them� Our problem is rather large� although� it is not as large as controlling a nu�

clear power plant or of the same high economic value� While the multiagent problem

does not entail the second characteristic group� it is a complex group of agents with

dynamic processes with many outputs and inputs� The multiagent problem is composed

of multiple robots� two of which include manipulator robots� therefore� the individual

agents are composed of many sub�systems such as described in characterization three�

We are proposing a semi�autonomous system� This implies the multiple agents will work

autonomously until they require assistance� Also� we monitor the actions of the agents�

thus encompassing characteristic four� As our system is not extremely large� we do not

encounter characteristic �ve� Many processes in our system are event driven therefore

we must consider the sixth characteristic� Characterization seven is met by our system�

We may need to interact with objects in the environment through the mobile agents such

as moving a large object employing two agents� We also have speci�c procedures which

are de�ned for particular situations� hence� the eighth characteristic is relevant� Pooling

�

LEVELS OF ABSTRACTION

FUNCTIONAL PURPOSE

Production flow models, system objectives

ABSTRACT FUNCTION

Causal structure, mass, energy &information flow topology, etc.

GENERALIZED FUNCTIONS

"Standard" functions & processes,control loops, heat transfer, etc.

PHYSICAL FUNCTIONS

Electrical, mechanical, chemicalprocesses of components andequipment

PHYSICAL FORM

Physical appearance and anatomy,material & form, locations, etc.

PHYSICAL BASISCapabilities, resources causes of malfunction.

PURPOSE BASISReasons for proper function requirements

Figure �� Rasmussen�s abstraction hierarchy�

these characteristics with the multiagent�s problem characteristics would imply the use

of the supervisory control method�

Rasmussen �Rasmussen� �� de�nes a supervisory control system as�

a feedback system with the task to monitor the actual operating state of the

system� and to keep it within the speci�ed target domain�

He believes the system should be developed iteratively based upon his abstraction hi�

erarchy�s properties shown in Figure �� The lowest abstraction hierarchy level is the

�Physical Form level which comprises the system�s physical appearance and con�gura�

tion� The �Physical Functions level represents the physical system processes and is the

level which detects the physically limiting properties and malfunctions� The �General�

ized Function level typically represents the functional system structure in a form which

is above the levels of standard components� The �Abstract Function level represents

the overall system function utilizing a generalized causal network� The �Functional Pur�

pose level describes the system�s purpose� This hierarchy can be employed to determine�

bottom�up� the utilization of the system components and functions� and top�down� to

de�ne how the proposed system may be implemented as the functions and components�

��

Human Supervisor

Human−Interactive Computer

(combines high level controland expert advisory system)

controlinstructions

feedbackadvice

requestsfor advice Human−Interactive System

(in control room or cockpit)

Multiplexed Signal Transmission

(may involve bandwidthconstraints or time delays)

Task−InteractiveComputer 1

(SensorBot)


(PumaBot)


(VisionBot)


(ZebraBot)

Task 1 Task 2 Task 3 Task 4

Task−Interactive System(remote from operator)

Task or Controlled Process

(may be continuous process,vehicle, robot, or etc.)

Figure �� The supervision model of multiple computers and tasks in our case roboticagents�

This manner of development permits identi�cation of the necessary control constraints

as each level of abstraction is developed�

Figure �� from �Sheridan� �� presents our basic multiagent system model� Sheri�

dan utilizes this example to display supervisory control for multiple computers and tasks��

The Human�Interactive System is the human operator interface employed to command�

control and monitor the Task�Interactive System� The Task�Interactive System is com�

posed of multiple computers and�or robots which individually execute a task to reach

an overall goal�

Sheridan also proposes a supervisory control model in which the supervisor controls

process information similarly to our proposed supervisory control method� This is dis�

played in Figure �� This particular model passes information through the physical

system external to the human via various information channels� The human may receive

three types of system input� as de�ned by Sheridan�

�� Those that arrive via loop � directly from the task direct seeing� hearing�

or touching�

�Essentially this model is the same for our multiple agents which may also be composed of multiplecomputers for example the manipulation robots�

��

Human Operator

Displays Controls

Human InteractiveComputer

Sensors Actuators

Task InteractiveComputer

Task

1

23

4 5

6

78

910Human−Interaction

subsystem

Semiautomatic subsystem

Figure �� As presented in �Sheridan� �� supervisory control as multiple and mir�rored loops through the physical system�� Task as observed directly by human operator�s own senses�� Task as observed indirectly through arti�cial sensors� computers� and displays� Thistask�interactive�system TIS feedback interacts with that from within human�interactivesystem HIS and is �ltered or modi�ed�� Task as controlled within TIS automatic mode�� Task as a�ected by the process being sensed�� Task a�ects actuators� and in turn is a�ected�� Human operator directly a�ects task by manipulation�� Human operator a�ects task indirectly through controls interface� TIS�HIS comput�ers and actuators� This control interacts with that from within TIS and is �ltered ormodi�ed�� Human operator gets feedback from within HIS in editing a program� running a plan�ning model� or etc�� Human operator orients herself relative to control or adjusts control parameters�� Human operator orients herself relative to display or adjusts display parameters�

��

�� Those that arrive via loops �� and � through the arti�cial display and

are mediated by the computer�

�� Those that arrive via loops � and �� from the display or manual controls

without going through the computer� �� display itself such as brightness

or format� or present position of manual controls� ��

There also exist three output forms from the human operator� as de�ned by Sheridan�

�� Those sent via loop � directly to the task the human operator bypasses

the manual controls and the computer and directly manipulates the task�

makes repairs� etc��

�� Those that communicate instructions via loops �� and � to the task�

�� Those that modify the display or manual control parameters via loops �

and �� without a�ecting the computer i�e�� change the location� forces�

labels or other properties of the display or manual control devices�

Sheridan proposes this model will permit the human operator to intervene at the

various physical system levels� Thus permitting the system to be directly controlled or

controlled at a higher level by the supervisory control system� This idea is similar to ours�

Although� our model does not permit the human operator to directly observe execution

via their own physical sensors or to manipulate a task with direct physical manipulation

of the robotic system� In our method� the human operator must always observe and

manipulate the environment through the human operator interface� Therefore� commu�

nication channels � and � of Figure �� are nonexistent in our system�

Another essential supervisory control consideration is how the human and computer

should share and�or trade control� Sheridan states�

The computer can extend the human�s capabilities beyond what she can

achieve alone� it can partially relieve the human� making her job easier� it

can back up the operator in cases where she falters� and it can replace her

completely�

This statement was considered in our system development� We permit the interface to

extend the human�s capabilities and to relieve the human but we also allow the human

��

to take control of any system level� This permits the human to assist the system when

di�culties occur� In essence� sharing control indicates the human and computer operate

on aspects for which each are better suited and trading control implies the human can

take control of the system�

In �Hirai and Sato� �� Hirai et al�� Sato and Hirai� �� Hirai et al� pro�

posed the cooperative control system for telerobotics� This approach permits the human

operator to superpose various control schemes onto the direct maneuvering� The super�

posed control schemes include� the rate control scheme� the incremental control scheme�

the indexing scheme� the software jigs and the programmed control scheme� While these

schemes permit the operator to modify the slave actions� the ability to work within

and control all system levels is not shown� Their teleoperation intelligence includes�

�planning intelligence which plans the task cooperation between the human operator

and the robot� �execution intelligence which realizes skill cooperation� and �evaluation

intelligence which maintains the environmental changes during task execution� Their

proposal is a form of supervisory control but we believe our method elevates supervisory

control to a higher level�

Nakamura developed a human�supervised control system for a �exible manufacturing

system FMS� His system design is based upon the following principles as de�ned in

�Nakamura� ��

�� A human in the control loop must be responsible for supervising the

automation� monitoring material �ows and outputs� and intervening to

diagnose and either correct or compensate for machine failures and other

unanticipated events �Ammons� ��

�� A human can make more e�ective solutions to the complex problem

by modifying the computer solutions �Nakamura� �� For the sake

of it� the knowledge�based system and the intelligent human�computer

interface are required to aid the human�s decision�making�

His system is composed of three components� the human supervisor� the knowledge�based

system� and the intelligent interface as shown in Figure �� The human supervisor�s

��

Human−supervisor

Gantt Chart

Monitoring andSimulation

IntelligentInterface

Knowledge−based System

Human−supervised Control System

Controller

Decisions Measures

FMS

Figure �� Nakamura�s structure of a human�supervised control system�

responsibilities include monitoring the system and system interventions� The knowledge�

based system�s purpose is to aid decision making� The intelligent interface is composed

of a FMS screen for monitoring purposes and a Gantt chart for scheduling� This system

requires the knowledge�base to determine a solution for all instances and then the human

must determine if this is the correct solution� This di�ers from our approach as we do

not employ a knowledge�base and decisions are either directly determined by a process

or the human� The human may override a decision but the human�s veri�cation is not

necessary�

Bellingham and Humphrey �Bellingham and Humphrey� �� have incorporated su�

pervisory control into the behavior based layered control described in the behavior based

control section� They consider only the behavior based control layers in which the human

and computer cooperatively control the system� Their �user override mode permits the

operator to create vehicle control commands while also allowing the system�s avoidance

behaviors to override the operator�s commands� The �behavior modi�cation mode per�

mits the operator to modify the internal vehicle behaviors settings� The �architecture

modi�cation mode permits the operator to turn behaviors on and o� or assign a new

��

priority to the overall layered control structure� They have shown that supervisory con�

trol may be integrated into the subsumption architecture� While this is an interesting

approach� it still does not resolve all the di�culties associated with the subsumption

architecture�

�� General Human�Machine Interface Review

Human�machine interfaces have been developed by numerous researchers� These inter�

faces range from the early versions composed of dials and mechanisms to today�s virtual

reality systems �Ellis� �� Stans�eld� �� Hodges et al�� Zyda et al�� As

this dissertation research is concentrated on the development of a human�machine in�

terface� it is necessary to review the various aspects incorporated into human�machine

interface development� The remainder of this subsection focuses upon principles which

apply to all human�machine interfaces and the following subsections concentrate upon

other areas which are applicable to our system�

Traditionally� the human operator was provided direct control into a dynamic sys�

tem� As Rouse �Rouse� �� observes� the human operator�s responsibilities are quickly

changing� The human operator�s task is becoming one of monitoring and supervising a

self controlled system� While the human operator�s role is changing� Hancke and Braune

�Hancke and Braune� �� believe that humans are capable of handling the uncertain�

ties through advanced technology� They view automation as an assistant to the human

in abnormal situations and also as a provider of various information and control tools�

Durand �Durand� �� supports this belief and explains why the human will remain a

necessary system element� It remains to determine the proper task allocation trade o�

between the human and the machine�

As Weir observes in �Weir� ��

The human�machine interface provides a means whereby users operator can

a�ect the course or operation of the machine or underlying process� Control

�ow dialogue includes all operator actions on the application and all status

data� including results of operator actions� Such actions are the basis for

intelligent control�

An interactive system�s e�ectiveness is the human�s ability to control key system factors�

��

What Constitutes a Good Interface�

There exist many beliefs as to which characteristics compose a �good interface� Bodker

�Bodker� �� quanti�es a �good interface as one which �allows the user to focus on the

objects or subjects that the user intends to work with � She believes a bad user interface

is one which �forces the user to focus on other objects or subjects than the intended �

More speci�cally� she contends that a �good interface should�

� permit the user to conduct activities through various actions and operations de�

pendent upon the user�s operation skills and the actual material conditions�

� permit all actions to be directed toward their appropriate objects and subjects

rather than toward the artifact�

While Bodker�s view is fairly general� Cox and Walker�s �Cox and Walker� �� view

is much more speci�c� They contend a �good interface is designed considering the

following four characteristics�

� User Control� the user has ultimate control at all times and determines the task

to be performed� not vice versa�

� Transparency� the interface should provide the user with the ability to clearly and

completely monitor the task�

� Flexibility� the interface can be used for various tasks including those which the

designer may not have considered�

� Learnability� The interface must be easy to use and provide the capability for the

users to improve their skills with its use�

As can be observed from these two views� there are many aspects one should consider

when developing a �good human�machine interface� The remainder of this section will

review proposed design schemes� interaction mechanisms and data display methods�

�As de�ned by Bodker in �Bodker� �� Artifacts are things that mediate the actions of the humanbeing toward another subject or toward an object�

��

Design Approaches

There exist many ideas� methods and theories for human�machine interface design� For

instance� Kirlik et al� �Kirlik et al�� believe a human�environment interaction the�

ory must consider both the human and the environment� The environmental component

speci�es those world features which are psychologically relevant to the desired behavior

and provides a descriptive representative language�

Edmonds �Edmonds� �� illustrates that batch process designers are typically con�

cerned with the input and output designs and then determine how processes will achieve

the de�ned output� On the other hand� when designing an interactive system a similar

technique would entail �rst designing the interface� This is considered much more di��

cult� Therefore� designers usually are more concerned with the processor dynamics� Some

designers believe the design process entails the analysis of both human�machine interac�

tions and communication with complex systems� Others� such as �Grant and Mayes� ��

believe in a cognitive approach� or a knowledge based approach �Johannsen� ��

Shneiderman �Shneiderman� �� illustrates �ve speci�c interface design issues�

� command language vs� menu selection�

� response time and display rates�

� wording of system messages�

� on�line tutorials� explanations� and help messages�

� hardware devices�

The trade o� between command language and menu selection will be discussed in more

detail in the following subsection� The response time is considered as the time required

for the results of actions to appear on the monitor� such as characters or images� If there

is a large delay in this measurement the user will become aggravated and displeased

with the system� Shneiderman also found when displaying familiar information� the user

prefers it displayed faster while unfamiliar information should be displayed at slower

rate� The manner in which the system words messages also plays an important role in

the system usability� Cryptic messages and those which provide little information are

found to frustrate users� The designer should consider this fact when creating all system

��

messages including input prompts� menu selection choices� and�or help messages� When

users must disrupt their activity to locate a manual they may easily loose track of their

current e�orts� Therefore� it has become necessary to provide on�line tutorials� help and

explanations as the provision of such materials is found to be less disruptive to the user�s

task� When determining the hardware devices of the system� the designer must consider

the hardware complexity� While devices may be interesting to incorporate� the user may

�nd them di�cult to use�

technology−based design

working process based design

concepts,approaches

concepts,approaches

fully automated

partly automated

computer assisted

manual

Figure �� The dual design approach to human�machine interface development�

An interesting design approach presented in �Brandt� �� is the dual design system

development approach� This approach� shown in Figure �� is based upon a set of

principles which are employed to ensure the proper development of the technical and

human aspects of an interface� The technology�based design model is employed to create

fully automated systems in which the human is not considered until late in the design

process� The working�process based design approach attempts to solve the problem with

less automation while considering the human interactions sooner� The optimal approach

is to combine both models when designing a system as it introduces the human aspects

early into the design process while also considering the technological concerns�

Eberts de�nes four design strategies in �Eberts� �� which may be employed indi�

vidually or combined� These strategies are generalizations which invariably encapsulate

the above methods� They are the most comprehensive of the design strategies and are

typically taught in user interface design courses�

The �rst method is the empirical approach� In this method the experimenter formu�

lates a research question or hypothesis� and then identi�es the items to be tested� The

��

task is de�ned and the various independent and dependent� variables are de�ned� The

designer is required to design� and implement the interface and then analyze the results

of an empirical study� This type of study requires a well controlled environment in which

to carry out the experiments� This is necessary to ensure that variables other than the

independent variables do not vary� The results are then analyzed for statistical signif�

icance� It is important to ensure that the experimental variables are not confounded��

such that the results are interpretable and generalizable to other situations� Eberts states

the advantage of this design is that it is an �alternative to intuition in determining the

best design � The disadvantages of this approach include the ease of introducing method�

ological problems which then make the applicability of the results questionable� as well

as the lack of theoretical guidance with such a design�

The cognitive approach applies theories of cognitive science and cognitive psychol�

ogy� The theories state how the human perceives� stores and retrieves information from

memory� then manipulates that information to make decisions and solve problems� In

this design approach the human is regarded to be adaptive� �exible and actively involved

in interacting with the environment to solve problems or make decisions� This approach

views human�computer interaction as presenting problems which must be solved by the

operator� The advantage of such a system is the view that the user is a �exible� adaptable

information processor actively attempting to solve the problem employing the computer�

The only perceived disadvantage of this method is that some of the approach �s termi�

nology may not be well de�ned for the human�computer interaction domain�

The predictive modeling approach attempts to predict the human�s performance while

interacting with the computer� Such a model may be employed to determine if an

interactive design is �viable in terms of usability before it is prototyped � This type of

modeling requires the designer to understand the existing models and techniques to apply

the models to various tasks� The advantage of this modeling method is the designer�s

ability to determine fairly accurate error and time estimates for a task� A disadvantage

is that implementation requires interpreting the task which can lead to more than one

�Eberts de�nes an independent variable to be the variable which is being manipulated by the experimenter� He de�nes the dependent variables to be those which depend upon the independent variable�

�A confound is an uncontrolled extraneous factor which could have caused an eect in one level ofthe results but not all levels�

��

task analysis�

The anthropomorphic approach employs the application of human�human communi�

cation as a model for the human�computer interaction� It is believed that by designing

the human�computer interaction based upon the human�human interaction the interface

which results will provide easy interaction with natural communication� It will accept

and assist with mistakes and errors and will provide the user with assistance when the

user is in trouble� Eberts proposes the advantages of systems designed under this ap�

proach are� the interaction mode can be adaptable to the user�s experience level� and it

can also provide a novel computer interface design� The disadvantage of employing this

design methodology is that it is overly dependent upon technological advances such as

natural language processing�

Eberts believes that the best design approach combines all of these approaches� It

is very di�cult in reality to create this type of combination� It is this realization which

has led others to develop the other design methods discussed in this section�

Interface Interaction Methods

In general� there are three distinct techniques of human�machine interface interaction�

They are the command� menu and direct manipulation techniques which are de�ned

in �Whiteside et al�� Shneiderman� �� Jacob� �� This subsection describes

these interaction methods and also displays some experimental data to demonstrate

which method is considered desirable�

CommandDriven Interaction Method� The �rst mechanism for human�machine

interaction was the command driven system� When employing this method� the user

communicates via a specialized language which requires keyboard entry of commands� As

Shneiderman observes in �Shneiderman� �� the use of a command language protocol

requires the user to memorize possible commands and their combinations� One option

of combining complex command sequences is to develop macro commands�

As a user becomes more skilled with a system� it is necessary to provide abbreviations

as the users desire faster and simpler entry techniques� This method reduces the produc�

tion time while retaining relevant information� Maher and Bell �Maher and Bell� ��

designed a man�machine interface for abbreviation oriented interaction� They chose

��

�meaningful abbreviations for their identi�ers� They also permitted multi�line com�

mands which they found were frequently employed�

Menu Driven Interaction Method� An interface utilizing menu�oriented interac�

tion presents the user with various options from which she or he may choose� It eliminates

the user�s need to memorize commands� while providing a clearer understanding of the

command options�

Widdel and Kaster�s �Widdel and Kaster� �� users determined pull�down menus

were faster than command driven input� Although� their study found a higher degree

of error with menu use� This degree of error was concluded to be insigni�cant when

compared to the cost of correcting errors in command input interactions� Shneiderman

�Shneiderman� �� also studied menu driven interfaces� He observed that they eliminate

the user�s need to memorize commands as they provide an explicit list of the possible

commands� He found a di�culty associated with this method is the menu display rates�

If the display rate is slow� it hinders performance and the users become aggravated� This

annoyance is especially prevalent if the user must wait until the entire menu is displayed

before they may enter their choice or other short cuts such as key commands are not

provided�

Direct Manipulation Interaction Method� Stramler �Stramler Jr�� de�nes

direct manipulation as�

A user�computer interface in which the entity being worked is continuously

displayed� the communication involves button clicks and movements instead

of test�like commands� and changes are quickly represented and reversible�

Our interface and those of �Bach� �� Keil�Slawik et al�� are examples of interfaces

which employ direct manipulation�

Hollan et al� �Hollan et al�� view an interface built with this interaction method

as one designed for communication considering the cognitive task the system supports�

They believe the displays support the system but do not guarantee directness� This is

a result of the interface language matching the manner in which the user contemplates

the task� They state�

��

Directness is thus not a property of interfaces but involves the relationship

between the task a user has in mind and the way in which the task can be

accomplished via the interface�

In �Shneiderman� �� he presents the characteristics of a direct manipulation in�

terface which are also described by Jacob �Jacob� �� Jacob� �� They are�

� Continuous representation of the object of interest�

� Physical actions movement and selection by mouse� joystick� touch

screen� etc� or labeled button presses instead of complex systems�

� Rapid� incremental� reversible operations whose impact on the object of

interest is immediately visible�

� Layered or spiral approach to learning that permits usage with minimal

knowledge�

Shneiderman views the advantages of direct manipulation to be� novice�s ability to

quickly learn interaction� expert�s ability to rapidly interact with the system� infrequent

user�s ability to retain operation use� the relative infrequent need for error messages� the

user�s ability to observe immediate action feedback� and the user�s ability to gain system

mastery as they are able to initiate actions� feel in control and are able to predict system

responses�

Jacob �Jacob� �� believes the primary advantages of this interaction method are

psychological as there is a decreased demand on the user�s short and long�term memory�

Long�term retention is reduced as the user must only remember a few commands� Short�

term memory is reduced as changes to objects are immediately available� Memory is also

a�ected by the reduced number of states and modes the user must execute� In general�

as most people �nd recognition memory easier than recall memory� presumably direct

manipulation and menu�driven interfaces are superior as they present the alternative to

the operator� Motor operations should be minimized� such as typing commands� which

implies direct manipulation and menu�driven interfaces are favored�

Direct manipulation interfaces are easiest to apply to domains which permit concrete

graphical representations� They are more di�cult for abstract domains as the object

��

representation may not facilitate user visualization� Another di�culty associated with

direct manipulation interfaces is the rigidly �xed one level of abstraction� This constrains

the user in a situation where a command driven interface would be more �exible� The

users of direct manipulation interfaces are generally not provided methods to create new

commands as may be feasible in command systems�

Both Shneiderman and Jacob have signi�cantly studied this area� Shneiderman� in

�Shneiderman� �� reports when employing this method the user reported�

� Mastery of the system�

� Competence in performance of their tasks�

� Ease in learning original system as well as new features�

� Con�dence in their capacity to retain mastery over time�

� Enjoyment in using the system�

� Eagerness to show it o� to novices�

� Desire to explore more powerful system aspects�

He found that the system aspects which provided the users with these feelings were

related to their ability to observe the object they were interested in� their abilities to

rapidly reverse actions� and the disappearance of the complex command language syntax�

Jacob discusses his �ndings in �Jacob� �� He found direct manipulation permits

the user to directly operate upon the interface objects as opposed to carrying on a

�dialogue about them � A measure of successful direct manipulation implementations is

a low level of cognitive distance� He also found�

� The direct manipulation interface to comprise a collection of many rel�

atively simple individual dialogues�

� The individual dialogues of a direct manipulation interface to be related

to each other as a set of coroutines�

� The dialogue should be speci�ed as a sequence of abstract input and

output events� with layout and graphic details given separately�

�As de�ned by Hutchins et al� �Hutchins et al�� cognitive distance is the mental eort requiredto translate from the input actions and output representations to the operations and objects of theproblem domain itself�

��

� Direct manipulation interfaces have de�nite modes or states� despite

their modeless appearance�

This permits the direct manipulation interface construction to be a combination of �in�

dividual� possibly mutually interacting interaction objects� organized around the ma�

nipulable objects and the loci of remembered state in the dialogue � In other studies

conducted on all three interaction methods� Shneiderman established that experienced

users believe command driven interaction is faster than menu driven interaction� while

novices were found to prefer menu driven interaction�

Whiteside et al�� Whiteside et al�� tested the various system interaction meth�

ods described� The following questions motivated their experiments�

� Are there large and uniform di�erences in the usability and evaluation of systems�

� Are any di�erences which exist related to the class of interfaces command� menu�

iconic�

� Are some types of systems more suitable for certain types of users�

They tested �� users and classi�ed them into novice� transfer and system groups� Novice

users were those with minimal or no computer experience� Transfer users were those who

used interactive computers daily but never the test system� System users were those who

previously used the test system� They tested six command driven systems� three menu

driven systems and two direct driven systems� They found a signi�cant degree of variance

in usability and participant systems evaluations� This included the levels of success

novices had with various systems� Some systems were easily used while others were

very di�cult� They also found the interaction type made little di�erence in the system

usability� Systems which should have been easier to use were misleading� They found the

systems considered easy to use by experts were also considered such by novices� They

determined not only are command driven interfaces di�cult to use� but some users also

found the complex interaction of mouse button presses for direct manipulation di�cult�

They also found menu choices and icons were easily misinterpreted� The most important

�nding was that the interface interaction style can not solve usability problems� Another

��

interesting determination was that more mature systems were considered to have a higher

usability�

As Schneider observes �Schneider� �� when developing the interface system the

designer must realize as users become more skilled with the interaction mode� the mode

must change with their abilities� As the user becomes more experienced� she or he

will desire a more concise interaction while a novice user will prefer a more detailed

interaction�

Display Methods

There exist many varied representations for display methods� Information should be dis�

played in a useful manner for interpretation but not in a manner which overloads the op�

erator�s abilities to understand the displays �Woods� �� It is necessary to display not

only raw data readings but also levels of the abstract�processed data �Frey et al��

The availability of such data levels are useful for di�erent tasks� There exist tasks for

which the raw data may be the most informative display� while for other tasks� a higher

level of data abstraction is more meaningful� There are also considerations of color use

�Kraupner�Stadler� �� textual displays �Cox and Walker� �� and pictorial displays

�Jacob� �� Shneiderman� ��

Whalley �Whalley� �� describes three considerations for display design� First� the

information should be easily viewable and readable� Secondly� the information layout

for a task should consider the user�s ability to check static� dynamic� interactive� and

abnormal information types� Finally� the integrated system should provide the user with

a complete and accurate understanding of the system�

Misue and Sugiyama �Misue and Sugiyama� �� believe users have speci�c display

requirements� They feel users desire displays which enable the viewing of necessary

system details� display the entire display objects� permit the simultaneous display of

complete information while allowing the display of only one image at a time and �nally

the consideration of the hardware display capabilities�

Rouse �Rouse� �� discusses another perspective of display design� termed display

scanning� The amount of time the operator spends viewing data displays and the likely

transition of displays should be considered during display development� For instance� he

��

suggest displays which require a signi�cant amount of attention from the operator should

be centrally located� Those displays between which many transitions occur should be

located in close proximity to one another or merged into one display� As our model has

one main window which permits various data to be displayed simultaneously� we have

essentially merged many of our displays into one main display�

Traditionally� as Woods observes �Woods� �� in industrial control rooms the data

displays are dominated by the one display element philosophy which focuses upon the

availability and accessibility of raw data called base data units� He cites many issues

which must be considered when designing an interface for data availability� Generally�

base data units are assigned to one display mode in one location and are not readily

integrated with other data units� He believes the data overload problem is more prevalent

in systems designed for data availability� It is more di�cult to extract the signi�cance

and meaning of the data in relation to other data� The operator must remember and

mentally extract such meaning� They fear that an interface designed strictly for the

purpose of data availability will not assist the operator when making judgments which

should be based upon an entire data set� Thus the operator will base his or her decisions

on partial data information�

Bar�eld and Kim �Bar�eld and Kim� ��a� observed it is di�cult and tiring for

the human operator to visualize the data in three dimensions when observing three�

dimensional information on a two dimensional screen� This is due to �the relationship

between the perspective geometry parameters used to design such displays and the ac�

curacy with which observers can reconstruct the spatial information contained within

the perspective projection � Graphical images provide the operator with a data im�

age� This information is easier to understand and to remember �Herot� �� Graphical

user interfaces may be classi�ed into two categories� those which are two�dimensional

and those which are three�dimensional representations of the environment or system

information� The use of three�dimensions provides the operator with a sense of real�

ity �Regan and Pose� �� Borys� �� have also considered ergonomic issues with

graphical display methods�

Hwang and Wang conducted experiments to study the e�ects of display format types�

volumes of data and the layout of data in the human�machine interface� This experiment

��

is described in �Hwang and Wang� �� They found format type had a signi�cant e�ect

upon the operator�s task executions and that graphical formats were better than digital

formats� They felt the graphical format reduced the operator�s mental overload� They

also found when the volume of data increased the operator�s performance was increased

by utilizing a proper layout method�

�� Graphical User Interfaces

A Graphical User Interface GUI as de�ned by Stramler� in �Stramler Jr�� is an

interface for�

the use of direct manipulation and icons or other graphical symbols on a

display to interact with a computer�

There has been considerable research in the graphical user interface area�

�Pejtersen and Nielsen� �� employ iconic interfaces� �Eichelberg and Ackermann� ��

review general topics for object oriented systems� �Heger et al�� study decision

support systems� �Thomas and Goss� �� discuss general topics for three�dimensional

graphics� �Chen and Trivedi� �� addresses multiple sensor based robots� work within

an outer space domain was conducted by �Askew and Diftler� �� etc� As our inter�

face incorporates a three�dimensional graphical user interface this will be the focus of

this subsection�

Foley and Van Dam �Foley et al�� state interactive graphics�

permit extensive� high�bandwidth user�computer interaction� This signi��

cantly enhances our ability to understand data� to perceive trends� and to

visualize real or imaginary objects that we can explore from arbitrary points

of view�

There have been many applications combining graphics and raw images� Bejczy et

al� �Bejczy et al�� incorporate a �phantom robot into their time�delayed teleop�

eration system� The �phantom�s purpose is to permit the operator to view both �a

real�time simulated display of the manipulator and an accurate display of static objects

from the delayed video� They overlay real time graphics onto the video camera image�

��

The �phantom provides the operator with an indication of the actual manipulator lo�

cation and the manipulator�s delayed position via the image providing the operator with

predictive displays�

Matsui and Tsukamoto �Matsui and Tsukamoto� �� discuss the development of a

Multi�Media Display employed in teleoperation tasks� The Multi�Media Display permits

the superimposition of a model onto the real images� This allows the operator to deter�

mine the di�erence between the model and images� The display incorporates multiple

visual medias� multiple windows� real�time graphics and stereoscope� It permits the dis�

play of real images� graphical models and text on one screen� They incorporate video

images and CAD models to learn the environment in which they will perform teleop�

eration tasks� This is further discussed in �Hasegawa et al�� The environmental

modeling system displays an image on a monitor� When the operator recognizes an ob�

ject she or he teaches this objects name to the system and then designates points on the

object� These points are used to determine the proper three�dimensional shape from a

CAD database�

Sayers and Paul �Sayers and Paul� �� employ a three�dimensional graphical user

interface as part of their master station in their time�delayed teleprogramming system�

This interface employs a virtual reality display� They developed synthetic �xtures which

permit very precise slave manipulator control� They also employ overlays of their working

environment images to calibrate their master station�

At Sandia National Laboratories there is an initiative to employ intelligent

robotic control for waste remediation� Their proposed system is described in

�Drotning et al�� Christensen et al�� Christensen� �� and includes a three�

dimensional graphical user interface which models the environment and controls the

robots� The initial model incorporates the known information about the environment

and unknown information is detected utilizing an ultrasonic proximity sensor system� a

metal detection system� a ground penetrating radar system� structured lighting range

mapping and ancillary systems� Cooke and Stans�eld �Cooke and Stans�eld� �� are

developing a method of interactively building graphical models employing telepresence

and virtual reality� Their method integrates live video with the incomplete graphical

��

model to permit the operator to incorporate previously unknown objects into the graph�

ical model�

One aspect of graphical user interfaces concerns the user�s interaction with the graph�

ical programs� Dai �Dai� �� suggests the system operator be provided with various

levels of data� The interface should permit the operator to �modify some parameters and

redo part of� not all of� the work according to the user�s guidance in real time� He sug�

gest the interaction functions should be centralized as this would simplify the program

structure and separate the graphical system and the application functions which he calls

centralized� application�oriented interaction control� He de�nes the interaction rules with

graphical elements such as menus� and callback functions� He proposes the interactions

for all system components should be handled via a centralized process associated with

the graphical user interface� He proposes that this model increases work e�ciency and

creates a system with simpler structure which is more device independent� Utilizing

this model� the user is able to obtain increased support via directly manipulating the

graphics�

Bar�eld and Kim conducted a study� described in �Bar�eld and Kim� ��b�� to inves�

tigate the user�s feeling of realism upon viewing images created with computer graphics

rendering techniques and computer synthesized images� They created an image of an

apple and designed the experiment such that they varied the number of lighting sources�

the number of specular highlights� the number of shadows� the presence or absence of

a color map� and various shading techniques� They created pictures of the apple with

changes amongst the various variables� The subjects then viewed these pictures� They

found�

there are diminishing returns in terms of computational resources required to

render realistic three�dimensional images versus subjective ratings of image

realism� The additional computing resources used to render multiple spec�

ular highlights and shadows were not e�ective in producing higher ratings

of subjective realism given the images and rendering techniques evaluated in

this study�

As is observed from this subsection� there are many approaches to incorporating

graphics into the human�machine interface� There also exist issues which are strictly

��

graphics oriented� such as how realistic the human perceives the graphical image� We

employ a three�dimensional graphical model and permit the overlay of real�time system

images� We do not build intricate models of the environmental objects� The image

overlays provide the supervisor with a realistic environmental perspective�

�� Human Factors Considerations

There are numerous human factors considerations beyond basic interface design� They

relate to the operator�s optimal workload� the operator�s desire to use the system and the

operator�s ability to correctly perform the fault diagnosis task� This subsection discusses

these issues�

An interesting observation by Edmonds �Edmonds� �� which epitomizes the re�

quirement for human�machine interface developers to consider human factors issues is�

The more interactive a system is� and the more inventive or unpredictable

the human�s part in it is� the less we can discover from task analysis� etc� and

the more we must rely on evaluation of the performance of the system in use�

Workload

The human�s workload level is an predominant factor when developing a human�machine

interface� A common question as posed by Rouse �Rouse� �� is�

What fraction of the task responsibilities should be allocated to the human

at a particular instant in time in order to keep him su�ciently involved and

motivated to perform acceptably over weeks� years� or a whole career�

Whalley �Whalley� �� claims the average operator workload should be contained

within the �� and �� percent range of mental capability� She de�nes workload as �the

time required to complete actions against the time available with the estimated workload

assessed against the ergonomic recommendation of between �� and �� percent �

Rouse �Rouse� �� observes that situations of short�term mental workload stress are

tolerable though they are not tolerable over the long term and may lead to human error�

He has studied the dynamic division of tasks and believes tasks should not be strictly

divided between the human and the computer� Instead he suggest that in situations

��

where the task may be performed either by the human or the computer� the task should

be allocated to the one with the lowest current workload� He proposes this method

will better utilize the system�s resources as well as create less variability in the human�s

workload�

Mental Workload

Performance

Figure �� The inverted �U hypothesis for performance vs� mental workload�

Workload may also be de�ned as a function of factors �Sheridan� �� These factors

are composed of those dependent and independent of the operator� Operator dependent

factors are� the operator�s perception of the task demands� his or her quali�cations�

capacities and motivations� as well as the operator�s behavior� The operator independent

factors are composed of� the task objectives� the hardware and software resources being

utilized� and the environmental conditions such as lighting� Figure �� displays the �U

hypothesis of a user�s mental workload capabilities as related to the user�s performance�

When the mental workload is small� the operator tends to have a lower performance level

as there is not enough to hold his or her attention� The operator�s performance will also

degrade when his or her mental workload is greatly increased� The optimal workload

level is the peak of the inverted �U � Thus optimal performance is related to the optimal

mental workload but this level is yet to be fully de�ned as it varies amongst operators�

Rencken and Durrant�Whyte �Rencken and Durrant�Whyte� �� propose an adap�

tive task allocation approach to deal with operator information overload� Their approach

permits the operator and the computer to independently determine decisions� The com�

puter is employed for backup decision making purposes when the operator appears un�

able to cope with all currently required decisions� Their proposed model is shown in

Figure �� The �measurement system determines the operator�s performance level

��

TaskAllocator

QueueingModule

MeasurementSystem

PredictedAllocationModule

Computer

Human

Arriving Tasks

Figure �� The adaptive task allocation human�computer interface�

and attempts to determine future performance� The �queuing module describes the in�

teractions amongst the operator and computer servers and the remaining portions of the

system� The �predicted allocation module �rst determines if the human requires assis�

tance with decision making tasks� then it determines the optimal task allocation between

the operator and the computer� The �task allocator assigns the tasks to the operator or

the computer� They found when the operator was assisted by the computer the overall

system performance improved while the operator�s performance level remained high�

The operator�s workload is an important aspect in the system design as all actions the

operator is expected to perform will be a�ected� This review demonstrates the di�culty

associated with determining the proper workload level as it varies for each individual�

Usability

A primary concern in interface development is whether the user will like and want to use

the interface� Bevan et al� �Bevan et al�� pose the question� �What is usability� �

There is no single de�nition therefore they describe their various views which comprise

usability�

� The product�oriented view that usability can be measured in terms of

the ergonomic attributes of the product�

��

� The user�oriented view that usability can be measured in terms of the

user�s mental e�ort and attitude�

� The user performance view that usability can be measured by examining

how the user interacts with the product� with particular emphasis on

either

� ease�of�use�� how easy the product is to use� or

� acceptability� whether the product will be used in the real world�

� The contextually�oriented view that product usability is a function of

the particular user or class of users being studied� the task they perform�

and environment in which they work�

Product Attributes

Task

Product attributes user is aware of

User’s performance with product

AttitudeUnderstanding

and mentaleffort

Organizationalcontext

Environmentalcontext

Figure �� The determinants of usability�

The elements which they feel compose the determinants of usability are displayed in

Figure �� These determinants include those relevant to the user� the task and the

environment� The product attributes include the interface and its general system prop�

erties� Finally� they believe the product itself is not usable or unusable but is composed

of the attributes which will determine its usability particular to the user� task and envi�

ronment��Bevan et al� �Bevan et al�� de�ne easeofuse as how well the product can be used� whether the

operator will use the product and how the user will employ the product�

��

Cox and Walker �Cox and Walker� �� de�ne a usable interface as one which is

transparent� controllable and �exible� They believe the user must conclude the inter�

face is satisfactory for the designed task� They propose a combination of the following

considerations to determine the product usability�

� Functionality� Is the user able to complete the required tasks�

� Understanding� Can the interface be understood by the user�

� Timing� Can the user complete the tasks within a reasonable time frame�

� Environment� Do the required tasks conform to the user�s environment�

� Safety� Will the system harm the user�

� Errors� Does the user make many errors during use�

� Comparisons� How does this interface compare to other manners in which the user

would carry out the task�

� Standards� Is this interface similar to other interfaces the user may utilize�

Rengger �Rengger� �� discusses the review of ten years of published materials on

usability conducted in conjunction with the ESPRIT MUSiC project� This review at�

tempted to create generic classes and types of usability measures based upon perfor�

mance� He determined four classes of usability measures based on the literature review�

� Goal achievement indicators

Indications of the level of success with which user�s attained their goals

where e�ectiveness is an indicator of goal achievement� The indicators

he describes are� success rate� success ratio� failure rate� failure ratio�

success to failure ratio� accuracy� and e�ectiveness�

� Work rate indicators

Indications of the rate at which users worked or attained their objectives�

Terms such as e�ciency and productivity are indicators of work rate�

The indicators he describe are� speed� completion rate� completion ratio�

e�ciency� productivity� productivity period� and productivity gain�

��

� Operability indicators

Indications of the user�s ability to make use of the system�s features and

the level of problems encountered while doing so� The indicators he

discusses are� error rate� error density� problem rate� problem density�

operability� function usage� and interactive density�

� Knowledge acquisition indicators

Indications of the user�s ability and e�ort to learn� understand and re�

member how to use a system� The indicators he discusses are� learn�

ability� learning period� and learning rate�

�Heinecke� �� and �Prothero� �� have also conducted similar studies for mea�

suring usability� �Wietho� et al�� approached the problem from a biological an�

gle� measuring the user�s heart rate etc� �Gunsth�ovel and B�osser� �� employs the

Skill Acquisition NEtwork SANE� model of cognitive skills in their studies and

�Gimnich et al�� conducted his studies on a direct manipulation graphical inter�

face� As can be observed� there are many approaches to determining this important

measure� The only true measure for each individual system is to test the actual users

and ask their opinions�

Fault Diagnosis

Another human factors concern is the human operator�s ability to correctly diagnose

problems� This issue has been addressed by many in the �eld� �Rouse and Hunt� ��

present numerous experiments designed to test the human fault diagnosis task� This

data was utilized to create various models to predict the human�s performance for such

a task� Based upon these models they deduced�

� Humans are not optimal problem solvers� although they are rational and

usually systematic�

� Human problem solving tends to be context�dominated with familiar� or

even marginally familiar� patterns of contextual cues prevailing in most

�As de�ned in �B osser and Melchior� �� The Skill Acquisition NEtwork �SANE� toolkit is a set ofCASE tools based on a language for cognitive modeling of user behavior in well de�ned domains such ashuman computer interaction�

��

problem solving�

� Human�s cognitive abilities for problem solving are de�nitely limited�

However� humans are exquisite pattern recognizers and can cope rea�

sonable well with ill�de�ned and ambiguous problem solving situations�

They found when the human performed suboptimally it was attributed primarily to the

human�s lack of problem understanding� They also found humans could successfully deal

with unfamiliar problems� They concluded that in order to take advantage of the human�s

cognitive abilities one needs to develop methods to overcome the human�s cognitive

limitations�

Sheridan �Sheridan� �� observed the need for operator interfaces to be more �trans�

parent to the actual system� This would facilitate the operator�s ability �see through

the displays to observe the system�s actions� He de�nes human error as �an action that

fails to meet some implicit or explicit standard of the actor or of an observer � The

following are his proposals for reducing human errors�

�� Design the interface to prevent error� this includes providing proper feedback and

redundant information�

�� Properly train the system operators� speci�cally for emergency situations�

�� Restrict exposure such that actor opportunity is limited�

�� Warn or alarm the operator while not overloading the operator�s mental capabilities

and creating so many warnings they begin to ignore them�

�� Permit the operator to correct human errors when they occur�

Morris and Rouse discuss a human error study and the situations which promote

it in �Morris and Rouse� �� They relate that human �slips typically occur during

the automatic routine task execution� when the operator is distracted or preoccupied� is

working in environments which are familiar and there are few unexpected events� They

also relate human mistakes occur more frequently under the following conditions� when

making a decision which requires the simultaneous consideration of numerous variables�

prominent environmental cues lead the human to an incorrect solution� when a solution

��

is used which is incorrect for the current failures but was su�cient for previous similar

failures� and if the solution must be approached in a new manner� The determination of

generalizations which quantify why human operators make errors is very di�cult�

Failure detection is extremely important in any system� The determination of why

a human misinterprets failure messages or does not employ the proper methods for

failure recovery is of grave importance to system designers� This subsection reviewed the

methods which may assist the operator during failure situations and demonstrated the

di�culty of determining the best methods to assist the operator during a failure�

�� Human Factors Analysis Design

There are many aspects to consider when developing a design for human factors analysis�

There are variables to constrain and various methods with which to conduct the analysis�

This section will cover some of the issues one must consider when developing a human

factors analysis similar to that which we describe in Chapter Five�

First one must determine what type of evaluation study will be conducted� Preece

et al� �Preece et al�� and Hix and Hartson �Hix and Hartson� �� de�ne a forma�

tive evaluation to be one which helps determine if an un�nished product will be useful�

A summative evaluation is used to make judgments about a �nished product and are

generally conducted at the completion or near completion of product development� Gen�

erally the product is compared to similar products or previous versions of the product�

Typically� such testing is very focused� Once one determines which type of study is to

be conducted they must then design the experiments� As we have conducted a formative

study in our design this is the strategy we will concentrate upon although many of the

concepts are also applicable to summative studies�

The initial step to the design process is to de�ne the purpose of the study and to

formulate the research question or hypothesis to be answered� An important component

of this hypothesis is the comparisons which will be made� Once the comparisons are

de�ned the experimenter is able to de�ne the independent and dependent variables�

When de�ning these variables careful consideration must be taken to ensure there exist

no confounding variables�

��

The next consideration relates to the subjects and which experimental design shall

be employed� independent� matched subject or repeated�measures� Preece et al�� in

�Preece et al�� de�ne an independent subject design as randomly allocating a sub�

ject group to one of many experimental conditions� A matched subject design pairs sub�

jects and then randomly allocates them to the possible conditions� A repeated measures

design requires all subjects to appear in all experimental conditions� The experimenter

must also determine how many subjects are required for the study� This is dependent

upon the number of experimental conditions to be tested as well as the design� In

�Virzi� �� he shows that

�� of the usability problems are detected with four or �ve subjects�

�� Additional subjects are less and less likely to reveal new information�

and

�� The most severe usability problems are likely to have been detected in

the �rst few subjects�

This is an important consideration� particularly when using a large subject group is

considered to be too expensive� cumbersome and�or time consuming�

Tasks which the subjects will execute must be de�ned to be complex enough to test

the system abilities� They must also be fully understandable by the user and completable

in a reasonable time frame� Tasks which may require a long time span to complete my

induce fatigue and boredom in the users thus introducing confounding variables� The

designer must also determine how training will be conducted� how long training should

last and if practice should be permitted and its duration� As training and practice can

a�ect the study results� these two factors must be given careful consideration� Training

must be identical between subjects as well as the length of practice sessions� if not

confounds may be introduced�

The experimental designer must ensure external factors do not introduce confounds

into the experiment� Such considerations include environmental factors such as the

temperature� the noise level and tra�c in the experimental environment� Confounds

may also be introduced through the subjects used in the experiments� The subject

group should contain subjects from all experience levels� age groups etc�

��

The designer must also de�ne the objective and subjective data to be gathered from

the experiment� As �Muckler and Seven� �� state� �knowledge is valid and objective

only when it is not dependent on human processes subjective states � There is a great

demand to extract objective performance measures in human factors analysis but in re�

ality �subjectivity is inevitable �Muckler and Seven� �� They believe objectivity and

subjectivity are not easily de�ned because they are tied to human experience� judgment

and feeling which are involved in the derivation of reality� Muckler and Seven state� �Ob�

jective measures deal with what is happening� Subjective measures can consider how well

one is coping� the resources consumed in coping� resources still in reserve� past experi�

ence� present knowledge� and probable level of motivation � Examples of objective data

are information related to task completion times� the steps the subjects used to complete

the tasks� and other measures which are not based upon the subject�s feelings or beliefs�

Examples of subjective data include perceived workload measures� questionnaires and

post�experimental interviews�

The preferred method of collecting objective data is to incorporate automatic saves

of the information into the interface� This method allows the experimenter to observe

the subject�s behaviors in an unobtrusive manner� Direct observation should be avoided

as it may cause the Hawthorne e�ect� Another indirect method of obtaining objective

data is to video tape the subject while executing the task� If possible the subject should

not be aware of the video taping process� If this is not possible� video taping should not

interfere with the subject�s ability to e�ectively execute the assigned task�

There are many examples of questionnaires used to collect the subject�s opinion on

a system� The questionnaire design must be carefully considered such that the ques�

tions are not ambiguous and will extract the desired information from the subject� As

�Eberts� �� explains� a questionnaire must be evaluated on its reliability� validity� and

factor loadings� The reliability of a questionnaire refers to �the probability that a ques�

tionnaire item will be answered the same way on two or more administrations of the

items � A questionnaire�s reliability can be determined by duplicating items� not neces�

sarily identically worded� on the questionnaire� The questionnaire validity relates to how

Preece et al��Preece et al�� de�ne the Hawthorne eect to occur when direct observation causesthe users to be constantly aware of their performance being monitored� which in turn alters their behaviorand performance�

��

well the questionnaire measures what it is designed to assess� This can be di�cult to

determine and Eberts describes methods to determine this� Factor analysis �determines

the number of factors which may be underlying the items and requires the designer to

examine the factors and name them�

Questionnaires can be composed of closed�ended or open�ended questions� Closed�

ended questions require the subject to choose an answer� Many closed�ended question�

naires employ the Likert rating scale� The Likert scale is a multi�point scale charac�

terized by a set of number choices which is always odd� Each of the numbered values

is associated with an adjective which describes the subject�s attitude for the particular

question� Eberts has compiled a list of possible adjectives employed with Likert scales in

�Eberts� �� Preece et al� present various methods of presenting such information to

a subject in �Preece et al�� Open�ended questions permit the subject to provide

their own non�restricted answers which put forth their personal opinions� Such questions

may not be as reliable as closed�ended questions as the subject may forget experimental

aspects� Shneiderman presents a highly referenced generic user�evaluation questionnaire

for an interactive system in �Shneiderman� �� This particular questionnaire employs

both closed�ended and open�ended questionnaires�

The subject�s workload level may be measured in many manners� via subjective mea�

sures� secondary task measures� or physiological measures� Subjective measures require

the subject to rate their perceived workload� typically their feeling of exertion and�or

e�ort during task execution� The secondary task measure method requires the user to

concurrently perform two tasks� As described by Eberts� �the task of central importance

is called the primary task and the secondary task is used to measure the workload of the

primary task� The secondary task serves as an indicant of the spare attentional capacity

from the primary task � The physiological methods measure the subject�s physiologi�

cal changes� This information can be collected by observing an electroencephalograph

EEG� eye movements� cardiac functions or muscle functions� As we have employed

subjective workload measures we will concentrate on some methods of obtaining the

perceived workload�

When employing subject workload measures one must consider the method�s sensitiv�

ity to various types of workload measurement� Nygren conducted a comparative study of

��

the National Aeronautics and Space Administration Task Load Index NASA TLX and

the Subjective Workload Assessment Technique SWAT methods� A description of this

study and of the two methods can be found in �Nygren� �� He found that both scales

provided a high reliability and sensitivity measure for perceived workload� although he

felt NASA TLX�s bipolar scale provided more sensitivity� The method was also be�

lieved to be useful for many general systems while the SWAT method was restricted to

cognitively modeled systems�

Hill et al� report in �Hill et al�� a study which reviews four subjective workload

measures� They studied the NASA TLX and SWAT methods as well as the Modi�ed

Cooper�Harper MCH scale� and the Overall Workload OW scale� Their overall results

found that all four techniques were sensitive to di�erent workload levels and each was

an acceptable technique of measuring workload� Of the four techniques they found the

NASA TLX and the OW methods were �consistently superior when considering sensi�

tivity� as measured by factor validity� and operator acceptance � They found the NASA

TLX method provided the experimenter with more detailed and diagnostic data� Their

comparison was based upon each method�s� sensitivity� operator acceptance� resource re�

quirements and special procedures� We chose the NASA TLX method for our workload

analysis measurements�

Upon completion of determining all these factors and completing the necessary prepa�

rations� the experimenter must then require the subjects complete the experiments and

collect the data� After the experimental sessions are complete� the data is statistically

analyzed for signi�cance� Our Human Factors Experimental Design chapter� Chapter

Five� describes the experimental design we employed� The strategy considers the factors

described in this section�

�� Human�machine System MediationIntervention

A feature of the mediation hierarchy which we have partially implemented in MASC is the

operator�s ability to intervene into all system levels� Traditionally� most human�machine

interfaces permit the operator to act as a monitor or supervisor without permitting

signi�cant interaction into the systems processes� This section discusses the work of

others who have employed intervention�mediation into their systems�

��

Sheridan �Sheridan� �� states�

the supervisor intervenes when the system state has reached the designated

goal and the computer must be retaught� or when the computer decides the

state is su�ciently abnormal and asks the supervisor what to do� or when

the supervisor decides to stop the automatic action because the system state

is not satisfactory�

The point of intervention may be in�uenced by� the criterion which de�ne an abnormal

state� the tools which the supervisor can employ for intervention� the criteria for risk�

taking� the decision on whether to wait until more information is collected or to intervene

immediately� or the supervisor�s mental workload� He notes that the intervention stage

is the most error sensitive� as human error is more apparent at that time�

Ammons �Ammons� �� suggest models for aiding real time �exible manufacturing

systems FMS� Her model consists of two components�

�� computers and microprocessors to perform automated control and rout�

ing activities� and

�� humans to supervise the automation� to monitor system �ows and out�

puts� and to intervene to diagnose and either correct or compensate for

unanticipated events�

Ammons focuses on identifying those scheduling models which could be applied to aiding

the real time control� She proposes three interaction levels� production planning� release

scheduling and item movement� This hierarchy is established such that the levels are

completed at the appropriate time in order for the solutions to be used by the other

levels� either above or below� in the hierarchy� The production planning level is the

highest level and it establishes the strategic direction of the FMS scheduling outcome�

For this level she proposes the human interactively establish the scheduling goals as

well as provide the human with the ability to formulate various �what if scenarios

for demand changes� machine downtime� etc� The release scheduling level attempts to

introduce new materials into the system while maintaining a balanced system� Again

at this level the human operator may compile �what if scenarios employing simulation

��

models or queuing networkmodels for workstation downtime� prioritize production items�

or alter the factors of the multicriteria problem� The item movement level determines

routing for items within the production system� This level may provide the human with

the system�s sensitivity to certain changes� This hierarchy does permit the operator to

work within the FMS system �ow but does not provide interactions with the various

hardware system components� The principle of interacting with the FMS is similar to

our high�level principle but is also domain speci�c and still a high�level supervisory task�

McKee and Wolfsberger �McKee and Wolfsberger� �� believed that a system was

required which permitted the operator of a robotic system to give the system commands

but not instruct it on how to perform the task� They proposed a graphical human�

machine interface system to permit the human operator to work with representations

of objects with in a three�dimensional model of their working environment� Their goal

was to create an interface in which the operators would control robots at a task level

rather than teach the robot or write a program to do it for a teleoperational robot� This

particular system is an example of the high�level interactions created for robotic systems�

It does not permit the operator to interact with the lower system levels�

Hasegawa et al� �Hasegawa et al�� incorporate a high level of intervention into

their telerobotic system� Their system is composed of manipulation skills with an en�

vironmental model� The purpose of the manipulation skills is to provide reliable task

execution when encountering unavoidable errors and uncertainties� The system will

autonomously execute speci�ed tasks based upon the model and skills� When they en�

counter a system failure during autonomous execution the operator issues a command

which switches the system to master�slave control� The operator can then teleoperate

the robot to recover from the failure situation� Upon recovery� the system switches back

to autonomous mode� They have attempted to provide autonomy into a teleoperation

system thus creating a semi�autonomous system� This is similar to our idea of a semi�

autonomous system but it is very high�level� It does not provide the operator with

mechanisms to avoid the failure or to work within the lower system levels�

��

Hirai et al� �Hirai et al�� have incorporated multi�level human interaction into

their Model Enhanced Intelligent and Skillful TEleoperational Robot MEISTER sys�

tem� They permit the operator to teach the system information via their teaching�

executing method� Tasks are simulated and the operator either approves the task or

manually controls the task execution� They permit the operator to superpose control

schemes onto the system� The rate control scheme permits the operator to create precise

linear motions in the joint angle or Cartesian coordinate spaces� The incremental control

scheme permits the operator to increment the slave manipulator position to an amount

speci�ed by the operator� The indexing scheme permits the master manipulator to be

posed at a position which is comfortable for the operator while the slave manipulator

is posed for task execution� The Programmed Control Scheme permits the operator to

create commands utilizing the master manipulator and while the slave has been work�

ing autonomously it will then execute the operator created commands� These schemes

permit the operator to e�ectively work with the manipulator while also providing a level

of autonomy� The MEISTER system is a semi�autonomous system with the ability to

work with the manipulator for better control� This system provides various interaction

levels but these interactions are still on a higher level than what is proposed with the

mediation hierarchy�

A human machine interface applied to multiple robots is currently being developed at

the Chemical Engineering Laboratory RIKEN �Yokota et al�� Suzuki et al��

Suzuki et al�� They believe the human interface should permit the operator to

know the current system state as well as permit intervention when necessary� Their

interface is composed of three modules� The �Presentation Interface module informs the

operator of the system conditions and permits operator inputs� The �Operator Dialog

Manager module coordinates message passing between the operator and the robotic

agents� The �System Monitor module permits the monitoring of system information�

So far their publications concentrate solely upon the monitoring and communications

of the multiple robotic agents� They have not presented how the human will interact

with the agents accept to state the human will communicate in a similar method to that

employed with the robotic agents� They have presented a prototype of a two�dimensional

human interface� thus one is lead to believe that they have yet to formalize their theory

��

and developed a complete human interface to their system�

This chapter stated our problem and outlined the remainder of the document� It also

presented a literature review of the various topics which are related to this dissertation�

��

Chapter �

Multiple Agent SupervisoryControl System �MASC� andApplication Description

We have developed a human�machine interface to interact with multiple mobile agents�

The interface has been developed for the University of Pennsylvania General Robotics

and Active Perception GRASP Laboratory�s multiagent project in conjunction with

four other graduate student project members� Therefore� this Chapter provides a brief

overview of the multiagents project and then the Multiple Agent Supervisory Control

System MASC� As this interface has been developed in conjunction with the multia�

gents project� most of the software processes which we have integrated into the MASC

system were developed for the multiagents project�

�� Multiagents Project

The purpose of the University of Pennsylvania General Robotics and Active Sensory

Perception Laboratory�s multiagents project has been to investigate the coordination

and monitoring of multiagent systems for intelligent material handling and is described

in �Adams et al�� Bajcsy et al�� The contributions of this work have been

an improved understanding of the fundamental problems underlying the control and

coordination of multiple agents and well as the development of algorithms for intelligent

exploration� organization and coordination of multiple agents�

��

�� Multiagents Architecture

The multiagents system is composed of four mobile platforms and the human�machine

interface� The mobile agent bases are TRC Labmates� A six degree�of�freedom manip�

ulator is mounted on each of the manipulatory agent bases� Since the platforms have

three degrees�of�freedom� there exists an extra degree�of�freedom in the three�dimensional

Cartesian space� Therefore� they are de�ned as redundant manipulators for more infor�

mation see �Wang� �� Wang and Kumar� �� The observation agents are equipped

with various sensing modalities and a general purpose workstation SPARC�� The obser�

vation agents also utilizes WINDATATM radio ethernet communications� The human�

machine interface has been developed on a Silicon Graphics Inc� Indigo�� The low level

software interface� PENGUIN �Sayers� �� was developed by Craig Sayers� It is written

in C and employs X�Motif windows and the Silicon Graphics Inc� Graphics Library�

The higher level interface is also written in C and employs the Silicon Graphics Inc�

ImageVision Library as well as the TCP�IP communications software�

Figure �� The Observation Agents� left SensorBot� and right VisionBot�

The two observation agents are portrayed in Figure �� The sensorBot� left� is

equipped with the following sensing modalities�

��

� a partial belt of sixteen PolaroidTM ultrasound and infrared sensors�

� a stereo camera pair�

� a light�striping device which projects three planes of light in front of the agent

on the ground and a camera o�set vertically which uses elementary projective

geometry to detect when an object intersects any of the light planes�

The ultrasound sensors are used to detect objects in the environment� As this sensing

modality may be unreliable� �Mandelbaum� �� the infrared sensors are used to verify

the ultrasonic readings� This information is currently employed to detect features in the

environment such as wall�like objects and corners� This information may also be utilized

to localize the SensorBot relative to these features� A stereo algorithm computes a

localized correlation and extracts three�dimensional information about the environment�

The stereo pair may also be employed for visually guided obstacle avoidance described

below� The light�striping mechanism is employed to detect objects and then obtain the

two�dimensional object information� It may also be utilized to localize the other agents

relative to the localized SensorBot� The odometry and heading readings are utilized to

monitor the agent as it moves through out the environment� This information may be

inaccurate due to slippage� etc�

The VisionBot� Figure �� right� is equipped with a stereo pair as well as a pan

platform� The stereo pair are employed for visually guided obstacle avoidance which de�

tects obstacles in the environment and then avoids them� The obstacles are detected via

the di�erence between the stereo images after applying the inverse perspective mapping�

This mapping is used to construct a free space map for the common �eld of view for

each camera see �Kosecka� �� for further details� The pan platform is composed of

a camera and a turntable� This mechanism is used to track objects or other agents in

the environment further details can be found in �Kosecka� ��

The PumaBot� Figure �� right� is equipped with a Puma �� Manipulator and

the ZebraBot left in the �gure is equipped with a Zebra�ZERO Manipulator� As

��

Figure �� The Manipulatory Agents� ZebraBot left and PumaBot right�

mentioned these are six degree�of�freedom manipulators� These agents have no sen�

sors other than force feedback and are therefore �blind � They are primarily em�

ployed for the object manipulation and relocalization� The primary algorithms de�

veloped with the agents have been for testing the coordination of manipulation and

locomotion as well as methods for redundant robots� The coordination of manip�

ulation and locomotion determined the preferred con�guration to carry objects and

how to recon�gure the platforms in order to avoid obstacles or to pass through small

passageways� while maintaining the carrying grasp� For full details of this work see

�Yamamoto� �� Yamamoto and Yun� �� Yamamoto and Yun� �� The redun�

dant robot research focuses upon �the determination of joint motions for a given end

e�ector displacement in kinematically redundant manipulators and is described in

�Wang and Kumar� �� Wang� ��

�� Multiagents Experiments

The experiments we performed with these agents employed the observation agents to

guide the �blind manipulatory agents� This occurred via communications received from

the observation agents to the MASC interface and then the MASC interface passed this

information on to the manipulatory agents� Thus when executing tasks the observation

��

agents not only gather sensory data for themselves but also for the manipulatory agents�

The observation agents are required to sense whether passageways are wide enough for

the manipulatory agents to pass through when carrying objects in the side by side con�

�guration or instruct them to change to a serial� follow the leader con�guration� As

the observation agents are not equipped to physically monitor the manipulatory agent�s

actions� this task was assigned to the human supervisor� One observation agent was posi�

tioned such that optimal images and other available sensory information can be provided

to the human supervisor for the monitoring of the manipulatory agents�

The actual experiments employed the manipulatory agents to carry an object while

the VisionBot explored the path speci�ed by the human supervisor� The VisionBot�s

task was to determine �rst the existence of all obstacles or passageways as well as if a

passagewaywas wide enough for the manipulatory agents to pass through� The SensorBot

was employed to remain in a position from which images of the manipulatory agents were

returned to the human supervisor� Thus the SensorBot followed the manipulatory agents

to the goal point� In the experiments we �rst instructed the agents to move from point

A to point B without scoping the path for obstacles� The purpose of this experiment was

to verify the agents were able to properly obtain the information necessary to complete

the speci�ed path as well as to verify the human supervisor was able to monitor the

manipulatory agents through the SensorBot�s images� The second experiment employed

the visually guided obstacle avoidance to scope the path in an environment where no

obstacles existed� The third experiment was similar to the second experiment except

that there existed an object along the desired path from point A to point B� The purpose

of this experiment was to verify that the information returned from the VisionBot would

be su�cient for the manipulatory agents to avoid the obstacle� This information was

found to be su�cient� The fourth experiment simulated a narrow passage way in which

the manipulatory agents were required to recon�gure from the side by side formation

to a serial formation� These four experiments were each successful and the agents were

capable of completing their assigned tasks�

In each experiment� the human supervisor speci�ed a goal point for the initial agent

in the con�guration� as seen in Figure �� The VisionBot then moved towards the goal

point employing its path following method� Information regarding the agent�s odometry

VisionBot

Goal Point

Initial Point

Figure �� The experimental set�up

and heading were sent to the MASC interface and stored for use by the remaining agents�

When the VisionBot achieved the goal point the path information was then sent to the

remaining agents� As the remaining agents proceeded to follow the path� the human

supervisor was able to monitor their progress via the images from the SensorBot as well

as comparing the agent�s displayed paths in the MASC interface� One interesting aspect

of these experiments was that three di�erent path following methods were employed�

one for the VisionBot� another for the manipulatory agents combined and a third for

the SensorBot� Even though these di�erent methods were employed the manipulatory

agents and the SensorBot precisely followed the VisionBot�s path� We were also able to

show that the agents were capable of executing this team behavior�

These experiments were completed in June of � In August� the multiagents

demonstrated this system at the Software Technology and Information Systems

Symposium� This demonstration was conducted with the human supervisor and the

MASC interface located at the West�elds International Conference Center in Chantilly�

Virginia and the remain agents residing at the GRASP Laboratory in Philadelphia�

Pennsylvania� The MASC interface was not originally designed or built for such long

distance interaction but for a industrial type domain where the human supervisor would

possibly be located at another location within the factory� At the demonstration site

there were many participants employing one gateway to communicate with the outside

world� This considerably slowed the communications between the site and the University�

�

Throughout the day we ran the third experiment �described above� approximately twelve

times� There were perhaps three runs in which the supervisor was too impatient with the

communications and caused the system to fail by attempting to send too many commands

at once� �This MASC interface problem will be described in more detail in Chapter Six��

In all other cases� the mobile agents were able to execute the task properly� We found the

human was able to su�ciently control the agents given the very slow communications

through the gateway coupled with the long distance communication slow down� This

was an added capability which we did not anticipate as feasible�

�� MASC System Overview

Control Buttons Virtual Camera Views

Main Interface Window Virtual Agents

Free SpaceMap

Obstacle AvoidanceState Diagram

System Camera Views

Figure �� The MASC system interface�

The Multiple Agent Supervisory Control System �MASC� is a human�machine interface�

It has been designed in such a manner that it may be applied to any number or type of

robotic agents� and is shown in Figure �� The individual robotic agents� their associated

manipulators and processes may be controlled by the supervisor through MASC� Our

�

objective is to create a semi�autonomous system which successfully completes assigned

tasks�

The human�s primary task is to �supervise� the actions of the agents during task

execution �Sheridan� �� Through MASC� the human supervises the system while

observing sensory data and images� The supervisor is permitted to assist the agents

when requested and may assume control of an agent when necessary� Each agent is

composed of multiple control and processing levels� MASC must permit the supervisor

to interact with these levels for the successful semi�autonomous execution of feasible

tasks� This interaction will permit the supervisor to revise incorrect agent decisions and

recon�gure the system after partial system failures�

�� MASC System Layout

The MASC interface provides the supervisor with a three�dimensional view of the envi�

ronment� The main working window is the large window in Figure �� The supervisor

speci�es necessary information on the model within this window were the default view

is a bird�s eye view� The supervisor may rotate� zoom or translate the view to accom�

modate his or her current requirements� as shown in Figure �� The right portion of

the interface is employed to display two images of the graphical model� as well as images

and process data �such as state diagrams�� The full interface window will permit the

display of up to eight such windows� The graphical model images are updated from two

virtual cameras located in the model� The top image is from a virtual camera located

in Figure �� outside of the door labeled as the �South Doorway�� The second virtual

camera view is from the camera located on the VisionBot in a position which coincides

with the camera located on the actual VisionBot�s turntable� The camera is positioned

on the agent labeled the �Virtual Vision Agent�� The third and sixth window displays

are images of the environment� The third is an image of the manipulatory agents taken

from one of the SensorBot�s stereo cameras� The other image is taken from the left stereo

camera on the VisionBot� The fourth and bottom window displays the state diagram

produced by the visually guided obstacle avoidance process� The �fth window displays

the free space map also created by the visually guided obstacle avoidance process� The

interface also provides a set of control buttons displayed on the top of the interface which

�

are marked �Control Buttons� in the �gure�

�� MASC System Control Buttons

The supervisor communicates with the agents through the MASC interface� We have

provided display push buttons� termed control buttons� On the interface they are dis�

played as three rows on the top of the interface as in Figure �� These buttons allow the

supervisor to specify system information� The top two rows of control buttons display

the four possible agents and their execution state� Figure �� displays the options for

Figure �� The MASC system robot control buttons�

the SensorBot and PumaBot� Initially all agent�s execution states are �inactive�� Once

the supervisor has initialized the system and chosen the agents and processes desired

for this task� the active agents will be in the �execute� state� The operator is aware of

the agent�s current state as it is marked in red �darker indentation in the above Figure��

The robot�s execution state options include pausing an agent�s current task execution�

continuing a paused task execution� halting an agent�s task execution and removing all

further commands from its command array as well as the issuance of an emergency stop

command� The active agent is depicted with its name button colored green� as the Sen�

sorBot�s button is in Figure �� in darker grey�� An agent which is active is an agent

expected to partake in the given task execution� The second set of control buttons� de�

Figure �� The MASC system mode control buttons�

picted in Figure �� permit the supervisor to alter the current system state and execute

various processes� These buttons are termed the �MASC System Modes� buttons� The

supervisor may choose from initialization� exploration� navigation or replay modes�

�� MASC System Modes

The initialization mode permits the human supervisor to specify any agents and their

associated processes which may be required for a task� Typically� the supervisor is not

permitted to enter the Exploration or Navigation modes until at least one agent has been

speci�ed� The supervisor is not required to initialize all agents which may be necessary

for a task before task execution begins� Although� if both the PumaBot and ZebraBots

are required� the supervisor must initialize them at the same time as they are controlled

by one process� Also� the initialization mode permits the supervisor to run any processes

which may be required prior to task execution� such as the planning and assignment

of the task to the active agents in the system� The Supervisor may edit the graphical

model� add objects to the world model from the overlay of particular system cameras

or display various system data on the interface� The purpose of this system mode is to

prepare the MASC system for task execution�

Phantom Agent

Virtual Agent andGoal Position

Teleoperated Path

Initial Position

Figure �� The �phantom agent� during teleoperation�

The exploration mode is a teleoperation mode� In this mode the human may teleop�

erate an agent to create locomotion commands� While working within this system mode�

there exists a �phantom� agent which is similar in principle to Bejczy et al� �phantom�

�

agent described in �Bejczy et al�� The purpose of this agent is to inform the super�

visor of the actual agent�s position� As the supervisor teleoperates the virtual agent to

create commands� its position no longer corresponds to the actual agent�s position� Thus�

the �phantom� agent� as shown in Figure �� is updated with the actual agent�s heading

and odometry readings� As can be seen in the �gure� the solid black lines and white tri�

angle �underneath the �phantom� agent� represent the creation of a move command� a

rotation command and a �nal move command� The virtual agent�s position corresponds

to the actual agent�s goal position� The �phantom� agent is shown as a white semi�

transparent replica of the virtual agent� Its position corresponds to the actual agent�s

position while executing the three commands� In this �gure� it has completed the �rst

move command and is about to begin the rotation and �nal move commands�

Initial Way Point Goal Way Point

Intermediate Way Point

Generated Path

Tables Pillar

Initial Way Point

IntermediateWay Point

Goal Way Point

GeneratedPath

Figure �� Left� The path created by the local R�geodesic path generator� �Right� Thepath created using the global path planning server�

The navigational mode is an autonomous mode� It permits the supervisor to

drive the agents based on path plans created by one of the two MASC system path

planners or the way point speci�cation method� The local path planner does not

consider environmental information and is an R�geodesic path generator described in

�Adams et al�� Wang� �� This planner is employed to plan short paths in well�

known environments� Figure �� Left� displays a path planned with this method� The

�

agent�s current position is considered the initial starting way point� There is an inter�

mediary way point along with the goal way point� The black triangles represent the way

points� The direction in which the triangles face determine the actual agent�s heading

at that way point� The solid line represents the path returned by the planner� The set

of global path planners are managed as a path plan server at Stanford University� This

server is described in �Becker� �� and is composed of the potential �elds and cell de�

composition planning methods as described in �Latombe� �� This planner considers

all environmental objects� The path planned employing the resolution potential �elds

method of the global path server is displayed in Figure �� Right�� Again� the current

position of the robot is considered the initial way point speci�cation� There exist inter�

mediary and goal way point speci�cations� represented as black triangles� The returned

path is displayed as a solid line passing through the de�ned way points� From this �gure�

it can be seen that the planning method considers the objects in the environment when

computing the path as the path avoids the tables and pillar� The third option available

in this mode is a way point speci�cation ability and is depicted in Figure �� This option

requires the operator to choose the desired way points for the agent to travel through to

its goal� The supervisor is required to choose way points �small black box in the Figure�

which appear to be safe destinations for the agents� The way point choices are then sent

to the agent�s control process which uses its own path following method to obtain the

desired way points� Environmental information is considered in as far as it is available

in or on the interface display� Unknown obstacles may lie between desired way points�

The replay mode permits the supervisor to replay the task execution within the last

�ve minutes� This mode is very helpful when a problem arises and the supervisor does

not recall what actions occurred� While this option does not permit raw image data

replay� it replays all other sensing modalities data displays� It replays a single virtual

�and �phantom� if in exploration mode� agent�s actions as well as any combination of

the active agents� The replay begins after the supervisor has speci�ed the agents and

the replay time frame� Once the replay has completed the supervisor may respecify the

agents and�or the time frame and replay again� If a particular agent was inactive during

the speci�ed time frame the supervisor is noti�ed� If only an inactive agent is chosen for

replay� no replay is provided� If other agents are also speci�ed� the human is noti�ed of

�

a particular agent�s inactivity and the replay continues with the other speci�ed agents�

This particular option is helpful for diagnosing uncertain situations�

This chapter has presented a review of our experimental test bed� the University of

Pennsylvania�s GRASP Laboratory�s multiagent project� We also presented an overview

of the multiagent�s experiments� and the MASC interface�

Chapter �

Mediation Hierarchy

The purpose of this chapter is to present a formal de�nition of the mediation hierarchy�

We begin by presenting our motivation for this theory development and then de�ne it�

�� Motivation for Development

Many human�machine interfaces interact at a high level but do not permit the human to

interact with the various low levels of the system� When creating such an interface for

a semi�autonomous robotic system� this may be a desirable feature� The robotic system

will eventually �nd itself in a situation where it is unable to correct for an error and�or

to right itself to its original goal� Interaction to all system levels will permit the human

supervisor to interact with the processes and correct such situations� This interaction

may be requested by the agents or the human may determine there is a problem and

intervene� This aspect will allow the human to correct the system and then permit it

to autonomously continue its original task� This approach is feasible for such a robotic

system as the one described in Section �� The idea is to build a more robust system

which can accommodate problems through the human�machine interface rather than

direct physical interaction with the system� Our hypothesis�

With the addition of the supervisor�s ability to interact with all levels of a

semi�autonomous system� the supervisor will be capable of correcting problem

situations and the system will successfully complete assigned tasks�

��

�� Mediation Hierarchy Description

A mediation hierarchy consisting of four levels has been formulated by Adams and Paul in

�Adams and Paul� ��a� Adams and Paul� ��b� Adams and Paul� �� These levels

dene the various intervention types into the diering levels of a robotic system and

furnishes the supervisor with these capabilities� This interaction should permit the su�

pervisor to correct situations which would cause a fully autonomous system to become

unstable and possibly fail its task execution� It is important to note the supervisor only

interacts with the agents when assistance is requested by the agents or when the super�

visor detects a situation where she or he deems it is necessary to intervene on an agent�s

behalf�

�� Level Descriptions

�� Task Level

There are numerous tasks which one would propose to assign a robotic system� One

manner by which to break up a task and assign the proper action set to each agent

is to hard code� the tasks and actions into the system� Unfortunately� this approach

inevitably limits the number of tasks the system can execute and does not create a

generalized system� In order to create a general system which executes various tasks� the

supervisor� or a task planner� must derive the proper assignments� Since the system will

not execute a task until these actions are taken� the task level resides atop our mediation

hierarchy� see Figure ��

Processing Level

Regulation Level

Task Level

Data Level

Figure �� Hierarchical levels of human interaction�

��

The task level permits the supervisor to specify the actions an agent� or a group of

agents� are to execute to complete an assigned task� Tasks may include exploration of the

environment to assist with model building� following an assigned path to a goal� observing

the task execution assigned to another agent� moving in a conguration� carrying items

such as pallets� and the navigation necessary to transport items from one location to

another�

�� Regulation Level

There exist minimal interactions which are necessary between a human�machine interface

and a robotic system� If an agent is on the verge of colliding either with another agent or

an obstacle� the supervisor should be able to prevent such a collision� If it is necessary for

one agent to complete a task before another agent begins its task� the second agent may

need to be informed to wait while the rst agent completes its execution� The supervisor

possesses a means of monitoring an agent�s actions� This may include video images�

displays of sensory data or positional updates� It is essential that the interface permits the

supervisor to choose such information for monitoring purposes� Also� in such a system�

the agent�s processes may require information from the supervisor prior to processing�

The interface must facilitate the means of providing this information� The regulation level

couples these interactions into one mediation level as displayed in Figure �� We have

developed three types of interactions on this level� control interaction� request interaction

and speci�cation interaction� which we dene below�

Specification Interaction

Request Interaction

Regulation Level

Control Interaction

Figure �� The interaction of the regulation level�

��

Control Interaction

MASC provides the supervisor with the capabilities to cope when an impending collision

must be avoided by issuing an emergency stop command via the control buttons described

in Section �� The supervisor may pause one agent�s task execution if it must wait for

another agent to rst complete its execution� These types of interactions are created via

the control interaction� Also included in this interaction type are situations when the

supervisor may teleoperate the agent to create locomotion commands� The supervisor

may employ teleoperation as an alternative to the path planning methods or to assist the

agent� The supervisor would assist the agent when the agent nds itself in a situation

where it is unable to determine its next action� Such a situation may be a dead end

passageway� The supervisor would teleoperate the agent to a location where it could

then autonomously continue its task execution� Formally� control interaction provides

the supervisor with the ability to control the agent�s progress while executing a task

either for the purpose of deterring or assisting progress�

Request Interactions

Systems contain various information which may be useful to the supervisor at dierent

times throughout the task execution� The objective is to avoid overloading the supervisor

with too much information �Sheridan� �� Whalley� �� Request interaction permits

the supervisor to request the sensory data and processed information from the agents only

when it is needed for error detection and�or monitoring purposes� When the supervisor

no longer requires this information� she or he can inform the agent�s processes to cease

transmission�

Formally� the request interaction permits the supervisor to request information di�

rectly related to the current task� This information is then employed by the supervisor

to monitor the task execution� If the supervisor believes a process is making incorrect

decisions� she or he may request more information to assist with problem detection� This

information may include images� ultrasound sensors or vehicle position� The supervisor

then reviews this information and draws conclusions as to the reason why the process

behaves as such�

��

Speci�cation Interaction

Various processes require information from the supervisor prior to the commencement of

processing� Such a process may be a path planning process for which the supervisor must

specify the starting� intermediary and goal way points of the desired path� The process

will then utilize this information and return a path for the supervisor to review� modify

and approve� The speci�cation interaction permits this form of interaction between a

process and the supervisor� Formally� speci�cation interaction provides the supervisor

with the means to interactively specify information pertinent for a process� execution�

�� Processing Level

There exist instances when a process may be incapable of determining a correct decision

based upon ambiguous information and must therefore request supervisory assistance�

There also exist situations when a process will formulate a correct decision in a local

context but the decision will not be correct in the global scheme� therefore the supervisor

should either assist with the decision making process or override the decision formulated

by the process with a correct decision�

While observing an agent�s actions based on a particular process� the supervisor may

determine the process is formulating its calculations based upon an incorrect interpreta�

tion� The supervisor may then intervene in the process to clarify the information� override

a decision or allow it to continue with its processing� The supervisor should be capable

of supplying variables� data and various processing decisions through this intervention

level to properly direct the process� For instance� assume an agent is employing visually

guided obstacle avoidance and another agent momentarily passes within its viewing eld�

in this case the visual agent would interpret the moving agent as an obstacle and begin

the obstacle avoidance task� The processing level permits the supervisor to override the

decision to avoid the obstacle� and instruct the agent to continue with its original as�

signment� Formally� the processing level permits the supervisor to aid a process when it

is unable to arrive at a decision and to rectify incorrect decisions deduced by a process

either upon the process� request for assistance or as determined by the supervisor� This

interaction level will protect the agents from entering unstable states�

��

�� Data Level

It is known that from time to time mechanical devices fail� and that the automatic recon�

gurations for such failures are not always successful� therefore� the supervisor should be

provided with the means to recongure the system� The mediation hierarchy�s data level

permits the supervisor to recongure the system when an automatic reconguration has

failed�

The outcomes determined by the higher�level processes are dependent upon correct

input data� If this data is not correct� the processes will likely formulate incorrect deci�

sions and commands which may force the agent into an unstable state� The data level

will also supply the supervisor with the ability to ensure processes receive correct data

for interpretations� For instance� as mobile agents move throughout the environment

executing an assigned task� they accumulate errors in their positional and heading read�

ings due to wheel slippage� If an automatic reset fails� it may become necessary for

the supervisor to reset the readings based upon localization information� Alternatively�

assume the focus of a camera from a camera pair has been corrupted� this may hinder

information retrieval for the process using these images� The supervisor should be able to

inform the process to stop image processing and instruct the agent to rely upon another

sensing modality to complete its task assignment�

Formally� the data level permits the supervisor to ensure correct data is passed up

through the system for interpretations and processing� It also allows the supervisor to

recongure the system during a hardware failure� This interaction type implies that

as data �ows upward through the system� the system will correctly interpret the data

implying correct actions will be executed which in turn imply the successful execution

of task assignments�

We have presented the motivation for the mediation hierarchy theory development

and formally dened the theory� providing denitions for each level�

��

Chapter �

Multiagents Process Integrations

into the Mediation Hierarchy

The chapter presents the mediation hierarchy�s implementations into the MASC system�

These implementations are presented in accordance to the level to which they apply�

�� Task Level

The main purpose of this level is to create plans which the agents employ for task ex�

ecution� This task is the human supervisor�s responsibility in the multiagents system�

Many others have implemented global planners for various applications similar to this�

thus we did not re�implement them� For instance� Ntuen et al�� in �Ntuen et al��

describe the Task Oriented Planner �TOP� which was designed for multiagent task

planning and scheduling� Ephrati and Rosenschein suggest an approach to multia�

gent planning in �Ephrati and Rosenschein� �� In �Tarn et al�� Tarn et al�

describe an event�based planner which they implemented on a PUMA �� dual�arm

system� Rocha and Ramos� in �Rocha and Ramos� �� describe the Task Planning for

Manufacturing Systems �TPMS� which is applied to Flexible Manufacturing Systems�

In �Hahndel and Levi� �� Hahndel and Levi discuss their distributed task planning

method for autonomous agents in the �exible manufacturing system domain�

��

�� Regulation Level

�� Control Interaction

At this interaction level we have provided the human with the ability to teleoperate

the vehicles using the mouse to create control commands� The human may create a

move command by clicking the mouse button and moving the virtual agent� a rotation

about the zero radius by clicking the mouse button and rotating the virtual agent or a

combination of move and rotation about various radii �similar to a draw type mode� by

clicking the mouse button and moving the virtual agent about the model�

Figure �� The error message displayed when the supervisor chooses an inactive agent�

Also� control buttons are created for all active system agents� These buttons are

described in Section �� as well as �Adams and Paul� ��a� Adams and Paul� ��b�

Adams and Paul� �� They permit the supervisor to take control of a specic agent�

If the human supervisor wishes to switch control between agents in the system he or

she simply clicks on the desired agent�s name� If this agent is inactive� in the system�

meaning it has not been initialized� the supervisor will receive an error message instruct�

ing him or her to choose one of the active agents listed in the message� as in Figure ��

If the agent is active�� the agent�s name button changes color from blue to green� The

green color indicates the supervisor�s ability to control that specic agent� The human

may then create a new path plan� teleoperate or issue a pause� halt or emergency stop

command for the specied agent�

�� Request Interaction

The MASC interface data displays permit the supervisor to request data from any of

the sensing modalities� MASC does not automatically display sensory data or attempt

to decide which data should be displayed at a particular instance� The supervisor is

��

responsible for requesting the data pertinent to the current task� The supervisor may

request all sensory data from any system mode�

The supervisor may request any of the system�s raw camera images� This information

is displayed to the right of the main window as in Figure �� The left and right images

from the VisionBot as well as the SensorBot�s left and right stereo and light�striping

images may be overlaid onto the MASC model in a manner similar to the free space

map overlay in Figure �� bottom�� These images are of particular importance when

monitoring other agent�s task executions� They also assist the supervisor in verifying

sensing modality information� such as the detection of an obstacle or passageway�

The overlays of the VisionBot�s stereo images and the SensorBot�s light�striping image

may be employed to add objects to the model� This algorithm assumes that the cameras

are in their usually dened manner and are thus facing the ground� The supervisor

may choose points which create a polygon and remove those points which are unwanted�

Once the points are chosen then the supervisor may display the polygon� If the supervisor

determines that the polygon either requires further specication or is not necessary� he

or she may remove the polygon from the model� As we do not fully calibrate the cameras

and employed only a simple calibration mechanism� this information in its own right will

be slightly inaccurate� As such� these polygons are still useful for instructing the agents

to further investigate a previously unknown object or as verication of the other sensing

modalities�

All agents record their odometry and heading readings during task execution� This

information is employed to update the position of the respective agent� It is well known

as mobile agents move throughout their environment they accumulate errors in their

odometry and heading readings due to slippage� This slippage can not be detected by

the human supervisor through MASC� For instance� if someone or something came into

the environment and picked up an agent while it was executing a command and then

placed it back on the ground at completion� the virtual agent would appear as if the

actual agent had successfully completed the command� Thus� it is necessary to employ

a localization procedure� Once the agents localize themselves using another sensing

modality� this information can be employed to update the virtual agent�s position� This

ability was not available as the localization procedure was still under development�

��

Infrared Cones

Ultrasound Cones

Front of Robot

Figure �� The display of the raw ultrasonic and infrared sensors�

The SensorBot�s raw ultrasound and infrared sensors may be displayed directly upon

the model as in Figure �� As explained in Section �� the ultrasonic readings may

be unreliable therefore� the infrared sensors are employed to verify these readings� When

there are no re�ections or the re�ection is beyond what is considered an accurate�

reading� no information is displayed for the sensor� Since there is a belt of sixteen

sensors for each modality� we display the readings from the corresponding actual sensory

position on the virtual agent� As can be viewed in Figure �� the ultrasonic readings

are portrayed as cones displayed to the actual reading distance �which have a �� degree

arc at full length of an accurate� reading�� Since one is unable to detect where along

the arch the re�ection occurs� the virtual displays are created to match the possible

re�ection area� The cones are transparent so the supervisor may view other information

in the model which coincides with the sensor displays� When utilizing infrared sensors�

one is unable to determine the distance at which a reading is re�ected� thus the infrared

sensors are either on or o� When the infrared detects a reading� the respective cones

are displayed as smaller cones to the entire predetermined distance�

The ultrasound sensing modality is also employed to detect features or objects in

the environment� When su�cient data has been collected and the ultrasound process

determines an object exist based upon the number and condence of the readings� this

information is passed to MASC� For instance� the process can detect walls and corners�

��

Detected "Coners"Detected

"Walls"

ActualWalls

Actual Corner

Detected "Corners"

Detected "Walls" Actual Corner

Figure �� top� The display of the detected features from a bird�s eye view� and�bottom� The display of the detected features from the view of the agent�

This information is displayed on the model as in Figure �� The readings are clustered

into two groups of data� tangent�segment clusters� which represent objects similar to

walls and corner clusters� which may be corners� Wall like clusters� are displayed as

wide lines at the actual sensor�s height� These clusters� are displayed in various shades

of red �shades of grey in the gure� which portray the ultrasound process� condence that

this object actually exist� The lighter the shade of red the less condence and the darker

the shade the more condence as to the object�s existence� This shading is also utilized to

represent the condence of the existence of a corner� The corners clusters� are displayed

as cylinders� also at the actual sensor�s height as in Figure �� The clusters� can then

be utilized to verify the existence of such features or objects in the model�

��

Agent’s perceived position

Agent’s localized position

Figure �� The localization information display determined by the ultrasound processand the agent�s perceived position information�

This process also possesses the ability to localize the agent with respect to the clus�

ters� it detects� The process produces a localized point and heading which is displayed

upon the model� as in Figure �� as green �in this gure black� triangles� The triangle

indicates the point�s location as well as the agent�s heading at that point with respect

to the detected cluster� information� As the agent�s heading and odometry updates are

received� this information is displayed upon the model to indicate the position at which

the agent believes it is located� This information is indicated as the light grey line in

the Figure� This permits the supervisor to visualize the dierence� Currently� there does

not exist a method of instructing the agent to physically localize itself to the correct

position� This work was still in progress by another student�

The SensorBot and VisionBot are both equipped with a stereo camera pair� As

discussed in �� this hardware may be employed for visually guided obstacle avoidance�

We have integrated a version of this process at a high level� Therefore� the human

supervisor is capable of process monitoring but is unable to interactively instruct the

process in problem situations� This process is actually composed of three processes� the

obstacle avoidance� a path follower and supervisor processes� The obstacle avoidance

detects the object and creates the commands to avoid it� The path following process

��

� monitors the given path execution either determined by teleoperation or one of the

path planning methods�

� sends the commands generated by the obstacle avoidance process or the human

supervisor to the robot control process� and

� determines how to return to the previously dened path�

The supervisor process is a Discrete�event system �DES�� supervisor whose task is to

monitor the communications between the processes�

Aside from the availability of the camera�s images� this process produces state dia�

grams �such as the one in Figure �� and a free space map� The obstacle avoidance and

path following processes supply MASC with the current processes� state information�

The state digram is updated with the current state of the processes �Figure �� displays

the state diagram for the obstacle avoidance process�� For instance� the state diagram

in the gure informs the supervisor the robot is avoiding to the right� of the obstacle�

the highlighted box in Figure �� The previous state would have informed the super�

Current Process State

Figure �� The state diagram display updated by the visually guided obstacle avoidanceprocess�

visor that an obstacle was detected� Once an obstacle is detected the process decides

from which side to avoid the obstacle based upon the amount of free space it detects

surrounding the obstacle� As the obstacle is avoided the process will update the state

diagram with states as follows� passing obstacle� then passed obstacle� and then it

will return to free space� as it no longer detects the obstacle� When the supervisor has

�As de�ned by Ramadge in �Ramadge and Wonham� �� A Discreteevent system DES� is adynamic system in which evolves in accordance with the abrupt occurrence� at possibly unknown intervals�of physical events�

��

assigned the agent a task with a specied path to follow and the agent begins to deter

from the assigned path� the supervisor must diagnoses why� The state diagram provides

one mechanism to be utilized in this diagnosis�

The free space map may be either displayed in a window to the right of the interface�

as in Figure �� top or may be overlaid onto the model as displayed in the bottom of the

gure� The overlay moves with the virtual or phantom� agent as it is updated with the

current odometry and heading readings� Occurring simultaneously� the free space map

is updated with its latest version� In both gures� the dark V� portion represents the

common eld of view for the stereo camera pair while the lighter area surrounding the

V� is unknown and uncommon to the stereo camera pair� Obstacles are represented as

white �or lighter color� areas within the common eld of view� In the gures� there are

three such areas with the remainder of the common eld of view pertaining to free space�

This process has been written to run in real�time� therefore� it does not extract exact

obstacle information� The free space map overlay provides the operator with an idea of

the obstacle�s location� but since the inverse prospective mapping projects behind the

object� it is di�cult to determine exactly where the object actually ends and free space

begins� The overlay is useful when teleoperating a vehicle since it provides the supervisor

a general� idea of how large the obstacle is�

�� Speci�cation Interaction

As mentioned in Section �� the MASC system includes two path planning mechanisms

and a way point specication method� Both planners and the way point specication

require the human supervisor to pre�specify data before a path can be planned� The

path planning mode control buttons permit the supervisor to add�� edit�� or remove�

way point specications as well as disregard� or display� the computed path� When

adding way points� the supervisor points the mouse and clicks� A triangle representing

the way point with zero heading appears� The supervisor may then rotate this point to

specify the agent�s heading angle� When adding additional way points the supervisor

points the mouse at the way point the new one should follow in the path and then moves

the mouse to the desired position of this point� If a previous way point is not selected�

no way point will be added and an error message will appear as in Figure �� When

��

Common Field of View

Detected Obstacles

Unknown Areas

Tables

VisionBot

Overlay of Free Space Map

Detected Obstacles

Figure �� The free space map as displayed in its own window �top� and as overlaidonto the model �bottom��

��

Figure �� The error message generated when improperly adding way points�

editing a way point the supervisor is permitted to only modify that point�s heading�

The point is chosen by clicking on it which then permits the supervisor to change the

heading� When removing a point from the way points list� the supervisor just clicks on

the desired point� These options are available to the supervisor during path planning

mode� so they are employed to determine the initial points as well as modify a computed

path� At any time all way points and the corresponding path �if computed� may be

disregarded� The supervisor may then start from scratch or abandon the path planning

process� The agent is instructed to execute a path �only when one exist� upon exiting

the path planning mode�

The local path planner requires the human to specify the way points and their re�

spective headings as described above� This information is sent to the planner and the

path is returned and displayed on the model� as in Figure ��Left�� Before the agent is

permitted to execute the path the human must verify it� At this point the path may be

interactively modied and a new path returned or executed by the robot�

The global path plan server requires the way point specications as described above�

This planner server must also be sent information regarding the objects in the world� the

types of robots used and their non�holonomic constraints� the desired planning algorithm

such as potential elds or cell decomposition and the required variables for the planning

algorithm such as requesting a safest or shortest paths and the path smoothness level�

The path is then returned and displayed for conrmation and then may be executed�

The way point specication option also requires the way point specications which

are described above� As there is no path planning involved the supervisor is responsible

for choosing points which should not put the agent in danger� Once the supervisor is

��

satised with the chosen way points they are sent to the agent� Each agent employs its

own on board path following method to obtain the desired way points�

�� Processing Level

Figure �� The error message generated for the local path planner singularity case�

The local path planning algorithm contains a singularity case� A normal path consist of

three segments� a beginning arc� a straight line and an ending arc� The case appears

when a way point is chosen directly behind the previous way point� In this instance� the

planner is unable to determine which direction to turn for the rst turn� either clockwise

or counter�clockwise� When this occurs� the planner sends a message to MASC which is

displayed as in Figure �� The human must acknowledge this message and then specify

the turn direction� Once the process receives this information it returns the desired path�

In this instance the process request assistance from the supervisor�

The global path plan server algorithms may be unable to determine a path if a way

point has been chosen too close to or inside an object� When this occurs� the planner

sends a message to MASC� This message is displayed on the model and the human must

then revise the selected way points�

The visually guided obstacle avoidance process sends the human information con�

cerning the current process state� The human can then instruct the system to stop its

actions� The process is unable to request assistance from the human but the human can

take control away from the process and directly control the agent� The supervisor may

tell the agent to stop and do some other task� teleoperate the agent around the obstacle�

etc�

��

Figure �� The pop up window which permits the supervisor to modify the clusteringvariables�

As mentioned in Section �� the ultrasound process displays cluster information

for tangent�segments� and corners�� Initially the variables utilized by the process to

determine these clusters are assigned default values� We have provided the ability for

the supervisor to modify these variables via the mechanism displayed in Figure �� The

segment threshold� permits the operator to lower or raise the threshold� If the super�

visor lowers the threshold more tangent�segment clusters� will be displayed� On the

other hand� if the operator raises the threshold� fewer clusters� will be displayed� The

supervisor�s ability to modify these values is particularly important for those situations

when he or she may desire more data� While there may exist erroneous data� there is a

likelihood that the supervisor will be more cautious� The corner angle� and number

of detections� thresholds are those associated with the corner clusters�� All clusters�

below the predetermined eccentricity value are not displayed� The eccentricity� value

in the Figure permits the operator raise or lower this value� This permits the display

of possibly noisier information by lowering the value and more accurate information by

raising the value�

As mentioned in Section �� this process includes a localization function which

attempts to localize the agent in conjunction with the clusters� it detects� The current

integration permits the use of localization only after two distinct walls have been detected�

��

Figure �� The pop up window which permits the supervisor to modify the distancetraveled between localizations�

Thus� the localization portion of the process is not activated until requested by the

supervisor� When localization is active� this process slows considerably� Thus� it is

optimal to only request the localization information after the agent has traveled a dened

distance� We have established the default value to be one meter� but have also provided

a mechanism to modify this value as seen in Figure �� The supervisor my reduce the

distance if more accurate information is required or increase it when localization is not

a primary concern� If the localization parameter is larger this implies localization will

occur less frequently thus speeding up the ultrasound process� execution�

This chapter presented the mediation hierarchy level implementations into the MASC

system� These implementations were presented in accordance to the level to which they

apply�

��

Chapter �

Human Factors Experimental

Design

The motivation for the mediation hierarchy�s development was to create a semi�

autonomous multiple robot system which can complete feasible tasks� Therefore� proof

of the mediation hierarchy theory entails executing various tasks until the agents require

supervisory assistance then demonstrating the supervisor�s ability to assist and correct

the problem through the MASC system interface followed by the agent�s ability to con�

tinue with the task execution to completion� We designed a human factors experiment in

an attempt to prove our hypothesis� During the experimental development� we found the

multiagents system as a whole was not su�ciently sophisticated to fully test the media�

tion hierarchy theory� Also� it was determined that di�cult experiments would require a

vast amount of training for a novice user� Thus we developed the experiment to employ

only a portion of the MASC system capabilities� The experiment was designed to pro�

vide data encompassing the subject�s perceived workload� the MASC system�s usability

and preliminary feedback on the mediation hierarchy� The purpose of this chapter is to

provide the experimental design methodology� This chapter and Chapters Six and Seven

follow the American Psychological Association�s presentation standards� �apa� ��

�� Purpose

We designed the experiment to follow human factors testing standards� We employed a

consultant to assist with the experimental design� During the design we determined if

we wished the subjects to execute di�cult tasks� the time and monetary requirements

��

would be beyond our means� This di�culty level was associated with the overall multi�

agents system design� The multiagents system is fairly complicated and would require

extensive training concerning the mechanisms and processes involved� Also� the overall

multiagents system is not sophisticated enough to execute di�cult tasks� Therefore� we

concentrated the experiments upon the subjects workload levels and system usability is�

sues� The experimental design permits some preliminary results to be drawn concerning

the mediation hierarchy�s role in the MASC interface�

The research question for this study was dened to be�

Is a novice user with proper training able to eectively interact with the sys�

tem levels �either when the system requests assistance or the user deems it

necessary� such that feasible tasks can be successfully completed in a reason�

able time frame with minimal human interaction�

Some questions we wished to answer through this study included�

�� Did the subjects workload levels increase as the number of agents increased�

�� Does workload level decrease over time and increased experience�

�� Did the time to complete assigned tasks increase as the number of agents increased�

�� Are there operator tasks which we should automate�

�� Did the human not detect problem situations which could have been averted�

�� Did the human create unnecessary interactions with the system�

�� What usability issues were detected�

�� Where the subjects able to interact eectively on the task and regulation levels of

the mediation hierarchy�

�� Tasks

The subjects were required to carry out three tasks� the single agent task� the two

agent task and the four agent task� These tasks were executed twice� sequentially during

��

each session� Subjects were permitted to employ the MASC system�s initialization and

exploration modes� This was due to the immense training required to operate the system

in the other system modes�

SensorBot

Garbage Can

Desired Path

Figure �� The single agent task

The single agent task required the subject to drive the SensorBot parallel to the

southwest wall into the corner� Once the agent obtained the position in the corner� the

subject was required to turn the agent and drive it diagonally across the room �the desired

path is the dashed line in Figure �� A tall garbage can was placed approximately two

thirds of the distance between the agent�s initial position and the corner �the garbage

can is shown as the solid rectangle in Figure �� The subjects were required to drive the

agent around the obstacle into the corner� This requirement stems from the fact that the

SensorBot is not equipped with an on�board obstacle avoidance procedure� Figure ��

displays the initial set up in the interface model and the actual environment was set up

identically�

The possible methods of executing this task� include driving up close to the obstacle�

turning away from the wall and then driving around the obstacle into the corner and

across the room� Another method involved turning the agent at its initial starting point�

such that it was on an angle to avoid the obstacle� While executing the task� subjects

required this agent�s sensing modalities to detect the object� While most subjects relied

upon the agent�s real�time images� they could also employ the raw sonar readings as well

as the ultrasound process� The raw sonar readings were useful for detecting the obstacle�s

��

position in relation to the agent� The ultrasound process was not as useful� as it requires

a large amount of data before detecting objects� Instances of conservative driving did

permit this process to provide the obstacle�s approximate location information�

SensorBot

Garbage Can

Desired Paths

VisionBot

Obstacle

Figure �� The two agent task

The two agent task required the subject to drive the SensorBot as described above

while simultaneously driving the VisionBot parallel to the Southeast wall into the corner�

The desired path for the SensorBot is the dashed line while the VisionBot�s desired path

is the dotted line in Figure �� There was an obstacle placed in the VisionBot�s path�

The VisionBot�s obstacle avoidance process was to be employed to automatically avoid

the obstacle� This obstacle and the garbage can� in the SensorBot�s path� are shown in

Figure ��

The possible methods to complete this task involve determining which agent to begin

moving� As there exist more information available than can be displayed at one time

the subjects were to choose the most relevant information for their current requirements�

The options for solving the SensorBot�s portion of the task are similar to the description

for the singe agent task�

The four agent task required the subjects to simultaneously control all four agents�

The VisionBot was positioned as in Figure �� The subjects were instructed to drive the

agent along the Southwest wall into the corner� �the dotted line in the Figure�� There

was an obstacle placed in front of the agent which the obstacle avoidance process was

to avoid� the smaller rectangle in the Figure� The other three agents were positioned as

��

SensorBot

Desired Paths

VisionBot

ObstacleZebraBot

PumaBot

Figure �� The four agent task

in Figure �� The two manipulatory agents were in a side�by�side conguration ahead

of the SensorBot� The SensorBot�s purpose was to observe the manipulatory agent�s

actions� Subjects were instructed to control the manipulatory agents in the combined

control method for as much of the task execution as was feasible� All three agents were

to be driven along the Southeast wall into the corner� �the dashed line in the Figure��

When this position was obtained� the agents were to turn and move diagonally across

the room to the goal position marked in Figure ��

The possible methods to complete this task are large� Some subjects attempted to

turn the manipulatory agents in the combined control� This could not be completed

successfully because of their positioning in relation to one another� their bumpers would

hit and thus halt their progress� Thus� each agent must be turned individually� Subjects

rst positioned and turned either the PumaBot or the ZebraBot� this was left unspecied�

The SensorBot�s specications for this task was to be maintained in a position such that

the manipulatory agents could be observed and its nal position was directly behind these

agents� Thus� the SensorBot�s position during the task was left unspecied� Subjects

could have left the SensorBot at its initial position and rotated it to obtain the desired

views or they could move the SensorBot along with the manipulatory agents�

��

�� Method

�� Subjects

The subject group was composed of thirteen computer literate members of the University

of Pennsylvania community� Subjects were novice users with mobile robots and had

various backgrounds in computer graphics� Most subjects had minimal training with a

graphical user interface� The subject�s ages ranged between seventeen and thirty�three

years and their educational backgrounds ranged from some high school to doctoral level

education� There were three female participants and ten male�

All subjects received identical training which included a system description� Training

was based only upon those system portions which the subjects would employ for these

experiments and lasted a total of thirty minutes� The subjects were paid a predetermined

amount for the entire experiment� Payment was not contingent upon completion of the

experiments or the amount of time required�

�� Apparatus

The MASC system version employed for these experiments was pared down from the

complete system� It was determined that the entire system would require a training

session signicantly longer than thirty minutes� This imposed constraints upon acquiring

subjects who would be willing to commit a vast amount of time to learning this system�

If this was an industrial experiment� in which actually users were involved� it would have

been feasible to use the entire system� The pared down version permitted the subjects

to use all four agents and their sensing modalities� The locomotion command generation

method was teleoperation and the autonomous locomotion methods were not employed�

The robotic agents employed are those described in Chapter Two� The agent�s cam�

eras were calibrated prior to the experiments and were not modied during the experi�

ments� All agent congurations were stable throughout the experimental period�

The agent�s starting positions were marked on the laboratory �oor to assure proper

placement for each trail� The curtains surrounding the laboratory�s eastern portion�

in which the experiments would occur� were closed� Also� doorways into this section

of the laboratory were blocked o� It was necessary to block the view of the area so

��

that the agent�s actions could not be observed during execution� Thus the subjects did

not perceive pressure from others observing their experiments� The closed curtains also

restricted the subject�s ability to view the area between trails�

Vision Laboratory

MultiagentsArea

SunRoom

N

S

EW

Work Area

Figure �� The generalized GRASP Laboratory�s �oor plan�

The MASC system was run on an Silicon Graphics Indigo� with �� megabits of

memory in the Vision laboratory� This is the room to the east of the main laboratory�

see gure �� for a laboratory diagram� Subjects were physically unable to view the

multiagent work area during the experiments� Also� since this is not a high tra�c area�

subjects were not observed by spectators during the experiments�

Measurement Method Type

Eectiveness Automatic interaction recording ObjectiveAutomatic robot status recording ObjectiveVideo tapes of the sessions Objective

Usability Evaluation checklist SubjectivePost�task Questionnaire SubjectivePost�experimental questionnaire Subjective

Workload NASA TLX Questionnaire SubjectivePost�experimental questionnaire Subjective

Table �� Data collection methods�

Data collection was performed employing the various methods displayed in Table ��

Automatic recording of the subjects interactions were built into the MASC system� Also�

information pertaining to the agent�s position and sensory readings were automatically

recorded� Each session was video taped for later review� Prior to beginning the ex�

periments� each subject completed a pre�experimental questionnaire �Appendix C� per�

taining to their previous experience with various systems which may be similar to the

MASC system� After each task trail each subject completed a NASA TLX workload

form �Appendix D�� After the completion of each task� subjects completed a post�task

��

questionnaire �Appendix E� which asked general questions pertaining to the system�s

ability to perform this particular task� Finally� upon completion of the entire exper�

iment� subjects completed the detailed post�experimental questionnaire �Appendix F�

pertaining to the tasks� the system�s abilities and usability issues�

The automatic interaction saves provided information such as�

� the task duration�

� how many commands were created�

� how many commands were executed�

� how many system mode changes occurred�

� what sensing modalities were employed�

� which agents were used at particular times� and

� how many and what type of errors did the subjects create�

Each of the above items was time stamped� The second form of automatically recorded

data pertained to the positional update information of the agent�s �x�y� and heading

during task execution� Also� it included information pertaining to the various sensing

modalities readings� This information was also time stamped�

A second monitor was placed to the left of the monitor on which the subjects worked�

This second monitor was used to video tape the session� A SONY XC�� Interline

Transfer Hyper HAD CCD color camera with a �� inch color sensor and an �mm focal

length lens was used to view the monitor� The information was transferred to an �mm

video recording device� All tasks and trials were recorded�

The NASA TLX workload form was employed to understand the subject�s perceived

workload level during a task� The questionnaire forms requested information which would

help us determine system usability� Each questionnaire also provided a section for the

subjects to state comments concerning the system and their experience�

I� the designer� completed Ravden and Johnson�s� �Ravden and Johnson� �� us�

ability evaluation check list� This was employed to raise usability issues and compare

��

them with those detected by the subjects� It also pertained to various system aspects of

which subjects had no knowledge�

�� Procedure

The experiment consisted of four phases� pre�experimental� training� experimental ses�

sion one and experimental session two� The pre�experimental phase required ten min�

utes� the training phase lasted thirty minutes and each experimental phase took up to

two hours� The pre�experimental� training and experimental session one phases occurred

on the rst day� The second experimental session phase occurred two days later�

The pre�experimental phase consisted of the subject reading and signing the required

consent form �Appendix B� to participate in the experiment� Then the subject completed

the pre�experimental questionnaire�

The training phase consisted of� agent training and the MASC interface training�

Each subject was taken into the experimental area where the agents would execute the

tasks� The purpose of the multiagent�s project was explained to the subjects� They were

provided with information concerning the mobile bases� their non�holonomic features

and general slippage information which occurs with these types of robots� Also� they

were instructed on the computers employed to control the agents� and how the agents

communicated with the interface� After this general introduction they were provided

information pertaining to the individual agents� This information included describing

the purpose of each agent �observation or manipulation�� their sensing capabilities� and

their abilities� Each sensing modality and associated processes were explained� Then the

subject was taken into the Vision Laboratory where they would run the MASC interface�

The interface was displayed on the screen and each step to running the interface was

described in the following order�

�� How to chose the proper model le and have it appear on the screen�

�� What the various windows were that appeared on the screen and their purpose�

�� What the various system and agent command buttons were and how they could be

used�

�� How to initialize the system by choosing the desired agents and processes�

��

�� How to display sensory information which is not provided automatically and its

uses�

�� The automatic updates of an agent�s position and heading�

�� The fact that dierences may occur between the actual agent�s position and the

agent�s virtual position�

�� How to switch between system modes�

�� How to determine which of the agents is currently within the human supervisor�s

control and all agents� execution status�

�� How to switch supervisory control between agents�

�� How to create individual locomotion commands for agents employing teleoperation�

�� How to create combined commands for the PumaBot and ZebraBot employing

teleoperation�

�� How to issue emergency stop� continue or pause commands to the agents�

�� What the phantom� agent is and its purpose�

�� How to shut the system down�

The rst experimental session consisted of a practice session� followed by the three

task executions� The practice session permitted the subjects to work with the interface�

create commands for the agents and explore the various sensing modality displays� They

were permitted to practice for twenty minutes� Subjects were not provided a script to

follow during this portion of the session� but were permitted to play� and familiarize

themselves with the system�

After the practice session� subjects were provided instructions for their rst task� The

tasks were randomized among the subjects� The rst ve subjects completed the tasks

in numerical order� single agent task� two agent task and four agent task� The remaining

subjects were given task orderings which exhausted the various permutations for the

three tasks� Originally� fteen subjects volunteered to participate in the experiment�

��

Two of the subjects did not participate� Based upon the assumption fteen subjects

would participate� the remaining ten subjects were separated into groups of two� Each

group was assigned a dierent task set randomization� For instance� the four agent task

followed by the single agent task followed by the two agent task�

Situations that would end the experiment� such as all involved agents crashing� were

explained to the subjects� They were also instructed that if something happened to one

agent to continue the experiment with the remaining agents� The task was described

in a manner which stated what they were to accomplish� that they should complete

the task as quickly and as e�ciently as possible and they could achieve the goal in

any manner� They were instructed that the environment may have changed since their

practice session� Then the subjects were instructed as to which agents would be required

for this task� which environmental model was to be used� which agent processes were

required� and which sensing processes were available as options for this task� After

receiving complete instructions� the subject then began the task� Upon task completion�

the subject completed the NASA TLX questionnaire�

The subjects were then required to perform the same task a second time� Again they

completed the NASA TLX questionnaire upon task completion� After the second task

trial� the subject completed the post�task questionnaire� The second and third tasks

were carried out in a similar manner�

The second experimental session� which occurred two days later� was similar to the

rst� The subjects were permitted a ten minute system re�orientation period� None of

the subjects used the entire ten minutes� many only desired a couple of minutes to review

the interface� The subjects then executed the two sequential trials of the three tasks in

the same order as the rst session� After each trial the subjects completed the NASA

TLX questionnaire and after each task they completed the post�task questionnaire� At

the completion of all tasks the subjects completed the post�experimental questionnaire�

Subjects were not given direction when they encountered problems or were unsure of

what their next action should be� They were permitted to continue and use the system

to determine what they should do to resolve the situation and obtain their goal�

This chapter has provided the detail design and methodology for our experiments�

��

Chapter �

Human Factors Experimental

Results

This chapter presents the human factors experimental results in accordance with the

standards established by the American Psychological Association �apa� �� The results

are sectioned in accordance to the particular data sets� Then we present results in which

various data were compared�

We calculated the various descriptive statistics for all data� We further statistically

analyzed some of the data for signicance� We created scatter plots of the data and then

applied a linear least�squares t� In most cases� a linear t proved to be better than

a quadratic or higher polynomial t� This information was then employed to compute

the various ANOVA statistics� Please note that it is infeasible to present all tables and

graphs in this chapter� thus we have provided them in the Appendices� Also� it should be

noted that on some graphs the tasks are referred to numerically� Task one corresponds

to the single agent task� task two to the two agent task and task three to the four agent

task�

�� Pre�Experimental Questionnaire

The purpose of the pre�experimental questionnaire was to obtain the subject�s back�

ground information� There were thirteen participants three female and ten male� The

average age of the participants was twenty six years� Nine participants use a computer

a majority of their day� Six participants possessed a fair amount of computer graphics

��

knowledge and only two of these six subjects expressed reasonable expertise with three�

dimension user interfaces� Only one participant reported knowledge of mobile robots

above a beginner�s level� The questionnaire and a graphical presentation of the responses

are presented in Appendix C�

�� Number of Commands

There exist many types of commands subjects may create when interacting with the sys�

tem� We have broken these commands into four groups� locomotion� system mode� agent

mode� and agent switch� Locomotion commands are the move and rotation commands

created via teleoperation� These are the commands which locomote the agent throughout

the environment� The system mode commands are the commands the subject issues to

move from one system mode to another� such as from the initialization mode to the ex�

ploration mode� The agent mode commands are the emergency stop� continue and pause

commands for the individual agents� The agent switch commands are the instances when

a subject chooses another agent to be under the human�s control�

Task Locomotion System Mode Agent Mode Agent TotalCommands Commands Commands Switches Commands

Single Agent ��

Two Agent ��

Four Agent ��

Total ��

Table �� The break down of all commands created by task for all data�

Table �� presents the number of each command created by task for all sessions and

trials� Locomotion commands composed �� of all commands� system mode com�

mands �� agent mode commands �� and agent switches �� of all commands�

Figure �� presents the average number of all commands created by task and session�

As expected there is an increase in the number of commands from the single agent task

to the four agent task� There is only a slight increase� �� in the number of commands

created for the single agent task compared to the two agent task in session one� The

number of commands created for these two tasks is essentially identical for session two�

As expected� the average number of commands signicantly increases from the single

��

1 2 3

10

20

30

40

19.1

21.5

46.1

17.3 17.2

39.7

Mea

n N

um. o

f T

otal

Com

man

ds

Tasks

Session 1

Session 2

Figure �� The means for all commands by task and session�

and two agent tasks to the four agent task� In both sessions� the mean number of com�

mands doubled for the four agent task� We computed the linear least�squared t and

the ANOVA results by pairing the total number of commands for the single agent task

with the total number of commands for the four agent task� As can be observed in

Table �� and Figure �� even though there existed a large increase in the total number

Parameter Slope PValue Value

� ��

x ��

Source of Degrees of Sum of Mean Computed PVariance Freedom Squares Square f Value

Regression � �� Error ��

Total ��

Table �� Total number of commands between the single and four agent tasks for alldata�

of commands� it was not statistically signicant� The ANOVA analysis of the data model

t found �f �� P � �� Reviewing the information in Table �� we nd

the relationship between this data for the number of commands is not signicant as the

x parameter�s P value is high and the slope of the tted line is small� It is interesting

��

15 20 25 30

30

40

50

60

70

80

All

Com

man

ds F

our

Age

nt

All Commands Single Agent

Figure �� The total number of commands for single agent task plotted against fouragent task for all data�

to note that the increase in the commands between the single and two agent tasks is

either very small or non�existent� In fact� the average number of locomotion commands

from the single agent to the two agent task falls in both sessions by approximately one

command in session one and two and a half commands in session two� Again the ANOVA

analysis shows this relationship is not signicant� This is also true of the relationship

between the two agent task and the four agent task�


� ��

x ��



Total ��

Table �� Total number of commands for the single agent task between sessions�

We also analyzed the total number of commands for the individual tasks between

sessions� We found the decrease in the number of all commands between sessions one

and two for the single agent task to be signicant� The analysis results for the single

agent task can be found in Table �� These results are signicant as both probability

��

values are low and the slope is positive� The results showed the decrease in the number

of total commands created in session two versus session one for the two agent and four

agent tasks were insignicant� The two agent task produced �f �� P � ��

and the four agent task �f �� P � ��

�� Sensing Modalities

Each subject was required to chose which sensing modalities to employ during each task�

We recorded which modalities they employed� the duration for which they were used�

and how many of the available sensing displays were employed� Information regarding

the agent�s position and heading was not included in this data as the system provides

this automatically� Each subject initialized their sensing displays prior to beginning the

actual task and none of the subjects modied their displays during a task� Table �� lists

the various sensing modalities by the percentage of time the modality was employed by

the subjects� Note that not all modalities may be employed in all tasks� The real�time

Sensor Number Times Total PercentageEmployed Occurrences

SensorBot Left Stereo Camera ��

SensorBot Right Stereo Camera ��

VisionBot Left Stereo Camera ��

Light Stripe Camera ��

VisionBot Right Stereo Camera ��

Obstacle Avoidance Free Space Map ��

Raw Ultrasound Display ��

Obstacle Avoidance State Diagram ��

Ultrasound Process ��

Ultrasound Process State Diagram � ��

Path Planning State Diagram � ��

Table �� The display methods and the percentage of time they were used�

images were the subjects predominant display choice� In fact� they were the subjects

top ve preferences� These were followed by the obstacle avoidance free space map� the

raw ultrasound and infrared displays and the obstacle avoidance state diagram� Most

subjects did not nd the information provided by the ultrasound process useful and

preferred not to use it� The path planning state diagram associated with the obstacle

��

avoidance process was the least used modality�

There exist a number of reasons we have provided the human supervisor with the

ability to turn on and o sensing displays� In particular� there exists limited real estate

for such sensing displays� The system permits the supervisor to display up to six windows

to the right of the main working window� If all sensing modalities are employed for the

single agent tasks� they require only four of these windows but the two and four agent

tasks encompass nine sensing displays� We collected information regarding how many of

the available displays the subjects employed for each task� We found subjects employed

all available displays� including those in the main working window� four out of the one�

hundred fty six trails� or �� of the time� Table �� presents the average number

of displays employed by task for all trails and sessions� The total number of available

Task Session One Session Two

Single Agent ��

Two Agent ��

Four Agent ��

Table �� The average number of displays employed by task and session for all trials

sensing modalities and displays for the single agent task was six and for the two and four

agent tasks there were eleven available sensing modalities and eight available displays�

As displayed by the table� the average number of displays employed for the single agent

task was only slightly below the total number of available displays while for the two and

four agent tasks the dierence was larger�

Figure �� displays the number of displays used for the single agent task during both

sessions� It is interesting to note that one subject attempted to execute the single agent

task with no sensing modalities� this subject also ran into the garbage can placed in the

experimental area� Also� all six displays were employed for two of the �� trials� As the

Figure shows the preferred number of displays for this task was three in session one and

four in session two�

The number of sensing displays available for the two agent task were eight� The

subject�s choices for all trials by session are displayed in Figure �� The least number

of displays employed was two and the largest number was seven� The most frequent

number of displays employed was six� This increase from the single agent task would be

��

2

4

6

8

10

1

0

6

11

3 3

2

0 0

6

8

10

2

0

Number of Displays Used

Num

ber

of S

ubje

cts

0 1 2 3 4 5 6

Session 1

Session 2

Figure �� The number of displays employed for the single agent task by session�

2

4

6

8

10

12

0 0

2

3

2

6

8

3

00 0 0

2

3

7

12

2

0


Num

ber

of S

ubje

cts

0 1 2 3 4 5 6 7 8

Session 1

Session 2

Figure �� The number of displays employed for the two agent task by session�

��

2

4

6

8

0 0

1

5

2

4

7

5

2

0 0 0

1

6

5

9

3

0


Num

ber

of S

ubje

cts

0 1 2 3 4 5 6 7 8

Session 1

Session 2

Figure �� The number of displays employed for the four agent task by session�

expected as we have introduced another agent as well as other sensing modalities into

the task�

The four agent task again increased the number of agents involved in the task but did

not introduce any new sensing modalities or displays� Figure �� presents the number of

displays the subjects employed by session for all trials during the four agent task� Again�

the lowest number of displays employed was two and in this task two trials employed all

available displays� The subject�s predominant choice was to employ six displays which

is equivalent to the two agent task�

�� Number of Errors

Every time a subject attempted to instruct the system to do something that was incorrect�

they would receive an error message and this information was automatically recorded�

These types of errors were encountered a total of thirty seven times� that rounds to

approximately three errors per subject� Only one subject received no errors� two subjects

received one error and four subjects received more than three errors�

Table �� lists the errors which occurred by their frequency� The predominant error

was attempting to instruct an agent� other than the agent currently under the supervisor�s

control� to stop� continue� pause or halt� This error accounted for over half� �� of

��

Error Occurrence

Attempting to work with un�chosen agent ��

Attempting to re�initialize an agent �

No Camera Server �

No SensorBot Robo Process �

No ZebraBot Robo Process �

No Path Following Process �

No Ultrasound Process �

Total number of errors ��

Table �� Errors subjects received by frequency of occurrence�

all errors� The second most frequent error was attempting to initialize an agent and

its processes after the subject had already completed the agent�s initialization� The

No Camera Server�� No Path Following Process� and No Ultrasound Process� errors

occurred when the subject attempted to display the processes sensory information when

the process had not been requested� The errors relating to the SensorBot and ZebraBot

occur when the subject�s attempted to use those agents and had not initialized them for

the task�

1 2 3

2.5

5

7.5

10

12.5

15

7

4

16

0

3

7

Num

ber

of E

rror

s

Tasks

Session 1

Session 2

Figure �� The number of errors by task and session for all trials�

The number of errors for each task by session for all trials is presented in Figure ��

As is presented� there were twenty seven errors in session one and only ten in session two�

��

In both sessions errors occurred most frequently during the four agent task� The overall

number of errors between the single and two agent tasks was equal� When we computed

the ANOVA�s for the number of errors during the single agent task compared to the

number of errors during the two and four agent tasks� there was no statistical signicance�

The comparison with the two agent task resulted in �f �� P � �� and for

the four agent task �f �� P � �� This was also true when we compared

the two agent task to the four agent tasks� �f �� P � ��

�� Task Completions

This section presents the results concerning the number of task completions and acci�

dents� The number of overall completions is broken into two groups� those who experi�

enced system failures and those that did not� System failure examples include instances

when an agent would not move because it believed its bumper was activated or when

an agent lost communications with the rest of the system� Subjects had been routinely

instructed that when a system failure occurred they were to continue to execute the

task with the remaining agents� All such instances resulted in what we quantify as a

completion under the circumstances��

There were one hundred thirty four successful completions� twelve system failure

completions and a total of eight accidents� These results are presented in Table �� by task

and session� This data shows that the subjects successfully completed the assigned tasks

Completions Incomplete Accident System ProblemSession Session Session Session

Task One Two One Two One Two One Two

Single Agent ��

Two Agent ��

Four Agent ��

Total ��

Table �� Break down of task completions results by task and session�

with no system problems �� of the time� The successful completion rate increases to

�� if we factor in the twelve trials in which system failures occurred and the subjects

continued on with the task� Accidents occurred in only �� of the trials� All the

��

single agent task trials were successful while two accidents occurred during the two agent

task� The two occurrences of Incomplete� for the two agent task were related to system

problems and the subject�s time constraints and thus did not complete these trials� The

highest number of accidents occurred during the four agent task� This is not surprising

as there are many more variables in this task�

The system failures which occurred were related to the following agent hardware�

serial line communications� batteries� bumpers and power cables� The serial line commu�

nications between the two manipulatory agents and the rest of the multiagents system

was the single most frequent failure occurring a total of six times� Three of the in�

stances involved only the PumaBot while the remaining three instances involved both

the PumaBot and the ZebraBots loosing communications� There was one instance when

the SensorBot�s front bumper was activated for an unknown reason and a similar an

instance with the VisionBot� There were two instances when the SensorBot�s battery

was dead and the agent was not able to function properly� Finally� there were two in�

stances when the SensorBot�s wheels became stuck on its power cable and it was unable

to move� In all instances� the subjects were able to continue on with the remaining agents

to complete the task�

�� Task Completion Times

The amount of time each subject required to complete a task was automatically recorded

in seconds� Figure �� presents the average completion times by task and session� As

expected� the gure shows the time to complete a task increased as the number of agents

involved increased� Also� there was a decrease in the means from session one to session

two for all tasks� The single agent task completion times dropped by �� seconds� the

two agent task times by �� seconds and the four agent task times dropped by ��

seconds between session�

We analyzed the completion time dierences between tasks� We found the variances

were signicant between the single and the two agent tasks� �f �� P � ��

but the relationship between the two and four agent tasks was not as signicant�

�f �� P � �� The relationship was insignicant between the single and

��

1 2 3

100

200

300

400

500

600

368.8

411.6

651.7

272.6

343.1

458.4

Mea

n C

ompl

etio

n T

imes

Tasks

Session 1

Session 2

Figure �� The mean completion times by task and session in seconds�

four agent tasks� �f �� P � as can be observed by reviewing table ��

The variance between the single versus four agent tasks is insigni�cant due to the large

probability value of the constant�


� ��

x ��


Regression � �� Error � ��

Total ��

Table �� Task completion times between the single and four agent tasks for all sessionsand trials�

We further analyzed the task completion time data between sessions for each task�

The results showed that there existed signi�cant result between sessions for the single

agent task� �f �� P � �� These results are presented in Figure �� which

shows the slope of the x parameter is positive� The constant probability value was ��

and the the linear parameter probability was �� The results were insigni�cant for a

similar comparison between sessions for either the two agent task� �f �� P �

��

300 350 400 450 500 550

225

250

275

300

325

350

375

Com

plet

ion

Tim

es S

essi

on T

wo

Completion Times Session One

Figure �� The completion times for the single agent task between both sessions for alltrials�

�� or the four agent task� �f � �� P � ��

�� Perceived Workload Measures

We recorded the subject�s perceived workload employing the NASA TLX method� At the

completion of each trial the subject completed a NASA TLX questionnaire� presented

in Appendix D� This information was combined with the subject�s responses to the

weighting measure pairs from the post�experimental questionnaire in Appendix F� The

perceived workload values were calculated as the method prescribes� The resulting values

range from zero to one hundred� The lowest perceived workload value was �� and the

highest was �� Figure �� displays the mean perceived workload values by task and

session� As can be observed� the means increased between the single and two agent tasks

as well as the two and four agent tasks for session one� During session two there was

only a slight increase in the value between the single and two agent tasks� Again there

existed an increase for the four agent task during session two� There was a decrease in

the mean values for all tasks between sessions one and two�

We further analyzed the data for signi�cance� We computed the ANOVAs between

tasks and found the comparisons to be signi�cant� The comparison between the single

agent and the two agent task found the constant probability to be �� and the linear

parameter to be with a positive slope value� The comparison of the two and four agent

��

1 2 3

10

20

30

40 36.75

40.81

48.15

32.1 32.83

43.09

Mea

n Pe

rcei

ved

Wor

kloa

d

Tasks

Session 1

Session 2

Figure �� The perceived workload means by task and session for all trials�

tasks found the constant parameter P � �� and the x parameter P � with a

positive slope value� Table �� presents the results for the comparison between the single

and four agent tasks� Again the probability value was very close to zero with a positive

slope value�


� ��

x ��


Regression � �� Error � ��

Total ��

Table �� Perceived workload measures between the single and four agent tasks for alldata�

We also analyzed the data between sessions for a single task� The results showed that

the di�erences in perceived workload for each task between sessions was not signi�cant�

The single agent task resulted in an a constant probability of �� f �� P �

�� Figure �� displays this result in graphical form� The result for the two agent

task computed the constant P � �� f�� P � �� The four agent

task results showed the constant P � �� f�� P � � As the constant

�

30 40 50 60 70

10

20

30

40

50

60

Perc

eive

d W

orkl

oad

Sess

ion

Tw

o

Perceived Workload Session One

Figure �� The perceived workload measures for the single agent task between bothsessions for all trials�

parameter probability values are insigni�cant� there exist no signi�cant relationship�

�� Multiple Data Comparisons

The previous sections presented the results for the individual data collection items� This

section will present the data analysis results between some of the data item groups� The

purpose of this analysis was to determine the signi�cance of these relationships�

�� Perceived Workload Measures

We were interested in determining what factors contributed to the subject�s perceived

workload measures� Thus we analyzed the workload data in comparison with the task

completion times� the total number of commands and the number of errors subjects

created�

Perceived Workload Measures Versus the Total Number of Commands

We began by exploring the general relationship between the total number of commands

created for all task compared with the perceived workload values for all tasks� This

analysis determined there existed a signi�cant relationship between the two data sets

which is presented in Table �� As this Table displays� the computed linear parameter

probability value is �� and the constant value is � This combined with the positive

��


� ��

x � � ��



Total ��

Table �� Total number of commands versus perceived workload measures for all data�

slope value indicate a signi�cant relationship� Based on this result we further analyzed

the data for all tasks between sessions�

20 40 60 80

10

20

30

40

50

60

Perc

eive

d W

orkl

oad

Total Number of Commands

Figure �� The perceived workload measures versus the total number of commands forall data during session one�

Figure �� presents the results for the comparison of the total number of commands

versus perceived workload data for all tasks during session one� The relationship was

found to be signi�cant as the constant P value was and the x parameter�s probability

value was �� with a positive slope value� Table �� presents the results of the same

comparison for session two� As the table displays� the computed linear coe�cient�s

P � �� while the constant P was with a positive slope value� Since these relationships

were found to be signi�cant we further analyzed the data by the individual tasks�

��


� ��

x ��



Total ��

Table �� Total number of commands versus perceived workload measures for sessiontwo�

The analysis of the individual tasks employed the data for all trials and sessions of

the particular task� In general� we found none of these relationships to be signi�cant�

The analysis for the single agent task found �f �� P � �� The analysis of

the two agent task resulted in �f �� P � �� Finally� the analysis of the

four agent task found �f �� P � � ��

Perceived Workload Measures Versus the Number of Errors


� ��

x ��



Total ��

Table �� Total number of errors versus perceived workload measures for all data�

The relationship analysis between the perceived workload measure and the number of

errors the subjects created was necessary as one would believe there should be a signi��

cant relationship� Table �� presents the results of this analysis� As the table displays�

there was no signi�cant relationship between the two data sets as the linear coe�cient�s

probability was �� As there existed no signi�cant relationship we did not further pursue

this relationship�

��


� ��

x � ��



Total ��

Table �� Task completion times versus perceived workload measures for all data�

Perceived Workload Measures Versus the Task Completion Times

Finally we explored the relationship between the perceived workload measures and the

task completion times� We found that in general when comparing the data sets for

all data there existed a relationship which was signi�cant� as displayed in Table ��

Comparing the probability values and the slope value we found the P values low while

the slope is rather small but still positive�

We continued to analyze the data by session as we found there was a signi�cant

relationship in the general analysis� The relationship for the data in the �rst session

revealed a constant P value of and a linear parameter P value of �� with a small but

positive slope value� The same analysis performed on the data from the second session

found an insigni�cant relationship as the constant P � ��

The analysis of the individual tasks found the relationship for all data in all tasks

was insigni�cant� The results for the single agent case found �f �� P � ��

for the two agent task the constant P � �� and for the four agent task� �f ��

�� P � �� The results for the two agent task are presented in Figure ��

Based upon the combination of these results we can state in general the subject�s

perceived workload is e�ected by the total number of commands created for a task and

by the task completion time� It does not appear to be adversely e�ected by the number

of errors a subject committed�

��

400 500 600

10

20

30

40

50

60

Completion Times

Perc

eive

d W

orkl

oad

Figure �� The perceived workload versus completion time for the two agent task forall data�

�� Number of Commands

Next we explored the relationship between the total number of commands with the total

completion time� and number of errors� As discussed in Section �� we analyzed the

relationship between the command data and the perceived workload�

Number of Commands Versus the Number of Errors

We analyzed the relationship between the number of commands created versus the num�

ber of errors which occurred� The model results were signi�cant with constant P �

and the linear term�s P � �� with a positive slope� This relationship is displayed in

Figure ��

Number of Commands Versus the Completion Times

It was expected that we should see a signi�cant relationship between the total number

of commands created and the task completion times� As number of commands increases

one would expect to see an increase in the completion times�

We analyzed the data for all tasks and session and found in general the linear term

probability value was signi�cant with a positive slope but that the constant probability

value was not signi�cant as shown in Table ��

��

0.5 1 1.5 2 2.5

20

40

60

80

Perc

eive

d W

orkl

oad

Total Number of Errors

Figure �� The number of errors versus the number of commands for all data�


� �� x ��



Total ��

Table �� Task completion times versus total number of commands for all data�

We then analyzed the data between sessions� The data from session one displayed a

relationship which is insigni�cant with a constant P � �� The constant probability

value for the second session was �� which is also considered to be insigni�cant�

The data analysis between sessions for the individual tasks displayed signi�cant re�

sults for the single and four agent tasks� The relationship for the single agent task found

the constant P value to be �� and the linear term�s probability to be �� with

a positive slope value� The results for the analysis of the two agent task were found

to be insigni�cant as the constant probability value was �� Figure �� displays the

relationship between sessions for the four agent task� The constant probability value was

�� while the probability for the x term was with a positive slope value�

��

200 400 600 800 1000 1200

30

40

50

60

70

80

Completion Times

Tot

al N

umbe

r of

Com

man

ds

Figure �� The total number of commands versus completion times for the four agenttask between sessions�

Number of Commands Versus the Perceived Workload

The analysis results of the number of commands versus perceived workload relationship

were discussed in Section �� therefore we will not repeat them in detail� It was found

this relationship was signi�cant in the general comparison as well as the between session

and tasks�

�� Number of Errors

The relationship between the number of errors and perceived workload was found to

be signi�cant and was discussed in Section �� A similar comparison between the

number of errors and the total number of commands was found to be signi�cant and

was discussed in Section �� The remaining relationship to report is the relationship

between the number of errors and the task completion times�

Number of Errors Versus Completion Times

The relationship between the number of errors and the completion times for a task was

generally found to be signi�cant� Table �� presents the analysis results� As can be

observed from the table� both probability values were signi�cant and the slope was also

signi�cantly positive�

As the results show� there was a signi�cant relationship between the number of errors

��


� �� x ��



Total ��

Table �� Total number of errors versus task completion times for all data�

created and both the task completion time and the total number of commands created for

a command� The relationship with the perceived workload was found to be insigni�cant�

�� Completion Times

The analysis results comparing task completion times have been previously discussed in

this Chapter� The comparison with perceived workload was discussed in Section ��

with the total number of commands in Section �� and with the number of errors

created in Section ��

The general relationship between completion times and perceived workload was found

to be signi�cant but the analysis results between sessions and tasks was mixed� The

relationship analysis with the number of commands revealed no signi�cant relationship

in the general case as well as between sessions and was mixed between tasks� The

relationship between this data and the number of errors created was also found to be

signi�cant�

�� Post�task Questionnaire

As subjects completed the two trials for a task during both experimental sessions� they

then completed the post�task questionnaire �see Appendix F� This questionnaire asked

speci�c questions about the task�s di�culty and the system�s ability to perform this task�

The subject�s responded to questions by circling a number from to � on a Likert scale�

The �rst question inquired as to the task di�culty� It was found that during the �rst

session of the single agent task six subjects found the task fairly easy while during the

��

second session eight subjects found the task �easy�� No subjects found this task to be

in the di�cult to impossible range� The same question for the two agent task produce a

majority of �easy� results during both sessions and none of the subjects found the task

�di�cult�� The subjects� responses for the same question after the four agent task for

session one found that most subjects did not �nd the task easy� This result �ipped when

asked the question again during the second session� with the majority �nding it much

easier�

When the subjects were asked to rate the task as �confusing� or �clear�� roughly � �

of the subjects found all three tasks to be clear during both sessions� Only one subject

found the four agent task somewhat �confusing� during the �rst session�

The subjects were asked to rate the task�s stimulation factor� In general� the subject�s

replies �t into all spectrums of the scale for the single and two agent tasks during both

sessions� The subjects found the four agent task to fall more towards �stimulating�

than �dull�� Many subjects commented that during the second session they were bored

with executing the same tasks and would have preferred new tasks which were more

challenging�

The subjects gave varying results when asked to rate the task on a scale from �frus�

trating� to �satisfying�� At least one subject found each task somewhat frustrating

during both sessions� The subjects who found the task somewhat satisfying during the

�rst sessions tended to increase their satisfaction level during the second session�

The second question queried the subjects on their level of system control during a

task� The scale for this question was between �rarely� and �always�� During the single

agent task of the �rst session most subjects felt they had considerable control over the

system� This feeling increased during the second session for the same task� The results

for the two agent task during the �rst session showed that most subjects did not feel they

were always in control of the system and one subject appeared to only feel in control

about �� of the time� This again dramatically increased for the second session� While

the subject�s responses remained on the scale�s high end for the four agent task during

the �rst session only two subjects felt as though they �always� had control of the system�

These responses increased for the second session� These results were likely due to the

subjects� increased familiarity and practice with the system�

��

When the subjects were asked about their ability to understand and interpret the

data readings during a task most replied they �always� understood during the single

agent task for both sessions� During both sessions of the two agent task this number

slightly decreased but all subjects replied in the scale�s upper portion� This would be

expected as the two agent task introduces more information into the equation� The

responses to the same question after completion of the four agent task proved similar

to those of the two agent task� This particular result is not surprising as the types and

number of sensing displays available between the two and four agent task is stagnant�

Subjects who received errors felt they were �always� or almost �always� capable of

correcting their errors during all tasks for all sessions� Only one subject in both sessions

of the four agent task did not feel this was true� This is a result which shows the interface

provides the user with the information and capabilities to correct errors�

When asked if they felt as though they were �always� or �rarely� capable of complet�

ing a task� only one subject during the second session of the four agent task responded

in the scale�s bottom half� All other subjects felt they were able to complete the tasks�

The subjects were asked if they felt in control of the individual agents during the

task� As would be expected� the single agent task resulted in the highest number of

�always� replies� This task requires only one agent� therefore the subject does not have

the opportunity to become distracted while working with another agent� This result was

true during both sessions of this task� The responses for the two agent task showed that

during the �rst session the number of replies stating �always� was nearly half that of

the single agent task� The responses increased during the second session but still were

not as high as the single agent task� In general� most subjects felt they maintained a

reasonable amount of control over the agents during the two agent task� This was also

true in the four agent task� The number of subjects that �always� felt in control of the

individual agents was nearly half that of the single agent task and slightly lower than

the two agent task�

When asked if the system�s power to complete this task was �inadequate� or �ade�

quate� the general response was that it was mostly �adequate� for the single and two

agent tasks� These responses increased between sessions one and two� A fair number felt

the system maintained adequate power during the four agent task but this result was

��

lower than that of the �rst two tasks�

The last question asked if the system was �exible enough to perform this task� The

results were quite mixed for all tasks� While the responses remained in the scale�s upper

portion� they were scattered for the single agent task during the �rst session� The second

session resulted in a higher opinion of the system�s �exibility for this task� As the number

of agents increased� the subject�s opinions of the system�s �exibility decreased but tended

to increase slightly again during the second session�

This questionnaire also provide a space for the subjects to provide comments about

this particular task and the system� Their comments will be discussed in Section ��

along with their comments from the post�experimental questionnaire�

�� Post�Experimental Questionnaire

The post�experimental questionnaire was presented to the subjects at the completion

of all tasks at the close of the second session� Its purpose was to obtain their general

opinions of the system and to extract system usability issues� The questionnaire and a

graphical presentation of the results is presented in Appendix F� Each question required

the subject to circle a response between zero and nine on the Likert scale� This section

will discuss the results for this questionnaire�

The �rst question asked for an overall reaction to the system based on six scales� The

�rst nine value scale rated the system from �terrible� to �wonderful�� The majority of

the responses� seven� rated the system a seven while the remaining six responses rated it

eight and nine� Although the system is not �wonderful�� it still was rated in the scale�s

upper half� The second scale rated the system from �frustrating� to �satisfying�� Two

subjects found the system somewhat frustrating� two more subjects fell into the middle

of the scale and the remaining subjects rated it in the scale�s higher third� In general�

this tells us that the system tends not to frustrate the users� The third scale rated the

system�s stimulation and dullness factors� All but one subject rated the system in the

scale�s upper third� with seven responding with ratings of eight or nine� The responses

on the scale�s lower portion may be due to the task repetition� as some subjects found

this repetition boring� When asked if the system was �di�cult� to �easy� nine subjects

��

responded in the scale�s upper third� One subject felt the system rated a three� The

�fth scale rated the system�s overall power as �inadequate� or �adequate�� All but one

response was in the scale�s upper third� The �nal scale rated the system�s rigidity and

�exibility� Eleven subjects responded in the scale�s upper third for �exibility while two

subjects felt the system rated a �ve� The overall reaction to the system was generally

favorable�

The second set of questions related to the system�s screen or window layout� The

�rst question asked if the current layout was �helpful�� Five subjects responded that it

was while the remaining replies were scattered about the scale� This may be due to the

user�s inability to reorganize the small windows to the left of the main window� Nine

subjects found the one main working window �helpful� while the remaining four subjects

responded in the scale�s upper portions� Subjects were queried as to the frustration level

of turning on and o� the data displays� Six subjects found this a �satisfying� feature

while one found it �frustrating�� The remaining six responses were in the scale�s upper

half� When asked if they found this ability �easy� or �di�cult� eight subjects felt it was

�easy�� three subjects rated it somewhat easy and one subject rated it a �ve� When

queried if this ability was �rigid� or ��exible�� eight rated it in the scale�s upper third

as �exible while the remaining �ve rated it in the middle third�

Subjects were asked if they felt that the use of command buttons was �logical� or

�illogical�� Seven responded that it was �logical� with a total of twelve responses in the

scale�s upper third� The �nal response rated it a six� When asked if they found the

command buttons �frustrating� or �satisfying�� eight subjects responded in the scale�s

upper third while the remaining �ve fell into the scale�s middle third� Nine subjects found

the use of the command buttons �easy� while the remaining responses were scattered in

the scale�s upper half�

The subjects were questioned as to the �phantom� agent�s e�ectiveness during tele�

operation� Nine subjects found its use �helpful� while the remaining four subjects found

it moderately helpful� Nine subjects found its use �logical� while all responses remained

in the scale�s upper third� No subjects found the �phantom� agent�s use �frustrating�

but one subject rated it a �ve� Eleven subjects rated its use in the upper third of the

�satisfying� scale and one subject had no opinion�

��

All subjects found the amount of information which could be displayed on the screen

in the scale�s upper third from �inadequate� to �adequate�� Eight subjects rated it

�adequate�� The subjects had mixed feelings as to the logic of the screen information

arrangement� Four subjects stated it was �logical� while four more rated it in the scale�s

upper third� Four subjects revealed mixed feelings and one subject felt it was much more

�illogical� than �logical��

The third question set was related to the terminology and messages throughout the

system� Ten subjects rated the terminology usage as �consistent� while the remaining

three rated it in the scale�s upper third� The same results were received when the subjects

were ask about the consistency of the command button labels throughout the system�

When asked their opinions of the clearness associated with the command button labels�

eight subjects felt they were �clear�� Four other subjects rated it in the scale�s upper

third and one subject responded with a �ve� In general� the results to the three questions

show that the system�s messages and terminology is consistent� which is very important

in the user�s understanding and ability to use the system�

The remaining questions in the questionnaire�s third section are related to the error

messages subjects received� Consistently throughout these questions� four or �ve subjects

had no opinion�

When the subjects were asked about the appearance of error messages on the screen

the nine subjects with opinions rated it in the scale�s upper third for consistency� The

position at which the messages appear on the screen were rated as �consistent�� Subjects

generally found the error message content to be �clear�� with eight responses between

the scale values of eight and nine� One subject rated the content a six� The ten responses

to the error messages helpfulness were in the scale�s upper third� with seven responses

of �helpful� and two responses of the scale value eight� In general� the nine respon�

dents found the error messages �easy� to read� Subjects felt error messages clari�ed

the problem frequently with all responses falling in the the scale�s upper third� only two

felt it was �always� helpful� Most subjects felt the phrasing of error messages leaned

toward �pleasant�� while one subject rated the phraseology a �ve� The error message

phrasing was rated somewhat �clear� by the respondents and the messages helpfulness

was found to be more �helpful� than not� Only six subjects expressed an opinion to the

��

�nal question in this section� The question asked if the instructions for correcting errors

was �confusing� or �clear�� Only two subjects felt the instructions were �clear� and the

remaining four fell towards the middle of the scale�

The questionnaire�s fourth section related to the subject�s ability to learn and remem�

ber the system�s functioning� Six subjects felt the system was easy to learn while the

remaining seven felt it was moderately easy� Two subjects felt getting started using the

system was slightly di�cult while the remaining subjects rated it as fairly easy� Eleven

subjects felt the time required to learn to operate this system was quite fast while the

remaining two felt it was fairly fast� Eleven subjects believed remembering the com�

mand buttons uses and names was �easy� while the remaining two subjects found it a

little more di�cult� Only six subjects found remembering speci�c rules about entering

commands �easy�� The remaining replies felt this was somewhat to fairly easy� Seven

subjects felt tasks could �always� be performed in a straight forward manner while the

remaining six subjects rated two a piece from six to eight on the scale� When the sub�

jects were queried as to the number of steps required per task� we received mixed results�

This is likely due to the fact that many subjects believed the system initialization should

occur automatically� All subjects believed that the steps to complete a task followed a

logical sequence of events from almost �always� to �always��

The questionnaire�s �fth section pertained to the system�s overall capabilities� None

of the subjects felt the general system speed was fast enough� In fact� seven subjects

felt the system was particularly slow� Eight subjects felt the response time for most

operations was fairly slow� while �ve subjects felt it was almost fast enough� The rate

that information was displayed received mixed replies� Most subjects� eleven� felt it

should occur faster� Five subjects felt system failures occurred fairly seldom while the

remaining subjects felt the failures occurred fairly frequently� two responses� to somewhat

seldom� Finally� subjects were asked if they felt novices could accomplish tasks after

proper training� Seven responded they felt novices could accomplish this �easily�� while

the remaining six subjects ranged from somewhat easy to fairly easily�

Section six of this questionnaire asked the subjects to circle a member in a pair of

the NASA TLX workload variables� This information is used to weight the information

subjects provided on the NASA TLX questionnaires�

��

Finally� the subjects were provided a section in which they were permitted to write

their own comments regarding what they liked or disliked about the system� These

responses will be discussed in the next sub�section�

�� Subjects Written Comments

In general� the subjects provided numerous comments� Some were as general as �This

system is fun� to very detail oriented system related issues� All system oriented comments

were constructive and would improve the system�s usability� This section�s purpose is

to provide the reader with their comments� These comments were provided on both the

post�task and post�experimental questionnaires� As many of them overlap between the

two questionnaires� we are presenting them together in this section�

The subjects commented it was di�cult to know the clearance a robot had in situa�

tions when it was close to a corner or object� The images did not provide enough sense

of the agent�s true position and that some form of localization was necessary�

The �blindness� of the agents was noted as a negative� One subject suggested the

addition of cameras on the agent�s side� One subject states�

Again� blindness was a problem� This time resulting in the failure of the task�

Another subject suggested providing the ability to pan� tilt and zoom the agent�s cam�

eras� In particular� this subject felt it would be useful when attempting to monitor the

manipulatory agent�s actions� Some subjects stated they would prefer a lower�quality

image at a higher frame rate to the current images� This was particularly true when the

system�s communications were very slow�

One subject felt using the agent command buttons was di�cult when switching be�

tween agents� This subject suggested that it may be better to permit the operator to

choose the agent by clicking on the desired virtual agent� Some subjects found the sys�

tem mode control buttons confusing because they could not �visualize� where they were

located in the control button tree� They were unsure which mode they were in and what

they should do next� Some subjects disliked the requirement to specify that they wanted

to create commands for both manipulatory agents or for a single agent� They found this

time consuming and cumbersome�

��

Some subjects found it di�cult to use the mouse to rotate the agent� In particular� if

they placed the mouse on top of the agent� they did not have �ne control of the rotation

movement�

One subject stated that since the VisionBot was able to avoid obstacles with its

on�board process� he basically ignored it after creating the necessary commands for its

portion of the task� This is a problem as the agent may run into di�culties� if the human

is not monitoring problems may arise� One subject commented on the fragility of the

VisionBot�s obstacle avoidance process� He stated�

The VisionBot goes crazy when it sees the stain on the carpet in the corner

and the user has no control over it�

This is a true statement� The process is very sensitive and is not fully integrated into

the system� thus the human cannot completely control it in such situations� Another

subject commented in relation to the two agent task�

The only problem with this task was that the VisionBot always ends up

turning in circles�

This again refers to the process� sensitivity�

One subject felt the raw sonar displays complicated the agent�s navigation� Thus

he did not use them� Another subject only used the raw sonar displays �to watch the

changing colors on the screen��

The initialization of the agents requires too much time and is very repetitive in the

current system because the human must initialize the system every time they start a

task� Many suggested allowing the system to automatically initialize the agents either

via a preference �le choice or just those processes� such as the robo process� which are

always required by the agents� This second suggestion is not feasible with the manner

in which the multiagents system�s communications have been established� one is unable

to request processes as desired� This issue will be discussed further in the discussion

section�

Double�clicking on buttons generally caused a system failure and is a system bug

that appeared during the experiments� This problem was particularly di�cult for those

��

subjects who were accustomed to working with systems which require double clicking�

such as a Macintosh system� In the particular case of requesting windows to the right

of the interface� if the subject double clicked� two windows would appear� One window

would display the image� the other would remain black and there was no mechanism to

remove the useless window� Thus it would limit the user�s ability to display information�

There were also times when the subjects were unsure if they had clicked a button� as

they could not observe that the system was taking a long time to process their request�

Two suggestions where� change the button color when it is selected or display a message

that the system received the command and is processing it�

One subject thought it would be preferable to be able to control the agent�s speed�

This would permit them to increase the agent�s speed in situations where there was little

concern of an accident and reduce the speed in populated environments�

Many comments were received on the small images to the right of the main window�

Many subjects suggested that each image displayed have a title bar associated with it

so one would know which agent�s image they were observing� Another dislike was the

inability to reorganize the windows such that image pairs could be arranged beside one

another� In particular� subjects wanted to do this with the stereo camera image pairs�

Subjects also thought that the ability to turn on and o� the virtual camera images would

be useful as it would provide two more display areas� One subject thought the smaller

displays should be placed across the top of the main �oor plan as it seemed to him that

this would provide a more natural gaze from one to the other� thus eliminating the left

to right gazing which this subject found distracting�

Some subjects did not feel the system provided enough sense of �presence� within

the environment� Another subject stated they became impatient with the system as they

gained familiarity with it because it was too slow�

The subjects liked many things about the interface� In particular� there were nu�

merous favorable comments on the system�s graphics and the �phantom� agent concept�

While some subjects had di�culty using the mouse to create commands for the agents�

others found it easy� Subjects found the interface to be �very user�friendly�� One subject

stated the system was �complete� and �user�friendly�� Another felt�

��

It is quite easy for someone to learn how to use this interface� even someone

with little or no computer experience� It�s also visually attractive and logical�

Still another subject stated the system was easy to learn and it was �very easy� to control

all four agents during the four agent task�

�� Ravden and Johnsons Evaluation Check list

Ravden and Johnson�s �Ravden and Johnson� �� evaluation check list was created to

provide a practical tool with which to rate a system�s usability� It is based upon a set of

software ergonomics criteria which a well�designed user interface should encompass� As

we wished to apply this check list to the entire system and use it as a reference point to

the responses the subjects provided� we completed the check list ourselves� As has been

previously stated� the interface version employed for the experiments was only a small

portion of the entire interface�

The check list is composed of nine sections all of which relate to a separate usabil�

ity criterion� These nine sections permit the respondent to choose from the following

responses� Always� Most of the time� Some of the time� and Never� At the end of each

section the respondent gives an overall rating on this particular topic� Section ten ex�

plores system usability problems with responses of� No problems� Minor Problems and

Major Problems� The last sections provide general questions with an open response� We

completed this check list after the experimental sessions but prior to analyzing the data�

�� Visual Clarity

This section is related to the system�s visual clarity� in particular� the clarity� organiza�

tion� unambiguousness and ability to read information on the screen� As this is a general

check list� we must regard the questions as they apply to our interface� All questions are

related to the screen and window displays� We found from completing this list�

� The interface does not clearly identify windows with an informative title�

� Most of the time important information is highlighted on the screen�

� It is clear where the user should enter information and commands for the system

as well as the required format�

��

� Information usually appears to be logically organized on the screen�

� All the various types of information are clearly separated on the screen�

� Large amounts of information are always properly separated from other informa�

tion�

� Columns of information� such as the smaller windows� are always properly aligned�

� Colors are properly displayed and help to make the display clearer�

� Most display aspects would be easy to view if a low resolution screen was used and

if a user is color blind� One subject was color blind and performed as well as the

others�

� Information on the screen is always easy to see and read�

� The screen usually appears uncluttered�

� Pictorial displays are properly drawn but are not annotated�

� It is usually easy to �nd required information�

The overall system rating in terms of visual clarity was �moderately satisfactory�� This

rating was chosen because the images displayed from the agents are not labeled� thus

a user may become confused� Also� if the system response time is slow� currently one

may not know if they properly pressed a command buttons� A mechanism is required to

indicate the system is busy�

�� Consistency

This section explores the manner in which the system looks and works at all times for

consistency� The results of this section were�

� Colors are always used consistently throughout the system�

� Abbreviations� acronyms� etc� are always used consistently�

� Pictorial and graphical information are always used consistently�

��

� The same types of information� �error messages are always displayed in the same

location and in the same layout�

� Information which appears in numerous places� such as the agent data display

buttons� are always displayed in the same format�

� The input information format and method is always consistent throughout the

system�

� The action required to move the cursor �mouse around the screen is consistent�

� The method of selecting options throughout the system is consistent�

� There exist standard procedures for performing similar and related activities� An

example is the method of choosing way points for the various planning methods�

� The manner in which the system responds to a user action is somewhat consistent

through out the system� There exist cases where a command button may be chosen

that does not cause the command line menu to change�

We gave the system a �very satisfactory� rating in this category� The last item mentioned

is a minor problem with consistency and does not appear frequently�

�� Compatibility

This section requires one to examine the system�s compatibility with the appearance and

functionality of other systems a user may use regularly� As we do not know of another

interface of this exact nature� we have based our responses on general computer systems�

We found�

� Colors assigned to graphical objects were as close as possible to the real object�s

colors� Most of the time colors associated with actions were the conventional asso�

ciations� The active agent�s color in the agent control buttons is green but it does

not change to red when the operator issues an emergency stop command�

� Graphical representations are usually easy to recognize but images from the agents

may not be easy to recognize because they are not labeled�

��

� There exist established conventions for the information displays format�

� Presented information is always in units which the user would be familiar with such

as millimeters�

� When the user inputs information using the mouse� the movement on the screen

usually corresponds to the mouse�s movement� There are instances when rotating

agents with the mouse positioned directly above the agent that the agent will move

much farther than intended�

� Most system information is presented in a form which �ts the user�s task view�

� Most of the time the graphical display is compatible with the user�s view of what

they are representing� Discrepancies can occur due to slippage of the agent�s wheels�

� The sequence of events to complete a task usually follow what the user would

anticipate� Working with the manipulatory agents is slightly di�erent� as they may

be controlled simultaneously�

� In general the system works as the user would expect�

The system rated �Moderately satisfactory� in terms of compatibility� It is possible the

system does not �t the user�s perception of the task as every user di�ers� It is also possible

users will become very confused when �rst attempting to control the manipulatory agents

in the simultaneous control� While the command generation method for these agents is

the same as for a single agent� there are many more factors to consider� such as� if I turn

the agents and they are too close together then they will hit each other thus ending the

task�

�� Informative Feedback

This section examines a system�s ability to provide the user with clear and informative

information concerning their location within the system� what actions they have already

completed and the action�s success and failure and what actions should be taken next�

Our system does not attempt to instruct the user as to which actions should be taken

next but the other aspects are relative to the MASC system� The results showed�

��

� Instructions and messages are always displayed in a concise and positive manner�

� Displayed messages are always relevant�

� Instructions usually clearly indicate what is required�

� It is usually clear what actions a user can take at a particular instance� There are

some system command button states which are not always clear to users�

� It is usually clear what the user needs to do in order to create a desired action�

� It is always clear what information should be entered when data is requested by

the system�

� There do not exist any short cuts�

� It is not always clear what changes occur on the screen as a result of an action�

This can occur when the system response rate is slow� thus an appropriate system

response may not appear as a result of the input action�

� The system does not provide status messages while it is busy�

� It is not always clear when the system has completed the requested action� Some

of the system command buttons do not change the system state and the user may

not notice a change in the button�

� In general� the error messages state what the error was� where it occurred and why

it occurred�

� It is generally clear to users what action is required to correct an error�

� The system does not clearly indicate which system mode it is in at all times�

As there are many interface aspects which could be improved to provide the user more

informative feedback� the overall rating for this criterion is �Neutral�� In particular�

when the system is slow to respond� there should exist an indication that the system is

busy and will return to the user shortly�

��

�� Explicitness

This section explores the manner in which the system is structured and works for user

clarity� The results of this section are�

� It is not always clear what stage the system has reached in executing a task�

� It is usually clear what is required of the user to complete a task�

� Lists of options generally have clear meanings to the operator�

� It is not always clear to the user in which system state they are in�

� It is very clear what the various system modes are and their functions�

� It is not always clear how changes may a�ect other system aspects�

� Generally� the system organization and structure is clear�

� The system structure may not be immediately obvious to a user� After working

with the system� it is presumed this understanding could be gained�

� The system is not necessarily well organized from a user�s point of view� There exist

system aspects� such as initialization� which could be automated� The di�erences

in the robot controls is also not well organized from a user�s perspective�

� It is generally clear what the system is doing during a task�

There are instances where the system could be more explicit� thus we give it an overall

rating of �Neutral�� When the system is busy it currently is not clear what the user

should do to continue with the task� The user must wait for a response before continuing

with the task�

�� Appropriate Functionality

Appropriate system functionality refers to the ability to meet the user�s needs and re�

quirements while executing tasks� The results from this section are�

� The use of the mouse as an input device is generally appropriate� Some users

experience di�culties in properly rotating the agents employing the mouse�

��

� The information presentation manner regarding the task is generally appropriate�

Particularly as we permit the user to choose the information which they choose to

display�

� In general� the system provides the user with information and necessary options at

any particular stage� The inability to add new processes once an agent has been

initialized is one instance when this is not true�

� Users generally feel the system provides them with the ability to complete tasks�

� Aside from system delays� the feedback from the system is appropriate for tasks�

The system does not provide help and tutorial facilities which many users may �nd

helpful� This combined with the system di�culties listed above lead us to chose an

overall rating of �Moderately satisfactory� for the MASC system functionality�

�� Flexibility and Control

An interface should provide the necessary �exibility to meet the requirements of all

possible users and their preferences while also permitting them to feel in control of the

system� We found from this section�

� There is not always an easy manner for an instruction to be �undone��

� There exists no mechanism to redo an undone action�

� Users generally have control over the order in which information is requested and

activities can be carried out after system initialization�

� It is easy for the user to return to the general system state from any other system

state�

� The user can easily move between di�erent system modes�

� The user is only able to e�ect the rate information is displayed with the ultrasound

process� This would be a useful feature for the other sensing modality information�

� The system does not permit the user to store user preferences for later user�

��

� Users can tailor the sensing displays for their preferences by turning them on or

o�� It would be useful to permit them to rearrange the small windows layout to

their preference�

The overall system rating based upon this aspect is �Moderately satisfactory�� There

are options� such as the redo and the display rates� which would be useful additions to

the system while other aspects do permit the user to customize the interface and provide

them with a sense of control�

�� Error Prevention and Correction

This evaluation section examines the system design to determine if it minimizes the

possibility of user error and provides the user with the ability to verify their inputs and

correct potential error situations� The following lists this section�s results�

� The system validates most user input�

� The system clearly and promptly displays an error box on the interface when an

error is detected�

� The system permits the user to verify most inputs prior to instructing the agent

to execute the command� for instance the path planners�

� The system does not provide a cancel key to reverse an error situation� but the

user is able to instruct an agent to stop if its locomotion commands will lead to an

undesired state�

� The system ensures the user corrects all detected errors before the input is pro�

cessed�

� The user can explore possible path planning options without instructing the agent

to execute them� This is not true with the teleoperation control�

� The system catches trivial errors such as choosing an inactive agent but does not

deal with instances of double clicking� thus it does not always protect against trivial

errors�

��

� We have found that double clicking on items in the interface can cause a system

failure�

� Aside from a few errors� the system is generally bug and error free�

In general� the system is quite e�cient at detecting and preventing errors thus we have

chosen an overall rating of �Moderately satisfactory� for this section�

�� User Guidance and Support

This section explores the system�s availability of informative� easy�to�use and relevant

guidance and support to assist the user� This support should be provided in both hard

copy and on�line documents� We have already stated that there exists no on�line help

facility� At this time there also exist no formal hard�copy documentation� The docu�

mentation that will be provided� in hard�copy� will consider the factors explored in this

section� As we currently lack documentation� the overall rating for this system aspect is

�Very unsatisfactory��

�� System Usability Problems

This section explores possible system usability issues� The results from this section are�

� Users encountered minor problems when learning how to use the system and while

attempting to understand how to execute tasks�

� Some users experienced minor di�culties �nding the information they desired and

then determining how this information related to other system aspects�

� Colors and other information appear clearly on the screen and do not over populate

the system�

� The system is fairly �exible�

� Users experienced situations in which they felt lost in the system during the exper�

iments� We imagine this �feeling� would increase when using the complete system�

� The system does not require the user to retain signi�cant information about the

task in memory�

��

� System response times are slow enough that the user generally knows what is

happening in the system� In situations where the system response time is too slow

the user does not feel as they know what the system is doing�

� All textual information which appears on the screen remains until the user has read

it and supplied a response�

� The user may experience very slow system response times�

� If the user is accustomed to a double�click oriented system� this system will gener�

ally fail when they double�click�

� Employing the mouse as an input device makes it easier to use in most instances�

� The user always knows where to input information�

� System initialization requires too much input time but other input aspects do not

require a signi�cant amount of time�

� In general� the user does not have to be extremely careful about causing error as

there are mechanisms to detect them�

The �nal evaluation section asks open questions about the system� Each of the

questions in this section have been previously answered in the other sections of this

Chapter� therefore� I will not reiterate�

This evaluation provided deeper insights into the entire system abilities beyond what

we were capable of testing with the human factors experiments� Therefore� it has been

a useful tool�

This chapter has presented the human factors experimental data analysis results�

We have presented the results of the subject�s responses to the various questionnaires�

our responses to Ravden and Johnson�s usability evaluation check list� as well as the

statistical analysis of data related to task completion times� number of commands and

errors created and perceived workload� We also presented information regarding the

sensing modalities subjects employed as well as their task completion data�

�

Chapter �

Discussion

The purpose of this chapter is to discuss the results presented in Chapter Six� The �rst

section discusses the results for the MASC interface� then the mediation hierarchy and

�nally the Multiagents system�

The results from the human factors experiments may only be generalized for a popula�

tion of computer literate novice MASC system users� This is based upon the backgrounds

of the subjects who participated� Also� the experimental results may only be generalized

to the reduced MASC system version� We can only predict what the results would be

when the subjects encountered the complete system� Thus these experiments provide

signi�cant analysis for the task and regulation levels of the mediation hierarchy� while

permitting preliminary evaluation of the processing and data levels�

�� MASC System Discussion

�� General Discussion

There exist signi�cant evidence that a novice MASC and multiagent systems user can

successfully complete tasks� There were only eight accidents during one hundred �fty six

trail runs� We can also state� as the number of agents involved in the task increased� the

number of accidents increased� There does not exist enough evidence that the number

of accidents would decrease over time and practice� but one would conjecture this would

be true�

As was expected� the number of commands required by the users increases signi��

cantly for the four agent task� in fact� it practically doubled from the other two tasks�

� �

As the subjects moved from performing the single and two agent task to the four agent�

there was an increase in the task completion time� Also� there was a decrease in both the

number of commands and task completion time during the second session� This shows

as the subjects became more familiar and practiced with the system they were able to

improve their performance� Surprisingly� there did not exist a vast di�erence in the

number of commands created between the single and two agent tasks� It was expected

there would be more commands for the two agent task� During session two� the average

number of commands created was essentially equal to that created for the single agent

task�

It was interesting to note which sensing modalities subjects preferred� It was expected

that a majority would gravitate toward a few sensing modalities� It is plausible to believe

the subjects preferred the agent�s images because they were more familiar with that

display format and did not feel they could properly interpret the other data display types�

For those subjects without a formal training in computer science� they commented on

the di�culties of understanding the concept and interpretation of a state diagram� The

subjects felt the ultrasound process took too long to provide information and therefore

was not useful� As some subjects did not feel certain sensing modalities were useful this

would account for the fact subjects only used all available displays four of the one hundred

�fty six trials� What was also interesting was the subject�s preference to establish all

sensing displays prior to beginning their work with the agents� They also did not change

the displays during the task executions�

It was expected the subjects would commit many more errors when �rst beginning

to use the system� While the number of errors during the �rst experimental session

was more than double that in the second session� the total number of errors was fairly

low for the given number of trials� This also shows that as the subjects increased their

experience level they committed fewer errors� One would like to conclude the number

of errors would still decrease� but this experiment does not provide enough evidence to

support this statement�

The subject�s perceived workload was anticipated to increase as the number of agents

required for a task increased� The analysis showed that the values did generally increase

from the single agent to the four agent tasks and that this was a signi�cant relationship�

� �

We also expected as the subjects became more familiar with the system these values

would fall in the second session� The results showed this was true and in fact� the mean

workload values between the single and two agent tasks were essentially equivalent during

the second session� The analysis did not �nd the decrease in the perceived workload value

between sessions signi�cant�

We anticipated that some of these variables would be related� In particular� we

expected the subject�s perceived workload measure would be e�ected by the number of

commands and the amount of time the task required as well as the number of errors a

subject committed� The analysis showed the perceived workload was generally e�ected

by the number of commands and the amount of time required to complete the task� It

was interesting to note the errors the subject committed did not display a signi�cant

e�ect on the subject�s perceived workload measure� Since the most frequent types of

errors occur during the formal task execution� not the initialization period� one would

expect a relationship�

We have already stated the number of commands required for the tasks was essen�

tially equal for the single and two agent tasks and increased for the four agent task�

It was anticipated there would exist a signi�cant relationship between the number of

commands created and the amount of time required to complete the task� This was

generally not found to be the case� The analysis generally found this relationship was

insigni�cant� This implies that some subjects were capable of executing many commands

just as quickly as others who used fewer commands� As the agent�s speed remained the

same throughout the experiments� this is exceptable explanation� There also exists a

signi�cant relationship between an increase in the number of errors with increases in the

number of commands and task completion time� These relationships can be explained

by the fact that �� of all errors occurred during the four agent task which required the

greatest number of commands and completion times�

Based upon these results and the subject�s responses to the post�task questionnaire�

we can also deduce that indeed the four agent task was more di�cult than the single

and two agent tasks� As we encountered hardware di�culties with the VisionBot which

limited its abilities� the anticipated result that the two agent task would be more di�cult

than the single agent task is not upheld�

�

It was shown that the subject�s workload values increased between tasks� particularly

between the easier tasks and the four agent task� The perceived workload measures are

based upon a scale from zero to one hundred� It is interesting to note� the highest per�

ceived workload mean value was �� while the highest actual value was �� and these

values dropped during the second session� This shows the system does not signi�cantly

overload the user�s workload capabilities�

It was found that only twice users did not detect a problem situation� The �rst

time the subject was attempting the task without sensors and the second� the subject

forgot the SensorBot was unable to avoid obstacles automatically� So� for these given

instances the subjects were unable to detect problem situations� One situation which

many subjects encountered was the lack of knowing exactly where the agent was located�

This could be overcome with the addition of a localization process�

It was observed that subjects approaches to driving the agents were as varied as

the humans who participated� Some subjects preferred to create many commands and

then issue the emergency stop command if they did not like the agent�s actions� Others

preferred to drive very conservatively� creating one small motion� waiting for the agent

to complete it and then creating another� In both extremes� the subjects were theoret�

ically creating what would be classi�ed as �unnecessary� commands� Of course� these

particular subjects would likely disagree with this statement�

Subjects generally found the tasks they were asked to execute interesting during the

�rst session but were somewhat bored during the second session� They would have pre�

ferred new tasks� They also found they generally felt in control of the system during

the tasks� Both results were not anticipated and can be accounted for in two manners�

Either the tasks were too easy or the system e�ectively provides the user with the capa�

bilities required for the tasks� The second assumption was substantiated through their

responses to questions concerning the system�s abilities and �exibility to complete the

required tasks�

While we would have preferred the subjects to give the system a �terri�c� overall

rating� this is not realistic� The results do show that the overall rating was generally

high� The results also showed that the system tends not to overly frustrate users�

� �

The subjects raised several usability issues concerning the screen layout and the

display of information in the smaller windows� Many subjects desired the ability to

rearrange the windows into their preferred customization once they had displayed all

the information� They also would have preferred the ability to turn o� the views from

the virtual cameras so that those windows could be used to display other sensory data�

The subjects agreed the main working window was preferable and useful� These results

agreed with our results from the usability evaluation check list�s �visual clarity� section�

While the subjects found the command buttons use logical� they sometimes lost track

of their current position in the system state tree� Given this fact� subjects still responded

the buttons were generally easy to use� This is a result we are happy to �nd�

The subjects found the messages and terminology used throughout the system con�

sistent� This upheld our belief that such information was consistent� They also agreed

the position of error information was consistent� We anticipated this and it substantiated

the evaluation check list results�

It was interesting the �nd the subjects believed the system was easy and quick to

learn� When we were explaining the system during the training session� many subjects

appeared to be completely confused� These same subjects stated later that once they

began to work with the system� it was much easier than they had anticipated�

The issue of the system speed and information update were identi�ed by users as a

problem in the usability check list� The experimental results substantiated this �nding�

In general� the users found many constructive usability issues� The usability eval�

uation check list� which we completed� also identi�ed many of these same issues� It

also brought notice to many good system aspects� such as the consistency of the various

system aspects� This was not an issue we signi�cantly considered when developing the

interface�

�� Mediation Hierarchy Discussion

While the above results show many good and bad things with the MASC interface�

we were primarily interested in substantiating the mediation hierarchy� As we have

mentioned� more research would be required to fully substantiate the hierarchy�

� �

The fact novice subjects were able to successfully instruct the agents to complete the

tasks shows they were capable of communicating with the agents at the Task level� This

result would be expected� as we have succeeded in communicating tasks to the agents�

but this could not be assumed� This implies the system is straightforward enough for

the user to act as the task planner�

Generally speaking� the concepts behind the three interaction types of the Regulation

level were found to be upheld� The command interaction permitted the subjects to

e�ectively teleoperate the agents employing the mouse while also permitting them to

control the agent�s actions with the agent command buttons� The subjects were also

able to e�ectively switch control between the agents� Two issues which subjects raised

were� they sometimes found the creation of rotation commands di�cult and an easier

method of switching between agents would be to click directly upon the virtual agent�

The results show the subjects found the concept behind the request interaction logical�

While they did not fully exercise this capability� they responded that requesting sensory

information was easy� We also anticipated we may �nd there was some information

displays which should be automated� The subjects substantiated this fact in stating it

would be preferred if the system remembered their preferences and displayed them upon

initialization� The user could then customize the displays by turning on and o� those

which were and were not desired for the particular task�

While the subjects did not employ the path planning processes which are generally

associated with the speci�cation interaction� they were required to specify the agent

processes required to interact with the agent� The analysis of their abilities to specify

the initialization provides us with some preliminary results for this interaction type�

This is true because the speci�cation interaction�s purpose is to permit the operator to

specify necessary information prior to the commencement of processing� The processes

required for a task must be initialized prior to commencing the task execution� The

subjects were able to e�ectively specify the necessary information required to establish

the agents� This speci�cation is fairly simple� as they must only choose the processes�

but this result displays the successful completion of the interaction� This result does not

permit us to extrapolate to the subject�s ability to complete such an interaction with the

path planner speci�cations�

� �

These experiments did not formally incorporate any of the Processing level�s capa�

bilities� but we can deduce some preliminary conclusions to this level�s necessity� For

instance� we did not provide the subjects with the training necessary to change the ul�

trasound process clustering variables� As a results of the fairly high default values for

these variables� the subjects did not prefer to use this sensing modality because they

found it provided very little or no information� Thus� if the subjects had obtained the

knowledge to interact with the process to change these variables� it is conceivable they

would have succeeded in doing such� Also� a few subjects suggested the ability to mod�

ify the image quality and frame rate of images received from processes� This type of

interaction would reside on the processing level� Subjects also demonstrated a certain

frustration level when the VisionBot�s obstacle avoidance process detected a wall and

proceeded to turn in circles in an attempt to avoid the obstacle� Many subjects desired

the ability to instruct the agent to stop attempting to avoid the wall� This would be an

instance of the human overriding the process�s decision� Also� if the process was able to

deduce it was in a situation where it could not �nd a way out� the process could request

the human�s assistance� Again� these are all preliminary conclusions as the experiments

did not formally test this level�s capabilities�

The experiments also were not capable of formally providing results to substantiate

the mediation hierarchy�s data level� We are able to establish some informal preliminary

results based upon the subject�s reactions and comments on the system� The most

common complaint amongst the subjects was associated with the di�culty of determining

exactly where the agent was in the real world verses the model� This was primarily due

to wheel slippage on these mobile agents� Subjects stated they would like a mechanism

�a localization procedure to determine the proper location and then either reposition the

virtual agent or reset the actual agent�s odometry and heading� This particular aspect is

one of the primary examples we use to substantiate this level�s existence� As this issue

arose frequently� we can deduce the data level is a necessary element�

� �

�� Multiagents System Discussion

Throughout the experiments many issues were raised within the entire multiagents sys�

tem� The purpose of this section will be to discuss these multiagent system issues�

In general� the overall system was found to be very fragile� both in a software as

well as a hardware sense� In some instances� the software processes were not sophisti�

cated enough or did not exist� The hardware problems range from di�culties with dead

batteries to loss of communications�

The agent�s ability to sense its environment as it exists is not su�cient for di�cult

tasks� In the tasks we required� the sensing capabilities were not tested to the envelope

of their abilities� For instance� the obstacle avoidance process version which is integrated

into the system is very fragile and unreliable� It does not always avoid obstacles but

when it detects a wall it will go around in a circle attempting to avoid it� Also� if the

focus or aperture of either of the two cameras is slightly di�erent than the other� this

process produces ambiguous data� It should be noted that a newer version of this process

does exist but is not integrated into the MASC system�

Another major di�culty is the agent�s narrow view provided by the cameras� Many

times subjects and agents are unable to acquire a signi�cant environmental view which

would assist them in their task� We as humans have a ��o �eld of view but when we

explore what we can obtain from the agent�s �eld of view� it is signi�cantly less� There

exist some people who have di�culties performing tasks with their own �eld of view� it

is infeasible to expect a robotic agent to properly understand its environment with such

a narrow �eld of view� Perhaps the new ��o �eld of view camera will assist with this

di�culty�

It is absolutely necessary to have a localization procedure� Many times throughout

the experiments we observed subjects working with an agent when they were not positive

of its location� In one instance� the virtual agent appeared to be almost into the GRASP

laboratory�s front o�ce while the real agent was located just in front of the pillar in the

eastern portion of the laboratory� This discrepancy was due to the fact the agent had

become stuck on its power cable and then came free� While the agent was stuck� the

odometry and heading readings were updated thus updating the virtual agent�s positional

� �

information� It is interesting to note� the subject eventually did successfully complete the

task relying upon the agent�s real time images� We believe this subject was an exception

and most subjects would not have been able to complete the task in a similar situation�

This is an extreme example� and we observed that even the normal slippage of an agent

is signi�cant enough that localization would be very helpful to the human operator�

One complaint from the subjects was that they disliked the requirement of initializing

all of an agent�s processes at once� They would have preferred to start one process for

an agent and then as was needed� add new processes� This currently is not permitted

in the multiagents system because of the TCP�IP communication protocol employed�

This protocol requires that a process know all other processes it will communicate with

when it begins� This is further complicated when various processes must communicate

with more than one process� All processes must be started before processing can begin�

This is not realistic� the human may not initialize a process when they start a task and

then may �nd it is necessary and is unable obtain the required information� Also� the

current con�guration does not permit the operator to shut down a process and restart it

later as required� Originally� the human operator interface was written in such a manner

as to start up and shut down processes as needed� but this had to be changed as more

processes were integrated into the system because of the communication protocol� This

is a very great restriction of the multiagent�s system�

Another di�culty associated with the system�s communications is the very unreliable

communications with the manipulatory agents� Currently� the manipulatory agents are

controlled by two personal computers and communicate with a Sun workstation via serial

communications through the personal computers� This communications channel is very

unreliable and frequently either one or both of the manipulatory agents would loose

communications with the personal computers and hence the rest of the system� This

leaves the human unable to work with these agents� As the experiments showed� this

was a signi�cant problem�

Currently the system runs the VisionBot and the SensorBot�s communication through

radio ethernet� It was found this medium was not su�cient to support communications

for both agents to the MASC interface� In fact� communications became so slow we

were required to connect the agents to their respective ethernet cables in order to obtain

� �

reasonable responses during the human factors experiments�

The system�s agents are heterogeneous� This leads to di�culties as we attempt to

execute more di�cult tasks as some agents have no sensing capabilities� The human

must rely upon information from the observation agents to monitor these other agents�

This limits the types of tasks the system can perform�

At the time of the experiments� the VisionBot�s batteries were dead� Thus for these

experiments the agent was only able to move forward and backwards� If one attempted

to turn the agent on a zero radius� the agent would fault� This limited the activities we

could request the agent to perform and thus reduced the experimental tasks di�culty�

New batteries were purchased but arrived too late to be used for these experiments�

There were a few instances when the agent�s bumpers were activated for no apparent

reason� In one instance� the problem was so consistent we disconnected the bumper�

Also� the ability to create a software override of the bumper which permits the operator

to move the agent away from the object and continue on with a task is necessary�

There are also problems associated with the consistency between system processes�

In some instances� processes require information in millimeters� in other centimeters�

There should be a standard established across all system code developed� This standard

should not only apply to measurements but also other aspects such as communication

protocols� There is also a need for developers to freeze code versions and place not only

the executable but all other associated �les into a directory where it will not be modi�ed�

During the MASC system development this was a recurring problem� We would integrate

a process and then it would change and we would be left with the executable and no

manner to rebuild it or repair its bugs� Also� in order for this type of a system to work

e�ectively� all system developers must consider not only their current needs but also

their own future needs� the needs of the entire system� the established standards and

the human operator�s requirements� As much of this system�s code was not developed in

this manner� it was extremely di�cult to integrate it into the MASC system in a useful

manner� For instance� if a process was developed and calculated information in a local

coordinate frame� it was not useful for the human operator who observes the world in

the global coordinate frame� The developer should be responsible for translating this

information into the global coordinates so it is useful to other system processes� As

��

the developer best understands the algorithm and the process� they are best equipped

for such a translation� This section discussed many problematic issues raised by these

experiments as well as the multiagent experiments�

Overall� the experimental results� combined with knowledge we have gained through

the formal multiagents experiments� indicate the basic mediation hierarchy concept and

purpose are upheld� We found the subjects were able to interact e�ectively with the

upper levels of the hierarchy while demonstrating a need for the types of interactions

the processing and data level provide� We also found the experiments upheld our basic

research question as the novice subjects were capable of e�ectively interacting with the

available system levels such that feasible tasks could be successfully completed in a

reasonable time frame� These experiments also displayed the system is usable while

there exist aspects which could be changed which would improve its overall usability�

We were also pleased to �nd subjects did not encounter signi�cantly high workload

levels� As these experiments were considered a feasibility study� the feasibility of the

concepts behind the development of the MASC system appear reasonable�

This chapter has provided a discussion of the experimental results� We discussed the

results in terms of the MASC interface� then in relation to the mediation hierarchy� and

�nally their relation to the entire multiagents system�

��

Chapter �

Summary

This chapter presents a summary of this dissertation� The research contributions are

presented followed by the future work beyond this dissertation� Finally� we present our

conclusions�

�� Contributions

Many supervisory control systems built with human�machine interfaces do not permit the

human supervisor to interact with all system levels� Most of these interfaces permit only

high level interactions which pertain to sensory data and system state monitoring� They

do not permit the supervisor to override or assist with low level decisions� For instance�

when dealing with robots working in a hazardous waste environment� it is di�cult and

unsafe for a human to physically enter the environment� While teleoperation has been

employed in these environments� there are numerous tasks which could be automated

if the human�machine interface permitted the human supervisor to take control of the

system� interact with the individual processes or permitted the system to interactively

request assistance�

The major research contribution is the mediation hierarchy theory development� This

hierarchy enables the multiagent system to successfully complete feasible tasks and has

been shown feasible through the human factors as well as the multiagents experiments�

The human factors experiments not only provide results concerning the supervisor�s

ability to interact with the system at the higher levels of the mediation hierarchy� but

permit us to draw preliminary conclusions pertaining to the lower level interactions

��

while also substantiating the need for such a theory as a basis for interface development�

This theory combined with a working experimental system demonstrates the hierarchy�s

ability for our test bed� We feel the mediation hierarchy theory is applicable to other

research areas including robotics� air tra�c control� command and control systems� etc�

It can improve the abilities of autonomous and teleoperated systems� By employing

this theory� autonomous systems may be converted into semi�autonomous systems which

succeed when the strictly autonomous versions would fail� Also� teleoperated systems

may be improved by increasing the amount of autonomy which under current system

de�nitions would be infeasible� This theory is applicable to control systems at remote

sites� For instance� there may exist a specialist for a speci�c robotic workcell� If the

company has many such workcells located throughout the country� the specialist would

normally have to travel to the workcell location to determine the problem and repair

it� The mediation hierarchy permits the specialist to remotely monitor the workcell in

question and interactively determine the problem and correct it� This would reduce

the workcell down time as well as reduce the transportation costs� Another research

contribution is the system�s ability to request assistance from the human supervisor�

In the scenario described above� this ability could keep the workcell from failing� The

workcell could request assistance from the operator before the problem becomes too

signi�cant�

The supervisor�s ability to work within all system levels via the mediation hierarchy

elevates supervisory control to a higher level� The supervisor is able to interact with

all levels� thus he or she may obtain a better system view and therefore better perform

the monitoring task� This permits the supervisor to obtain information concerning each

process� state and to interact with these states� While all these interaction levels may not

be used continuously� the fact they exist and that the supervisor may e�ect the process�

outcome through the interactions is of higher importance� If a process is producing

incorrect information� it is likely the system will fail� In this situation the supervisor will

be able to interact with the process to assist it or if absolutely necessary� take control

away from the process and either teleoperate the process or give control to another

process�

��

The supervisor�s ability to request only the data required for the current task is

another contribution� This ability reduces the chances that the human�s sensory abilities

will be exceeded as it is likely the human supervisor will not request unnecessary system

information� It also implies the human supervisor will be able to request relevant data

as opposed to miscellaneous data of little use for the current task diagnosis� The human

factors experiment found subjects did not request unnecessary data displays� This ability

also potentially reduces the communication load between the agents and the human�

machine interface� When data is not required� the corresponding information channel

can remain empty� thus reducing the number of channels the interface must monitor� As

a result of the human factors experiments� we found users favored this ability but tended

to prefer speci�c sensory data displays� Therefore� it would be bene�cial to retain their

preferences and then allow them to modify the displays for a particular task�

The supervisor�s ability to display processed as well as raw data increases the system�s

and supervisor�s capabilities� The supervisor may combine these two information forms

to verify the processed data� verify the existence of objects in the environment� localize

the agents in the environment as well as add newly detected objects to the world model�

The supervisor may also employ the raw data to handle tasks for which there exist

no processes� such as the monitoring of �blind� agents through image viewing from a

properly positioned observation agent� This extends the system�s abilities by permitting

the human supervisor to create a new �process��

The MASC system itself is a contribution to the robotics community� We have not

found evidence of such a complete interface for other multiple mobile robotic systems�

Also� the human factors analysis shows the interface is fairly complete� easy to use� and

consistent� This is considered a contribution as it can be employed by others as a basis for

future systems development� Also� as we have developed the MASC interface integrated

with our test bed� it permits testing of other multiple agent� sensory and robotics oriented

concepts� This integration permits the completion of tasks feasible within this test bed�

The results show novice users are able to e�ectively control the agents within the system

via MASC to execute tasks�

��

�� Future Work

There exist many extensions to this work and the multiagents system which can be

included in this section� It is necessary to discuss the future work required for the entire

multiagents system as it ultimately e�ects the MASC system�

�� MASC Interface Future Work

The human factors experiments raised many usability issues which could be improved

upon in the MASC interface� Many of these issues improve the system�s usability� but

would not include further process integrations or develop the mediation hierarchy� These

are very important issues within the MASC system development and should be considered

during the development of later interface versions�

There exist new versions of previously integrated processes which this system should

include� These new versions are more robust and would improve not only the MASC

system but also the multiagent�s system� There is also a need to either develop or locate

new sensing processes for the system� It is apparent the system requires more sensing in

order to execute di�cult tasks� One example is the integration of a localization process�

This would provide the human supervisor with a better sense of presence� Other aspects�

such as the integration of the robot manipulators� would also be helpful�

In general� there exists a need for the redesign� or partial redesign� of the multiagents

system� Currently within the MASC system there exist many modules which should

reside in other portions of the multiagents system� For instance� there should exist a

module between the MASC system and the mobile agents� This module should be re�

sponsible for recording and communicating information to all processes� The current

design requires MASC to receive all the information and then communicate it to the

other agents� This is a design �aw� Not only is it architecturally incorrect� but the re�

quirement increases the amount of processing required within MASC� The human factors

experiments found the subjects believed the system ran too slowly� This accounts for a

portion of this reduced speed� There also exists a need for the standardization of the

processes which compose the multiagents system� The interface currently must account

for every di�erence between the processes� This should be eliminated�

��

There were other issues discussed in Chapter Seven regarding the multiagents system

which should be considered� For instance� a better communication protocol between

the manipulatory agents and the remainder of the system is required� It is of little

importance how many or what level of interactions we provide the human supervisor if

he or she is unable to communicate with the agents�

It appears that the inclusion of multiple display types for sensory information would

be useful� While this currently exists for images� small image or overlay onto the model�

other sensory data displays could be created� Also� the ability to control information in

the displays� such as image resolution� would be helpful�

�� Mediation Hierarchy Future Work

The current MASC system does not fully implement the mediation hierarchy� In order

to show a complete proof of our hypothesis this implementation is required�

We decided not to integrate a task planner for this work because this was previously

shown to be feasible� In order to fully integrate the task level such a planning mechanism

should be incorporated�

While the current integrations on the regulation level e�ectively permit the three

interactions there exist revisions which would simplify and improve these interactions�

For instance� the human factors experimental results provided ideas for improved control

interactions�

We have integrated some interactions on the process level but there exist many more

interactions which could be created� Almost all currently integrated processes contain

further aspects which could be integrated at this level� Also� with the need to integrate

new processes� there will arise the need for further development at this level�

The MASC system currently does not have processes available which would permit

interactions on the data level� As the human factors experiments demonstrated� there ex�

ists a need for processes such as localization� this would provide opportunities to develop

the interactions at this level�

��

�� Human Factors Analysis

There are aspects of the human factors experiments which also fall into the future work

section� An enormous amount of data� beyond what was reported in this document� was

collected during the experiments� This data could be analyzed for further results� Also

the data which was reported could be further analyzed for other factors beyond those

reported� We can foresee further human factors testing� For instance� we could conduct

experiments with the entire MASC system as it currently exists� Also� upon resolving

the above issues with the overall multiagents system and increasing its capabilities more

testing could be conducted� The most vital human factors testing would occur upon

completion of further mediation hierarchy integration� The purpose of these tests would

be to fully substantiate our hypothesis which led to the development of the mediation

hierarchy and its integration into the MASC interface�

This section has touched upon some of the future work issues beyond the scope of

this thesis� There exist enough issues that work could continue for years�

�� Conclusions

We have presented the Multiple Agent Supervisory Control �MASC system which has

been developed in conjunction with the University of Pennsylvania General Robotics and

Active Sensory Perception Laboratory�s multiagent project� The goal was to create a

semi�autonomous system which successfully executes tasks� We have also de�ned the

mediation hierarchy� This hierarchy is the underlying basis for the MASC system de�

velopment� The mediation hierarchy provides the supervisor with the ability to interact

with the various multiagent system processing levels� It permits the system�s agents to

work autonomously until they request supervisory assistance or the supervisor detects a

problem in the system and takes control of the process in question�

This dissertation also presented a description of our test bed and the available pro�

cesses as well as the general MASC human�machine interface� We described the inte�

grations of these processes as well as the integrations of external processes such as the

global path planner�

��

This document provides a detailed description of the human factors experimental

design and implementation� a complete review of the data analysis as well as an in depth

discussion of the results� This information was provided in a format which adheres to

the standards published by the American Psychological Association�

While developing the mediation hierarchy concept� we intended that it would improve

the supervisor�s abilities and create a more robust system� The experimental results

substantiate the human operator�s need to work within all system levels� They also

partially substantiate our claim this hierarchy will permit the supervisor to interact with

all system levels in order to correct problems and permit the system to successfully

complete assigned tasks� It was found that the task and regulation level interactions and

the need for the processing and data levels were substantiated�

��

Appendix A

Further Graphical Presentation

of Human Factors Experimental

Results

The purpose of this appendix is to present the human factors statistical analysis results

which were not completely reported in Chapter Six�

A�� Number of Commands

1 2 3

5

10

15

20

25

13.612.9

25.2

12.2

10.5

21.3

Mea

n N

um. o

f L

ocom

otio

n C

omm

ands

Tasks

Session 1

Session 2

Figure A�� The mean number of locomotion commands by task and session�

��

1 2 3

1

2

3

4

5

6

2.1

2.6

6.6

1.92.2

5.

Mea

n N

um. o

f A

gent

Mod

e C

hang

es

Tasks

Session 1

Session 2

Figure A�� The mean number of agent mode commands by task and session�

1 2 3

2

4

6

8

10

0

2.8

11.2

0

1.5

10.5

Mea

n N

um. o

f A

gent

Con

trol

Cha

nges

Tasks

Session 1

Session 2

Figure A� � The mean number of agent switch commands by task and session�

Task Locomotion System Mode Agent Mode Agent TotalCommands Commands Commands Switches Commands

Single agent ��

Two agent ��

Four agent ��

Total ��

Table A�� Break down of all commands created by task as percentage of total commands�

��


� �� x ��



Total ��

Table A�� The ANOVA analysis of the number of commands between the single andtwo agent tasks�

All

Com

man

ds T

wo

Age

nt

All Commands Single Agent

15 20 25 30

20

25

30

35

Figure A�� The linear regression plot for the number of commands for the single versusthe two agent task for all data�


� �� x ��



Total ��

Table A�� The ANOVA analysis of the number of commands between the two and fouragent tasks�

��

15 20 25 30 35 40

30

40

50

60

70

80

All

Com

man

ds F

our

Age

nt

All Commands Two Agent

Figure A�� The linear regression plot for the number of commands for the two versusfour agent tasks for all data�

15 20 25 30 35

15

20

25

30

All

Com

man

ds S

essi

on T

wo

All Commands Session One

Figure A�� The linear regression plot for the number of commands for the single agenttask between sessions�


� �� x ��



Total ��

Table A�� The ANOVA analysis of the number of commands for two agent task betweensessions�

��

15 20 25 30 35 40

14

16

18

20

22

24

All

Com

man

ds S

essi

on T

wo


Figure A�� The linear regression plot for the number of commands for the two agenttask by session�


� �� x ��



Total ��

Table A�� The ANOVA analysis of the number of commands the four agent task betweensessions�

30 40 50 60 70 80

35

40

45

50

All

Com

man

ds S

essi

on T

wo


Figure A� The linear regression plot for the number of commands for the four agentby session�

��

A�� Number of Errors


� �� x ��



Total ��

Table A�� The ANOVA analysis of the number of errors received between single andtwo agent tasks�

1 2 3 4

-0.05

0.05

0.1

0.15

0.2

Err

ors

Tw

o A

gent

Errors Single Agent

Figure A�� The linear regression plot for the number of errors for the single versus twoagent tasks for all data�

��


� �� x ��



Total ��

Table A�� The ANOVA analysis of the number of errors received between the single andfour agent tasks for all data�

1 2 3 4

0.5

1

1.5

2

2.5

3

Err

ors

Four

Age

nt

Errors Single Agent

Figure A�� The linear regression plot for the number of errors for the single versus fouragent tasks for all data�


� �� x ��



Total ��

Table A� The ANOVA analysis of the number of errors received between the two andfour agent tasks for all data�

��

0.2 0.4 0.6 0.8 1

0.5

1

1.5

2

2.5

3

Err

ors

Four

Age

nt

Errors Two Agent

Figure A�� The linear regression plot for the number of errors for the two versus fouragent tasks for all data�

A�� Completion Times


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times between the single andtwo agent tasks for all data�

��

300 400 500 600

400

500

600

Com

plet

ion

Tim

es T

wo

Age

nt

Completion Times Single Agent

Figure A�� The linear regression plot for the completion times between the single andtwo agent tasks for all data�

400 500 600

200

400

600

800

1000

1200

Com

plet

ion

Tim

es F

our

Age

nt

Completion Times Two Agent

Figure A�� The linear regression plot for the completion times between the two andfour agent tasks for all data�

��


� ��

x ��


Regression � �� Error �� x ��

Total �� x ��

Table A�� The ANOVA analysis of the task completion times between the two andfour agent tasks for all data�

300 400 500 600

200

400

600

800

1000

1200

Com

plet

ion

Tim

es F

our

Age

nt

Completion Times Single Agent

Figure A�� The linear regression plot for the completion times between the single andfour agent tasks for all data�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times for the single agent taskbetween sessions for all trials�

��


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times for the two agent taskbetween sessions�

400 500 600

300

325

350

375

400

425

Com

plet

ion

Tim

es S

essi

on T

wo


Figure A�� The linear regression plot for the completion times for the two agent taskbetween sessions�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times for the four agent taskbetween sessions�

��

200 400 600 800 1000 1200

400

450

500

550

Com

plet

ion

Tim

es S

essi

on T

wo


Figure A�� The linear regression plot for the completion times for the four agent taskbetween sessions�

A�� Perceived Workload Measures


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the perceived workload measures between the singleand two agent tasks for all data�

��

10 20 30 40 50 60 70

10

20

30

40

50

60

Perc

eive

d W

orkl

oad

Tw

o A

gent

Perceived Workload Single Agent

Figure A�� The linear regression plot for the perceived workload measures between thesingle and two agent tasks for all data�

10 20 30 40 50 60 70

20

40

60

80

Perc

eive

d W

orkl

oad

Four

Age

nt

Perceived Workload Two Agent

Figure A�� The linear regression plot for the perceived workload measures between thetwo and four agent tasks for all data�

��


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the perceived workload measures between the twoand four agent tasks for all data�

10 20 30 40 50 60 70

20

40

60

80

Perc

eive

d W

orkl

oad

Four

Age

nt

Perceived Workload Single Agent

Figure A�� The linear regression plot for the perceived workload measures between thesingle and three agent tasks for all data�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the perceived workload measures for the singleagent task between sessions�

��


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the perceived workload measures for the two agenttask between sessions�

10 20 30 40 50 60 70

10

20

30

40

50

60

Perc

eive

d W

orkl

oad

Sess

ion

Tw

o


Figure A�� The linear regression plot for the perceived workload measures for the twoagent tasks between sessions�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the perceived workload measures for the four agenttask between sessions�

��

30 40 50 60 70

20

40

60

80

Perc

eive

d W

orkl

oad

Sess

ion

Tw

o


Figure A�� The linear regression plot for the perceived workload measure for the fouragent tasks between sessions�

A�� Combined Analysis

A�� Perceived Workload Measures

Perceived Workload Measure Versus the Number of Commands

20 40 60 80

20

40

60

80


Perc

eive

d W

orkl

oad

Figure A�� The linear regression plot for the number of commands versus the perceivedworkload measures for all data�

��


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus perceived workloadmeasures for session one�

20 30 40 50

20

40

60

80

Perc

eive

d W

orkl

oad


Figure A�� The linear regression plot for the number of commands versus perceivedworkload measures for all data in session two�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus perceived workloadmeasures for the single agent task�

��

15 20 25 30 35

10

20

30

40

50

60

Perc

eive

d W

orkl

oad


Figure A�� The linear regression plot for the number of commands versus perceivedworkload measures for the single agent task for all data�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus perceived workloadmeasures for the two agent task�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus perceived workloadmeasures for the four agent task�

��

15 20 25 30 35 40

10

20

30

40

50

60

Perc

eive

d W

orkl

oad


Figure A�� The linear regression plot for the number of commands versus perceivedworkload measures for the two agent task for all data�

30 40 50 60 70 80

20

40

60

80

Perc

eive

d W

orkl

oad


Figure A�� The linear regression plot for the number of commands versus perceivedworkload measures for the four agent task for all data�

��

A�� Perceived Workload Measures Versus the Number of Errors

0.5 1 1.5 2 2.5

20

40

60

80

Perc

eive

d W

orkl

oad


Figure A�� The linear regression plot for the number of errors versus perceived workloadmeasures for the all data�

A�� Perceived Workload Measures Versus the Completion Times

200 400 600 800 1000 1200

20

40

60

80

Perc

eive

d W

orkl

oad

Completion Times

Figure A�� The linear regression plot for the task completion times versus perceivedworkload measures for the all data�

��


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times versus perceived workloadmeasures for session one�

200 400 600 800 1000 1200

10

20

30

40

50

60

Completion Times

Perc

eive

d W

orkl

oad

Figure A�� The linear regression plot for the completion times versus perceived work load measures for session one�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times versus perceived workloadmeasures for session two�

��

300 400 500 600 700

20

40

60

80

Completion Times

Perc

eive

d W

orkl

oad

Figure A�� The linear regression plot for the completion times versus perceived work load measures for session two�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times versus perceived workloadmeasures for the single agent task�

300 400 500

10

20

30

40

50

60

Completion Times

Perc

eive

d W

orkl

oad

Figure A�� The linear regression plot for the completion times versus perceived work load measures for the single agent task for all data�

��


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times versus perceived workloadmeasures for the two agent task�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the task completion times versus perceived workloadmeasures for the four agent task�

200 400 600 800 1000 1200

20

40

60

80

Completion Times

Perc

eive

d W

orkl

oad

Figure A�� The linear regression plot for the completion times versus perceived work load measures for the four agent task for all data�

��

A�� Number of Commands

A�� Number of Commands Versus the Number of Errors


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of errors versus number of commandsfor all data�

A�� Number of Commands Versus Completion Times

200 400 600 800 1000 1200

20

40

60

80

Tot

al N

umbe

r of

Com

man

ds

Completion Times

Figure A�� The linear regression plot for the number of commands versus completiontimes for all data�

��


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus task completiontimes for session one�

200 400 600 800 1000 1200

20

40

60

80

Completion Times

Tot

al N

umbe

r of

Com

man

ds

Figure A�� The linear regression plot for the number of commands versus completiontimes for all data during session one�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus task completiontimes for session two�

��

300 400 500 600 700

20

30

40

50

Completion Times

Tot

al N

umbe

r of

Com

man

ds

Figure A�� The linear regression plot for the number of commands versus completiontimes for all data in session two�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus task completiontimes for the single agent task�

300 400 500

15

20

25

30

Completion Times

Tot

al N

umbe

r of

Com

man

ds

Figure A�� The linear regression plot for the number of commands versus completiontimes for the single agent task between sessions�

��


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus task completiontimes for the two agent task�

400 500 600

20

25

30

35

Completion Times

Tot

al N

umbe

r of

Com

man

ds

Figure A�� The linear regression plot for the number of commands versus completiontimes for the two agent tasks between sessions�


� ��

x ��



Total ��

Table A�� The ANOVA analysis of the number of commands versus task completiontimes for the four agent task�

��

A�� Number of Errors

A�� Number of Errors Versus Completion Times

0.5 1 1.5 2 2.5

200

400

600

800

1000

1200

Perc

eive

d W

orkl

oad


Figure A�� The linear regression plot for the completion times versus the number oferrors for all data�

��

Appendix B

Human Factors Experiment

Consent Form

Human Factors Analysis of the Multiple Agent

Supervisory Control System

July ��

Investigator Ruzena BajcsyDepartment Computer and Information Science

Phone �� Email bajcsy�central�cis�upenn�edu

CONSENT FORM

Purpose� The purpose of this study is to determine if allowing a human being tocontrol di�erent levels of a robotic system �made up of several robots� will result in therobotic system completing its task more often than if it were completely autonomous� Theprimary objective is to run experiments using a graphical computer interface that allowsa human operator to intervene at di�erent points during a robot task� The success�failurerate for the robot in performing these tasks will then be assessed�

Procedures� I will be asked to control one or more robots by using a computerinterface� This will involve sitting at a computer workstation and observing a robotic taskbeing carried out on the monitor� When problems arise in the execution of the robot�stask I will then use the computer interface to try to help the robot complete its task�I will participate in a training session� a practice session� and two actual experimentalsessions� all occuring over the course of one �or possibly two� days� for a total of threehours�

��

Risks� I will not be at any risk throughout the experiment�

Bene�ts� I will receive no direct bene�ts� other than monetary�

Compensation� I will receive �� to participate in this experiment�

Con�dentiality� I understand that all information collected in this study will bekept strictly con�dential� except as may be required by law� If any publication resultsfrom this research� I will not be identi�ed by name�

Additional information� I understand that any signi�cant new �ndings developedduring the course of the study that may relate to my willingness to continue participationwill be provided to me�

Disclaimer�Withdrawal� I agree that my participation in this study is completelyvoluntary and that I may withdraw at any time without prejudicing my standing withinthe University of Pennsylvania or my class�

Injury�Complications� I understand that in the event of an injury resulting fromresearch procedures� medical treatment in excess of that covered by third party payorswill be provided without cost to me� but �nancial compensation is not available�

Subject Rights� I understand that if I wish further information regarding my rightsas a research subject� I may contact the Executive Director in the O�ce of ResearchAdministration at the University of Pennsylvania by telephoning ��

I also understand that if I have any questions pertaining to my participation in thisparticular study� I may contact the investigators by calling the telephone numbers listedat the top of page one�

I have been given the opportunity to ask questions and have had them answered tomy satisfaction

��

Conclusion� I have read and understand the consent form� I agree to participate inthis research study� Upon signing below� I will receive a copy of the consent form�

Name of Subject Signature of Subject Date

Name of Signature of DateInvestigator Investigator

Name of Witness Signature of Witness Date

��

Appendix C

Pre�Experimental Questionnaire

C�� Quesstionnaire

Identi�cation NumberAgeSex Male FemaleLevel of education High School Bachelors Master�s Doctorate

Please circle the choice which most accurately re�ects your experiences�

�� On average� how much time do you use a computer daily�Never � � � � � � � � Always

�� How often do you play computer games�Never � � � � � � � � Always

�� What level of a computer expertise do you possess�None � � � � � � � � Expert

�� What is your experience level using direct manipulation interfaces�None � � � � � � � � Expert

�� What is your experience level using computer graphics�None � � � � � � � � Expert

�� What is your experience level using a three dimensional graphical user interface�None � � � � � � � � Expert

�� What is your experience level using a direct manipulation three dimensional graphicaluser interface�None � � � � � � � � Expert

� What is your experience level working with robots�None � � � � � � � � Expert

��

�� What is your experience level working with mobile robots�None � � � � � � � � Expert

C�� Graphical Presentation of Subjects Responses

Questionnaire Choices1 2 3 4 5 6 7 8 9

0.5

1

1.5

2

2.5

3

0 0

2

0

2

1

3

2

3N

umbe

r of

Res

pons

es

Figure C�� On average� how much time do you use a computer daily� �� Never�� Always�

��

Questionnaire Choices1 2 3 4 5 6 7 8 9

1

2

3

4

5

6

7

7

5

1

0 0 0 0 0 0

Num

ber

of R

espo

nses

Figure C�� How often do you play computer games� �� Never� � Always�

1 2 3 4 5 6 7 8 9

0.5

1

1.5

2

2.5

3

0 0

1

2

3

0

2 2

3

Questionnaire Choices

Num

ber

of R

espo

nses

Figure C�� What level of computer expertise to you possess� �� None� � Expert�

��

1 2 3 4 5 6 7 8 9

1

2

3

4

4

0

1 1

2

0 0

3

2


Num

ber

of R

espo

nses

Figure C�� What is your experience level using direct manipulation interfaces�� None� � Expert�

1 2 3 4 5 6 7 8 9

1

2

3

4

1 1

4

1

0

1 1

2 2


Num

ber

of R

espo

nses

Figure C�� What is your experience level using computer graphics� �� None�� Expert�

��

1 2 3 4 5 6 7 8 9

1

2

3

4

5

6

7

7

0

2

1 1

0

1 1

0


Num

ber

of R

espo

nses

Figure C�� What is your experience level using a three dimensional graphical userinterface� �� None� � Expert�

1 2 3 4 5 6 7 8 9

2

4

6

8

9

0

1 1

0

1

0

1

0


Num

ber

of R

espo

nses

Figure C�� What is your experience level using a direct manipulation three dimensional graphical user interface� �� None� � Expert�

��

1 2 3 4 5 6 7 8 9

1

2

3

4

5

6

6

1

0

1 1

2

0 0

2


Num

ber

of R

espo

nses

Figure C� � What is your experience level working with robots� �� None� � Expert�

1 2 3 4 5 6 7 8 9

2

4

6

8

10

11

1

0 0 0

1

0 0 0


Num

ber

of R

espo

nses

Figure C�� What is your experience level with mobile robots� �� None� � Expert�

��

Appendix D

NASA TLX Questionnaire

Identi�cation NumberTask

Part �� User AssessmentsPlease place a mark on each scale that represents the magnitude of each factor in thetask� The terms are de�ned on the next page�

Demands Rating for task

Mental DemandLow High

Temporal DemandLow High

FrustrationLow High

E�ortLow High

Own PerformanceLow High

��

Mental Demand�How much mental and perceptual activity was required �e�g� thinking� deciding� cal culating� remembering� looking and searching� etc�� Was the task easy or demanding�simple or complex� exacting or forgiving�

Temporal Demand�

How much time pressure did you feel due to the rate or pace at which the task had tobe performed� Was the pace slow and leisurely or rapid and frantic�

Frustration�How insecure� discouraged� irritated� stressed and annoyed versus secure� grati�ed� con tent� relaxed and complacent did you feel during the task�

E�ort�How hard did you have to work �mentally and physically� to accomplish your level ofperformance�

Own Performance�How successful do you think you were in accomplishing the goals of the task� How sat is�ed were you with your performance in accomplishing this goal�

��

Appendix E

Post�Task Questionnaire

E�� Questionnaire

Identi�cation NumberTask

Part �� User ReactionsPlease circle the choice which most accurately re�ects your experiences� NA � Not Ap plicable� There is room on the last page for your written comments�

�� This task wasImpossible � � � � � � � � Easy NAConfusing � � � � � � � � Clear NA

Dull � � � � � � � � Stimulating NAFrustrating � � � � � � � � Satisfying NA

�� I felt in control of the system during this taskRarely � � � � � � � � Always NA

�� I was able to interpret and understand the data readings during this taskRarely � � � � � � � � Always NA

�� I was able to correct my errors during this taskRarely � � � � � � � � Always NA

�� I felt able to complete the taskRarely � � � � � � � � Always NA

�� I felt in control of the individual agentsRarely � � � � � � � � Always NA

�

�� The system�s capabilities for this task wereInadequate Power � � � � � � � � Adequate Power NA

Rigid � � � � � � � � Flexible NA

Part �� User s Comments

Please write any comments you have in the space below or on the back of this page�

��

E�� Graphical Presentation of Subjects Responses


Session 1

Session 2

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0

1

0

2

6

4

00 0 0 0 0

1 1

3

8

0

Num

ber

of R

espo

nses

Questionnaire Choices1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0 0

3 3

2

5

00 0 0 0 0

2

1

3

7

0

Num

ber

of R

espo

nses

��


1

2

3

4

5

0 0

1

0

3

5

1

2

1

00 0

3

0 0

2

3

1

4

0

Num

ber

of R

espo

nses

Figure E�� This task was �� Impossible� � Easy� �� NA�Results for the single agent task �previous page top�� two agent task �previous pagebottom� and the four agent task �this page�

��


Single Agent Task

Two Agent Task

Four Agent Task

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

0 0 0 0

1

0

2

6

4

00 0 0 0 0

3 3

2

5

00 0

1

0

3

5

1

2

1

0

Num

ber

of R

espo

nses


2

4

6

8

0 0 0 0 0

1 1

3

8

00 0 0 0 0

2

1

3

7

00 0

3

0 0

2

3

1

4

0

Num

ber

of R

espo

nses

Figure E�� This task was �� Impossible� � Easy� �� NA�Results for session one �top� and session two �bottom�

��


Session 1

Session 2

1 2 3 4 5 6 7 8 9 10

2

4

6

8

10

0 0 0 0 0 0

1

2

10

00 0 0 0 0 0 0

3

10

0

Num

ber

of R

espo

nses


2

4

6

8

10

0 0 0 0 0

1 1

3

8

00 0 0 0 0 0

1

2

10

0

Num

ber

of R

espo

nses

��


2

4

6

8

10

0 0

1

0 0

1

0

2

9

00 0 0 0

1

0

2

10

0

Num

ber

of R

espo

nses

Figure E�� This task was �� Confusing� � Clear� �� NA�Results for the single agent task �previous page top�� two agent task �previous pagebottom� and the four agent task �this page�

��


2

4

6

8

10

0 0 0 0 0 0

1

2

10

00 0 0 0 0

1 1

3

8

00 0

1

0 0

1

0

2

9

0

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


2

4

6

8

10

0 0 0 0 0 0 0

3

10

00 0 0 0 0 0

1

2

10

00 0 0 0

1

0

2

10

0

Num

ber

of R

espo

nses

Figure E�� This task was �� Confusing� � Clear� �� NA�Results for session one �top� and session two �bottom�

��


0.5

1

1.5

2

2.5

3

0

2

1

0

2 2 2 2 2

0

1 1 1 1

2 2 2

0

3

0

Session 1

Session 2

Num

ber

of R

espo

nses


0.5

1

1.5

2

2.5

3

0

1

0

1

2 2

3 3

1

00

1 1

2

3

0

2

1

3

0

Num

ber

of R

espo

nses

��


1

2

3

4

5

0 0 0 0

1 1

4

3

4

00 0

1

0

1

0

5

2

4

0

Num

ber

of R

espo

nses

Figure E�� This task was �� Dull� � Stimulating� �� NA�Results for the single agent task �previous page top�� two agent task �previous pagebottom� and the four agent task �this page�

��


1

2

3

4

0

2

1

0

2 2 2 2 2

00

1

0

1

2 2

3 3

1

00 0 0 0

1 1

4

3

4

0

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


1

2

3

4

5

1 1 1 1

2 2 2

0

3

00

1 1

2

3

0

2

1

3

00 0

1

0

1

0

5

2

4

0

Num

ber

of R

espo

nses

Figure E�� This task was �� Dull� � Stimulating� �� NA�Results for session one �top� and session two �bottom�

��


1

2

3

4

5

0

1

0 0 0 0

5

4

2

1

0

1 1

0

1 1

3

0

5

1

Session 1

Session 2

Num

ber

of R

espo

nses


1

2

3

4

5

0

1

0

1

0

2

3 3

2

11

0

1

0

1 1

3

0

5

1Num

ber

of R

espo

nses

��


1

2

3

4

5

0

1

0 0

1 1

4

2

3

1

0

2

1

0

1 1

0

2

5

1Num

ber

of R

espo

nses

Figure E�� This task was �� Frustrating� � Satisfying� �� NA�Results for the single agent task �previous page top�� two agent task �previous pagebottom� and the four agent task �this page�

��


1

2

3

4

5

0

1

0 0 0 0

5

4

2

1

0

1

0

1

0

2

3 3

2

1

0

1

0 0

1 1

4

2

3

1Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


1

2

3

4

5

0

1 1

0

1 1

3

0

5

11

0

1

0

1 1

3

0

5

1

0

2

1

0

1 1

0

2

5

1Num

ber

of R

espo

nses

Figure E�� This task was� �� Frustrating� � � Satisfying� � � NAResults for session one �top and session two �bottom

��


2

4

6

8

0 0 0 0 0 0

4

3

6

00 0 0 0 0 0 0

4

9

0

Session 1

Session 2

Num

ber

of R

espo

nses


2

4

6

8

0 0 0 0

1

0

2

7

3

00 0 0 0 0 0

1

3

9

0

Num

ber

of R

espo

nses

��


1

2

3

4

5

6

7

0 0 0 0

1 1

5

4

2

00 0

1

0 0 0

1

4

7

0

Num

ber

of R

espo

nses

Figure E�� I felt in control of the system during this task� �� Rarely� � � Always�� NAResults for the single agent task �previous page top� two agent task �previous pagebottom and the four agent task �this page

��


1

2

3

4

5

6

7

0 0 0 0 0 0

4

3

6

00 0 0 0

1

0

2

7

3

00 0 0 0

1 1

5

4

2

0

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


2

4

6

8

0 0 0 0 0 0 0

4

9

00 0 0 0 0 0

1

3

9

00 0

1

0 0 0

1

4

7

0

Num

ber

of R

espo

nses

Figure E�� I felt in control of the system during this task� �� Rarely� � � Always�� NAResults for session one �top and session two �bottom

�

1 2 3 4 5 6 7 8 9 10

2

4

6

8

10

0 0 0 0 0

2

0

2

9

00 0 0 0 0 0 0

3

10

0


Session 1

Session 2

Num

ber

of R

espo

nses


2

4

6

8

0 0 0 0 0

3

1

3

6

00 0 0 0 0 0

1

4

8

0

Num

ber

of R

espo

nses

��


1

2

3

4

5

6

7

0 0 0 0 0

1 1

4

7

00 0 0 0 0 0

2

4

7

0

Num

ber

of R

espo

nses

Figure E�� I was able to interpret and understand the data readings during thistask� �� Rarely� � � Always� � � NAResults for the single agent task �previous page top� two agent task �previous pagebottom and the four agent task �this page

��


2

4

6

8

0 0 0 0 0

2

0

2

9

00 0 0 0 0

3

1

3

6

00 0 0 0 0

1 1

4

7

0

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


1 2 3 4 5 6 7 8 9 10

2

4

6

8

10

0 0 0 0 0 0 0

3

10

00 0 0 0 0 0

1

4

8

00 0 0 0 0 0

2

4

7

0

Num

ber

of R

espo

nses

Figure E�� I was able to interpret and understand the data readings during thistask� �� Rarely� � � Always� � � NAResults for session one �top and session two �bottom

��


2

4

6

8

0 0 0 0 0

1

0

2

8

2

0 0 0 0 0 0 0

3

9

1

Session 1

Session 2

Num

ber

of R

espo

nses


2

4

6

8

0 0 0 0 0

1

2 2

7

1

0 0 0 0 0 0

2 2

8

1

Num

ber

of R

espo

nses

��


1

2

3

4

5

6

7

0 0

1

0 0

2

1

3

6

00 0

1

0 0 0

2 2

7

1

Num

ber

of R

espo

nses

Figure E�� I was able to correct my errors during this task� �� Rarely� � � Always�� NAResults for the single agent task �previous page top� two agent task �previous pagebottom and the four agent task �this page

��


2

4

6

8

0 0 0 0 0

1

0

2

8

2

0 0 0 0 0

1

2 2

7

1

0 0

1

0 0

2

1

3

6

0

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


2

4

6

8

0 0 0 0 0 0 0

3

9

1

0 0 0 0 0 0

2 2

8

1

0 0

1

0 0 0

2 2

7

1

Num

ber

of R

espo

nses

Figure E�� I was able to correct my errors during this task� �� Rarely� � � Always�� NAResults for session one �top and session two �bottom

��


2

4

6

8

10

0 0 0 0 0 0 0

5

8

00 0 0 0 0 0

1

2

10

0

Session 1

Session 2

Num

ber

of R

espo

nses


1

2

3

4

5

6

7

0 0 0 0 0 0

2

4

7

00 0 0 0 0 0 0

6

7

0

Num

ber

of R

espo

nses

��


2

4

6

8

0 0 0 0 0

1 1

5

6

00

1

0 0 0 0

1

3

8

0

Num

ber

of R

espo

nses

Figure E�� I felt able to complete the task� �� Rarely� � � Always� � � NAResults for the single agent task �previous page top� two agent task �previous pagebottom and the four agent task �this page

��


2

4

6

8

0 0 0 0 0 0 0

5

8

00 0 0 0 0 0

2

4

7

00 0 0 0 0

1 1

5

6

0

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


2

4

6

8

10

0 0 0 0 0 0

1

2

10

00 0 0 0 0 0 0

6

7

00

1

0 0 0 0

1

3

8

0

Num

ber

of R

espo

nses

Figure E�� I felt able to complete the task� �� Rarely� � � Always� � � NAResults for session one �top and session two �bottom

��


2

4

6

8

0 0 0 0

1

0 0

3

9

00 0 0 0 0 0

1

3

9

0

Session 1

Session 2

Num

ber

of R

espo

nses


1

2

3

4

5

6

7

0 0 0 0 0

2 2

4

5

00 0 0 0 0 0

2

4

7

0

Num

ber

of R

espo

nses

��


1

2

3

4

5

6

0 0 0

1

0

1

0

6

5

00 0

1

0 0 0

2

5 5

0

Num

ber

of R

espo

nses

Figure E�� I felt in control of the individual agents� �� Rarely� � � Always�� NAResults for the single agent task �previous page top� two agent task �previous pagebottom and the four agent task �this page

��


2

4

6

8

0 0 0 0

1

0 0

3

9

00 0 0 0 0

2 2

4

5

00 0 0

1

0

1

0

6

5

0

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


2

4

6

8

0 0 0 0 0 0

1

3

9

00 0 0 0 0 0

2

4

7

00 0

1

0 0 0

2

5 5

0

Num

ber

of R

espo

nses

Figure E�� I felt in control of the individual agents� �� Rarely� � � Always�� NAResults for session one �top and session two �bottom

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0 0 0

2

5

6

00 0 0 0 0 0

1

4

8

0


Session 1

Session 2

Num

ber

of R

espo

nses


1

2

3

4

5

6

7

0 0 0 0 0 0

4

3

6

00 0 0 0 0 0

2

4

7

0

Num

ber

of R

espo

nses

��


1

2

3

4

5

6

7

0 0 0 0

1 1 1

3

7

00 0 0 0 0

1 1

5

6

0

Num

ber

of R

espo

nses

Figure E�� The system�s capabilities for this task were� �� Inadequate power�� Adequate power� � � NAResults for the single agent task �previous page top� two agent task �previous pagebottom and the four agent task �this page

��


1

2

3

4

5

6

7

0 0 0 0 0 0

2

5

6

00 0 0 0 0 0

4

3

6

00 0 0 0

1 1 1

3

7

0

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


2

4

6

8

0 0 0 0 0 0

1

4

8

00 0 0 0 0 0

2

4

7

00 0 0 0 0

1 1

5

6

0

Num

ber

of R

espo

nses

Figure E�� The system�s capabilities for this task were� �� Inadequate power�� Adequate power� � � NAResults for session one �top and session two �bottom

��


1

2

3

4

5

6

7

0 0 0 0 0

3

1

4

5

00 0 0 0 0 0

3 3

7

0

Session 1

Session 2

Num

ber

of R

espo

nses


1

2

3

4

5

6

0 0 0 0 0

2

3

6

2

00 0 0 0 0 0

4

3

6

0

Num

ber

of R

espo

nses

��


1

2

3

4

0 0 0 0 0

3 3

2

4

1

0 0 0

1

0

2

4

2

4

0

Num

ber

of R

espo

nses

Figure E�� The system�s capabilities for this task were� �� Rigid� � � Flexible�� NAResults for the single agent task �previous page top� two agent task �previous pagebottom and the four agent task �this page

��


1

2

3

4

5

6

0 0 0 0 0

3

1

4

5

00 0 0 0 0

2

3

6

2

00 0 0 0 0

3 3

2

4

1

Num

ber

of R

espo

nses

Single Agent Task

Two Agent Task

Four Agent Task


1

2

3

4

5

6

7

0 0 0 0 0 0

3 3

7

00 0 0 0 0 0

4

3

6

00 0 0

1

0

2

4

2

4

0

Num

ber

of R

espo

nses

Figure E�� The system�s capabilities for this task were� �� Rigid� � � Flexible�� NAResults for session one �top and session two �bottom

��

Appendix F

Post�Experimental Questionnaire

F�� Quesstionnaire

Identi�cation Number�

Part �� Overall User Reactions

Please circle the choice which most accurately re�ects your experiences� NA � Not Ap�plicable�There is room on the last page for your written comments�

�� Overall reaction to the system�Terrible � � � � � � � � Wonderful NA

Frustrating � � � � � � � � Satisfying NADull � � � � � � � � Stimulating NA

Di�cult � � � � � � � � Easy NAInadequate power � � � � � � � � Adequate power NA

Rigid � � � � � � � � Flexible NA

Part �� Screen

�� Is screen window layout helpful�Not at all � � � � � � � � Very much NA

�� Is one main window helpful�Not at all � � � � � � � � Very much NA

�� The ability to turn on and o� data displays is�Frustrating � � � � � � � � Satisfying NA

Di�cult � � � � � � � � Easy NARigid � � � � � � � � Flexible NA

��

�� The use of command buttons is�Illogical � � � � � � � � Logical NA

Frustrating � � � � � � � � Satisfying NADi�cult � � � � � � � � Easy NA

�� The use of a phantom agent during teleoperation is�Unhelpful � � � � � � � � Helpful NAIllogical � � � � � � � � Logical NA

Frustrating � � � � � � � � Satisfying NA�� The amount of information which can be displayed on the screen is�Inadequate � � � � � � � � Adequate NA

�� The arrangement of information on the screen is�Illogical � � � � � � � � Logical NA

Part �� Terminology and System Information

�� The use of terms throughout the system is�Inconsistent � � � � � � � � Consistent NA

�� Control button labels are�Inconsistent � � � � � � � � Consistent NAConfusing � � � � � � � � Clear NA

�� Error messages which appear on the screen are�Inconsistent � � � � � � � � Consistent NA

�� Position of error messages on the screen is�Inconsistent � � � � � � � � Consistent NA

�� Error messages are�Confusing � � � � � � � � Clear NAUnhelpful � � � � � � � � Helpful NA

Hard to read � � � � � � � � Easy to read NA

�� Error messages clarify the problem�Never � � � � � � � � Always NA

�� Phrasing of error messages is�Unpleasant � � � � � � � � Pleasant NAConfusing � � � � � � � � Clear NAUnhelpful � � � � � � � � Helpful NA

�� Instructions for correcting errors are�Confusing � � � � � � � � Clear NA

��

Part �� Learning

�� Learning to operate the system was�Di�cult � � � � � � � � Easy NA

�� Getting started with the system was�Di�cult � � � � � � � � Easy NA

�� Time to learn the system was�Slow � � � � � � � � Fast NA

�� Remembering names and uses of command buttons was�Di�cult � � � � � � � � Easy NA

�� Remembering speci�c rules about entering commands was�Di�cult � � � � � � � � Easy NA

�� Tasks can be performed in a straight forward manner�Never � � � � � � � � Always NA

�� Number of steps per task were�To many � � � � � � � � Just right NA

�� Steps to complete a task follow a logical sequence�Rarely � � � � � � � � Always NA

Part �� System Capabilities

�� System speed is�Too slow � � � � � � � � Fast Enough NA

�� Response time for most operations is�Too slow � � � � � � � � Fast Enough NA

�� Rate information is displayed is�Too slow � � � � � � � � Fast Enough NA

�� System failures occur�Frequently � � � � � � � � Seldom NA

�� Novices can accomplish tasks after proper training�With di�culty � � � � � � � � Easily NA

��

Part �� User Assessments

Please circle a member of each pair which provided the most signi�cant source ofworkload variation in the system� The terms are de�ned below�

Temporal Demand vs� Frustration Temporal Demand vs� Mental DemandTemporal Demand vs� E�ort Own Performance vs� Mental DemandOwn Performance vs� Frustration Frustration vs� Mental DemandOwn Performance vs� E�ort E�ort vs� Mental DemandTemporal Demand vs� Own Performance E�ort vs� Frustration

Mental Demand�

How much mental and perceptual activity was required �e�g� thinking� deciding� cal�culating� remembering� looking and searching� etc�� Was the task easy or demanding�simple or complex� exacting or forgiving�

Temporal Demand�

How much time pressure dud you feel due to the rate or pace at which the task had tobe performed� Was the pace slow and leisurely or rapid and frantic�

Frustration�

How insecure� discouraged� irritated� stressed and annoyed versus secure� grati�ed� con�tent� relaxed and complacent did you feel during the task�

E�ort�

How hard did you have to work �mentally and physically to accomplish your level ofperformance�

Own Performance�

How successful do you think you were in accomplishing the goals of the task� How sat�is�ed were you with your performance in accomplishing this goal�

��

Part � User�s Comments

�� What did you like about this interface�

�� What did you dislike about this interface�

F�� Graphical Presentation of Results

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0 0 0

7

4

2

0


Num

ber

of R

espo

nses

Figure F�� Overall reaction to the system� �� Terrible� � � Wonderful� � � NA

��

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0

1 1

0

1 1

2

5

2

0


Num

ber

of R

espo

nses

Figure F�� Overall reaction to the system� �� Frustrating� � � Satisfying� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

0 0 0

1

0

1

3

4

3

1


Num

ber

of R

espo

nses

Figure F�� Overall reaction to the system� �� Dull� � � Stimulating� � � NA

��

1 2 3 4 5 6 7 8 9 10

0.5

1

1.5

2

2.5

3

0 0

1

0

1

2

3 3 3

0


Num

ber

of R

espo

nses

Figure F� � �� Overall reaction to the system� �� Di�cult� � � Easy� � � NA


1

2

3

4

5

6

0 0 0 0 0

1

3

6

3

0

Num

ber

of R

espo

nses

Figure F�� Overall reaction to the system� �� Inadequate power� � � Adequatepower� � � NA

��

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0 0 0

2

0

2

5

4

0


Num

ber

of R

espo

nses

Figure F�� Overall reaction to the system� �� Rigid� � � Flexible� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0

1

0

1

3

1

2

5

0


Num

ber

of R

espo

nses

Figure F�� Is screen window layout helpful� �� Not at all� � � Very much� � � NA

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0 0

1

2

1

9

0


Num

ber

of R

espo

nses

Figure F�� Is one main window helpful� �� Not at all� � � Very much� � � NA

1 2 3 4 5 6 7 8 9 10 11

1

2

3

4

5

6

1

0 0 0 0

1

2

1

2

6

0


Num

ber

of R

espo

nses

Figure F�� The ability to turn on and o� data displays is� �� Frustrating�� Satisfying� � � NA

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0

1

0

3

1

8

0


Num

ber

of R

espo

nses

Figure F�� The ability to turn on and o� data displays is� �� Di�cult� � � Easy�� NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0 0

1 1

3

1

2

5

0


Num

ber

of R

espo

nses

Figure F�� The ability to turn on and o� data displays is� �� Rigid� � � Flexible�� NA

��

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0 0

1

2

3

7

0


Num

ber

of R

espo

nses

Figure F�� The use of command buttons is� �� Illogical� � � Logical� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

0 0 0 0

2

3 3

1

4

0


Num

ber

of R

espo

nses

Figure F�� The use of command buttons is� �� Frustrating� � � Satisfying�� NA

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0 0

2

1 1

9

0


Num

ber

of R

espo

nses

Figure F�� The use of command buttons is� �� Di�cult� � � Easy� � � NA

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0 0 0

2 2

9

0


Num

ber

of R

espo

nses

Figure F�� The use of the phantom agent during teleoperation is� �� Unhelpful�� Helpful� � � NA

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0 0 0

1

3

9

0


Num

ber

of R

espo

nses

Figure F�� The use of the phantom agent during teleoperation is� �� Illogical�� Logical� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0

1

0

1

3

7

1


Num

ber

of R

espo

nses

Figure F�� The use of the phantom agent during teleoperation is� �� Frustrating�� Satisfying� � � NA

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0 0 0

2

3

8

0


Num

ber

of R

espo

nses

Figure F�� The amount of information which can be displayed on the screen is�� Inadequate� � � Adequate� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

0 0

1 1

0

3

2 2

4

0


Num

ber

of R

espo

nses

Figure F�� The arrangement of information on the screen is� �� Illogical�� Logical� � � NA

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

10

0 0 0 0 0 0

2

1

10

0


Num

ber

of R

espo

nses

Figure F�� The use of terms throughout the system is� �� Inconsistent�� Consistent� � � NA

1 2 3 4 5 6 7 8 9 10

2

4

6

8

10

0 0 0 0 0 0

2

1

10

0


Num

ber

of R

espo

nses

Figure F�� Control button labels are� �� Inconsistent� � � Consistent� � � NA

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0

1

0

3

1

8

0


Num

ber

of R

espo

nses

Figure F�� Control button labels are� �� Confusing� � � Clear� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

0 0 0 0 0 0

2

1

6

4


Num

ber

of R

espo

nses

Figure F�� Error messages which appear on the screen are� �� Inconsistent�� Consistent� � � NA

��

1 2 3 4 5 6 7 8 9 10

2

4

6

8

0 0 0 0 0 0 0 0

8

5


Num

ber

of R

espo

nses

Figure F�� Position of error messages on the screen are� �� Inconsistent�� Consistent� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0 0 0 0

1

0

3

5

4


Num

ber

of R

espo

nses

Figure F�� Error messages are� �� Confusing� � � Clear� � � NA

��

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0 0 0 0

2

7

4


Num

ber

of R

espo

nses

Figure F�� Error messages are� �� Unhelpful� � � Helpful� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0 0 0 0 0 0

4

5

4


Num

ber

of R

espo

nses

Figure F�� Error messages are� �� Hard to read� � � Easy to read� � � NA

�

1 2 3 4 5 6 7 8 9 10

1

2

3

4

0 0 0 0 0 0

2

4

3

4


Num

ber

of R

espo

nses

Figure F�� Error messages clarify the problem� �� Never� � � Always� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0 0 0

1

0

1

3 3

5


Num

ber

of R

espo

nses

Figure F�� Phrasing of error messages is� �� Unpleasant� � � Pleasant� � � NA

� �

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0 0 0 0 0

2

3 3

5


Num

ber

of R

espo

nses

Figure F�� Phrasing of error messages is� �� Confusing� � � Clear� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0 0 0 0 0

1

4

3

5


Num

ber

of R

espo

nses

Figure F�� Phrasing of error messages is� �� Unhelpful� � � Helpful� � � NA

� �

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0 0

3

1

0

2

7


Num

ber

of R

espo

nses

Figure F�� Instructions for correcting errors are� �� Confusing� � � Clear�� NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

0 0 0 0 0

1

2

4

6

0


Num

ber

of R

espo

nses

Figure F�� Learning to operate the system was� �� Di�cult� � � Easy� � � NA

� �

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0 0

1 1

0

2

4

5

0


Num

ber

of R

espo

nses

Figure F�� Getting started with the system was� �� Di�cult� � � Easy� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

0 0 0 0 0

1 1

5

6

0


Num

ber

of R

espo

nses

Figure F�� Time to learn the system was� �� Slow� � � Fast� � � NA

�

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

0 0 0 0

1 1

0

5

6

0


Num

ber

of R

espo

nses

Figure F�� Remembering names and uses if command buttons was� �� Di�cult�� Easy� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

0 0 0 0

1 1

3

2

6

0


Num

ber

of R

espo

nses

Figure F�� Remembering speci�c rules about entering data was� �� Di�cult�� Easy� � � NA

� �

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0 0

2 2 2

7

0


Num

ber

of R

espo

nses

Figure F�� Tasks can be performed in a straight forward manner� �� Never�� Always� � � NA

1 2 3 4 5 6 7 8 9 10

0.5

1

1.5

2

2.5

3

0 0

1 1

3

1

2

3

2

0


Num

ber

of R

espo

nses

Figure F�� Number of steps per task were� �� Too many� � � Just right� � � NA

� �

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0 0 0

1

5

7

0


Num

ber

of R

espo

nses

Figure F� � �� Steps to complete a task follow a logical sequence� �� Rarely�� Always� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0

1 1

5

0

2

3

1

0 0


Num

ber

of R

espo

nses

Figure F� �� System speed was� �� Too slow� � � Fast enough� � � NA

� �

1 2 3 4 5 6 7 8 9 10

1

2

3

4

0

1 1

2

4

0

1

3

1

0


Num

ber

of R

espo

nses

Figure F� �� Response time for most operations is� �� Too slow� � � Fast enough�� NA

1 2 3 4 5 6 7 8 9 10

0.5

1

1.5

2

2.5

3

0 0

2 2

3

2 2

0

2

0


Num

ber

of R

espo

nses

Figure F� �� Rate information is displayed is� �� Too slow� � � Fast enough�� NA

� �

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

0 0

2 2 2 2

5

0 0 0


Num

ber

of R

espo

nses

Figure F� � �� System failures occur� �� Frequently� � � Seldom� � � NA

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

0 0 0 0 0

1

2

3

7

0


Num

ber

of R

espo

nses

Figure F� �� Novices can accomplish tasks after proper training� �� With di�culty�� Easily� � � NA

� �

Bibliography

�apa� �� Publication Manual of the American Psychological Association�American Psycholocical Association� Washington� D�C�� fourth edition�

�Adams et al�� Adams� J� et al� �� Cooperative material handling by humanand robotic agents� Module development and system synthesis� In IEEE�RJS Inter�national Conference on Intelligent Robots and Systems� IEEE�RJS�

�Adams and Paul� �� a� Adams� J� A� and Paul� R� �� a� Human�managed� hierar�chical control of multiple mobile agents� In Proceedings of the ��rd IEEE Conference

on Decision and Control� pages �� IEEE�

�Adams and Paul� �� b� Adams� J� A� and Paul� R� �� b� Human management ofa hierarchical control system for multiple mobile robots� In Proceedings of the ��

IEEE International Conference on Systems� Man and Cybernetics� pages �� IEEE�

�Adams and Paul� �� Adams� J� A� and Paul� R� �� Human supervisory controlof multiple mobile robots� In Proceedings of the �� IEEE International Conference

on Systems� Man and Cybernetics� IEEE�

�Ammons� �� Ammons� J� �� Scheduling models for aiding real time FMS con�trol� In Proceedings of the International Conference on Cybernetics and Society� pages�� IEEE�

�Askew and Diftler� �� Askew� R� and Diftler� M� �� Ground control testbed forspace station freedom robot manipulation� In Proceedings of the IEEE Annual VirtualReality International Symposium� pages �� IEEE�

�Bach� �� Bach� C� �� A customizable direct manipulation user interface withautomatic generation of help information� In Bullinger� H��J�� editor� Advances inHuman Factors�Ergonomics Human Aspects in Computing � Design and Use of In�

teractive Systems and Information Management� pages �� Elsevier� New York�v� ��B�

�Bajcsy et al�� Bajcsy� R�� Kumar� V�� Mintz� M�� Paul� R�� and Yun� X� ��A small�team architecture for multiagent robotic systems� In Workshop on IntelligentRobotic Systems Design and Applications� SPIEs Intelligent Robotics Symposium�Boston� MA�

��

�Bar�eld and Kim� ��a� Bar�eld� W� and Kim� Y� ��a� Computer graphics pro�gramming principals as factors in the design of perspective displays� In Bullinger�H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects in Comput�ing � Design and Use of Interactive Systems and Work with Terminals�� pages �� Elsevier� New York� v� ��A�

�Bar�eld and Kim� ��b� Bar�eld� W� and Kim� Y� ��b� Evaluation of computergraphics techniques for the design of images for human�computer interaction� InBullinger� H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects

in Computing � Design and Use of Interactive Systems and Work with Terminals��pages �� Elsevier� New York� v� ��A�

�Baron� �� Baron� S� �� A control theoretic approach to modeling human super�visory control of dynamic systems� In Rouse� W�� editor� Advances in Man�Machine

Systems Research� chapter �� pages � � �� JAI Press Inc�� London� England� v� ��

�Becker� �� Becker� C� �� Internet Path Planner Server Protocol� Stanford Uni�versity�

�Bejczy et al�� Bejczy� A�� Kim� W�� and Venema� S� �� The phantom robot�Predictive displays for teleoperation with time delay� In Proceedings of the IEEE

International Conference on Robotics and Automation� pages � � � �� IEEE�

�Bellingham and Consi� �� Bellingham� J� G� and Consi� T� R� �� State con�g�ured layered control� In Proceedings of the International Advanced Robotics Program�st Workshop on Mobile Robots for Subsea Environments� pages ��

�Bellingham and Humphrey� �� Bellingham� J� G� and Humphrey� D� �� Usinglayered control for supervisory control of underwater vehicles� In Proceedings of theConference on Remotely Operated Vehicles� pages ��

�Bevan et al�� Bevan� N�� Kirakowski� J�� and Maissel� J� �� What is usability�In Bullinger� H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects

in Computing � Design and Use of Interactive Systems and Work with Terminals��pages �� Elsevier� New York� v� ��A�

�Bodker� �� Bodker� S� �� Through the Interface A Human Activity Approach toUser Interface Design� Lawrence Erlbaum Associates� Publishers� Hillsdale�N�J�

�Bogoni� �� Bogoni� L� �� Subsumption architecture and discrete event systems�A comparison� Technical Report MS�CIS�� Grasp LAB �� University of Penn�sylvania�

�Borys� �� Borys� B��B� �� Ways of supporting ergonomically and technicallycorrect display design� In Weir� G� R� S� and Alty� J� L�� editors� Computers and

People Series Human�Computer Interaction and Complex Systems� chapter �� pages�� Academic Press� New York�

�B�osser and Melchior� �� B�osser� T� and Melchior� E��M� �� User centered de�sign� prototyping� and cognitive modeling with the sane toolkit� In ESPRIT ��

Conference Proceedings� pages ��

��

�Brandt� �� Brandt� D� �� Research strategies for human�centered design ofhuman�machine systems� In Proceedings of the � th World Congress� International

Federation of Automatic Control� pages �� v� ��

�Brooks� �� Brooks� R� A� �� A robust layered control system for a mobile robot�IEEE Journal of Robotics and Automation� RA��

�Brooks� �� Brooks� R� A� �� A hardware retargetable distributed layered archi�tecture for mobile robot control� In Proceedings of the IEEE International Conferenceon Robotics and Automation� pages ��

�Chen and Trivedi� �� Chen� C� and Trivedi� M� �� Simulation and animationof sensor�driven robots� IEEE Transactions of Robotics and Automation� ��

�Christensen� �� Christensen� B� �� Virtual environments for telerobotic sharedcontrol� In Proceedings SPIE � The International Society for Optical Engineering�pages � � �� SPIE� v� ��

�Christensen et al�� Christensen� B�� Greibenow� B�� and Burks� B� �� Graphicmodel based control of robotic systems for waste remediation� Transactions of the

American Nuclear Society� ��

�Cooke and Stans�eld� �� Cooke� C� and Stans�eld� S� �� Interactive graphicalmodel building using telepresence and virtual reality� In Proceedings of the IEEEInternational Conference on Robotics and Automation� pages � ��

�Cox and Walker� �� Cox� K� and Walker� D� �� User Interface Design� PrenticeHall� New York� second edition�

�Dai� �� Dai� F� �� Centralized� application�oriented graphical interaction controlusing an example of planning robotic tasks� Computers and Graphics� ��

�Drotning et al�� Drotning� W�� Christensen� B�� and Thunborg� S� �� Graphi�cal model based control of intelligent robot systems� IEEE Control Systems Magazine��

�Durand� �� Durand� J��P� �� Automated informational �ows� Why human in�tervention is still necessary� In Proceedings of the � th World Congress� International

Federation of Automatic Control� pages �� v� ��

�Eberts� �� Eberts� R� �� User Interface Design� Prentice Hall� Englewood Cli�s�N�J�

�Edmonds� �� Edmonds� E� �� The man�computer interface� A note on conceptsand design �� In Edmonds� E� A�� editor� Computers and People Series TheSeparable User Interface� chapter �� pages �� Academic Press� New York�

�Eichelberg and Ackermann� �� Eichelberg� D� and Ackermann� P� �� Integrat�ing interactive �D�graphics into an object�oriented application framework� In Proceed�

ings of the Vienna Conference on Human Computer Interaction� pages � � ��

��

�Ellis� �� Ellis� S� R� �� Nature and origins of virtual environments� A biblio�graphical essay� Computing Systems in Engineering� ��

�Ephrati and Rosenschein� �� Ephrati� E� and Rosenschein� J� �� Divide andconquer in multi�agent planning� In Proceedings of the � th National Conference onArti�cial Intelligence� pages ��

�Fleury et al�� Fleury� S�� Herrb� M�� and Chatila� R� �� Design of a mod�ular architecture for autonomous robot� In Proceedings of the IEEE International

Conference on Robotics and Automation� pages ��

�Foley et al�� Foley� J�� van Dam� A�� Feiner� S�� and Hughes� J� �� ComputerGraphics Principles and Practice� Addison�Wesley Publishing Co�� London�

�Frey et al�� Frey� P� R�� Rouse� W�� and Garris� R� D� �� Big graphics andlittle screens� Model�based design of large�scale information displays� In Rouse� W��editor� Human�Technology Interaction in Complex Systems� pages � � �� JAI PressInc�� London�

�Gimnich et al�� Gimnich� R�� Kunkel� K�� and Reichert� L� �� A usabilityengineering approach to the development of graphical user interfaces� In Bullinger�H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects in Computing� Design and Use of Interactive Systems and Work with Terminals�� pages �� Elsevier� New York� v� ��A�

�Grant and Mayes� �� Grant� S� and Mayes� T� �� Cognitive task analysis�In Weir� G� R� S� and Alty� J� L�� editors� Computers and People Series Human�

Computer Interaction and Complex Systems� chapter �� pages � � � �� AcademicPress� New York�

�Gunsth�ovel and B�osser� �� Gunsth�ovel� D� and B�osser� T� �� Predictive metricsfor usability� In Bullinger� H��J�� editor� Advances in Human Factors�Ergonomics

Human Aspects in Computing � Design and Use of Interactive Systems and Work withTerminals�� pages �� Elsevier� New York� v� ��A�

�Hahndel and Levi� �� Hahndel� S� and Levi� P� �� A distributed task planningmethod for autonomous agents in a FMS� In Proceedings of the IEEE�RSJ Interna�

tional Conference on Intelligent Robots and Systems� pages ��

�Hancke and Braune� �� Hancke� T� and Braune� R� �� Human�centered designof human�machine systems and examples from air transport� In Proceedings of the � thWorld Congress� International Federation of Automatic Control� pages � � � � �� v��

�Hartley and Pipitone� �� Hartley� R� and Pipitone� F� �� Experiments with thesubsumption architecture� In Proceedings of the IEEE International Conference on

Robotics and Automation� pages ��

��

�Hasegawa et al�� Hasegawa� T�� Suehiro� T�� Ogasawara� T�� Matsui� T�� Kitagaki�K�� and Takase� K� �� An integrated tele�robotics system with a geometric en�vironment model and manipulation skills� In Proceedings of the IEEE InternationalWorkshop on Intelligent Robots and Systems� pages �� IEEE�

�Heger et al�� Heger� A� S�� Duran� F�� Frysinger� S�� and Cox� R� �� Treatmentof human�computer interface in a decision support system� In Proceedings of the IEEEInternational Conference on Systems� Man� and Cybernetics� pages �� IEEE�

�Heinecke� �� Heinecke� A� M� �� Software ergonomics for real�time systems� InProceedings of the Vienna Conference on Human Computer Interaction� pages ��

�Herot� �� Herot� C� F� �� Graphical user interfaces� In Vassiliou� Y�� editor�Human Factors and Interactive Computer Systems� chapter � pages �� AblexPublishing Company� Norwood� N�J�

�Hill et al�� Hill� S�� Iavecchia� H�� Byers� J�� Bittner� A�� Zakland� A�� and Christ�R� �� Comparison of four subjective workload rating scales� Human Factors��

�Hirai and Sato� �� Hirai� S� and Sato� T� �� Multi�level manual and autonomouscontrol superposition for intelligent telerobot� In Proceedings of the NASA Conference

on Space Telerobotics� pages ��

�Hirai et al�� Hirai� S�� Sato� T�� and Matsui� T� �� Intelligent and cooperativecontrol of telerobot tasks� IEEE Control Systems� ��

�Hix and Hartson� �� Hix� D� and Hartson� H� �� Developing User Interfaces

Ensuring Usability Through Product � Process� John Wiley � Sons� Inc�� New York�

�Hodges et al�� Hodges� L� F�� Bolter� J�� Mynatt� E�� Ribarsky� W�� and van Teylin�gen� R� �� Lab review� Virtual environments research at the Georgia Tech GVUcenter� Presence� ��

�Hollan et al�� Hollan� J� D�� Hutchins� E� L�� McCandless� T� P�� Rosenstein� M��and Weitzman� L� �� Graphic interfaces for simulation� In Rouse� W� B�� editor�Advances in Man�Machine Systems Research� pages �� JAI Press Inc�� London�v� ��

�Hutchins et al�� Hutchins� E�� Hollan� J�� and Norman� D� �� Direct ma�nipulation interfaces� In Norman� D� and Draper� S�� editors� User Centered SystemDesign New Perspectives in Human�Computer Interaction� Erlbaum� Hillsdale� N�J�

�Hwang and Wang� �� Hwang� S��L� and Wang� Y��S� �� An experimental studyof CRT graphical display in process control systems� In Bullinger� H��J�� editor� Ad�vances in Human Factors�Ergonomics Human Aspects in Computing � Design andUse of Interactive Systems and Work with Terminals�� pages � � � � Elsevier� NewYork� v� ��A�

��

�Jacob� �� Jacob� R� J� �� A speci�cation language for direct�manipulation userinterfaces� ACM Transactions on Graphics� ��

�Jacob� �� Jacob� R� J� �� Direct manipulation in the intelligent interface� InHancock� P� and Chignell� M�� editors� Intelligent Interfaces Theory� Research andDesign� pages �� Elsevier� New York�

�Johannsen� �� Johannsen� G� �� Knowledge based design of human�machineinterfaces� In Proceedings of the � th World Congress International Federation ofAutomatic Control� IFAC�

�Keil�Slawik et al�� Keil�Slawik� R�� Plaisant� C�� and Shneiderman� B� �� Re�mote direct manipulation� A case study of a telemedicine workstation� In Bullinger�H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects in Computing

� Design and Use of Interactive Systems and Information Management� pages �� Elsevier� New York� v� ��B�

�Kelley� �� Kelley� C� �� Manual and Automatic Control � A Theory of ManualControl and its Applications to Manual and to Automatic Control� John Wiley andSons� Inc�� New York�

�Kirlik et al�� Kirlik� A�� Miller� R� A�� and Jagacinski� R� �� Supervisorycontrol in a dynamic and uncertain environment� A process model of skilled human�environment interaction� IEEE Transactions on Systems� Man� and Cybernetics��

�Kosecka� �� Kosecka� J� �In Progress� �� Discrete Event Systems for AutonomousMobile Agents� PhD thesis� University of Pennsylvania�

�Kraupner�Stadler� �� Kraupner�Stadler� H��C� �� Fundamentals of the useof colors in user interfaces� In Bullinger� H��J�� editor� Advances in Human Fac�

tors�Ergonomics Human Aspects in Computing � Design and Use of Interactive Sys�tems and Work with Terminals�� pages �� Elsevier� New York� v� ��A�

�Latombe� �� Latombe� J��C� �� Robot Motion Planning� Kluwer AcademicPublishers� Boston�

�Maher and Bell� �� Maher� P� K� C� and Bell� H� V� �� The man�machine in�terface � a new approach �� In Edmonds� E� A�� editor� Computers and People

Series The Separable User Interface� chapter �� pages �� Academic Press� NewYork�

�Mandelbaum� �� Mandelbaum� R� �� Sensor Fusion for Mobile Robot Localiza�

tion� Exploration and Navigation� PhD thesis� University of Pennsylvania�

�Martinengo et al�� Martinengo� A�� Campani� M�� and Torre� V� �� Com�plex tasks and control strategies of robots� In Proceedings of the IEEE International

Conference on Robotics and Automation� pages ��

��

�Mataric� �� Mataric� M� J� �� Integration of representation into goal�drivenbehavior�based robots� IEEE Transactions on Robotics and Automation� ��

�Matsui and Tsukamoto� �� Matsui� T� and Tsukamoto� M� �� Integrated robotteleoperation method using multi�media display� In Proceedings of the �th InternationalSymposium on Robotics Research� pages � � � ��

�McKee and Wolfsberger� �� McKee� J� W� and Wolfsberger� J� �� A graphical�rule based robotic interface system� In Proceedings of the Fourth Conference on Arti��

cial Intelligence for Space Applications� pages �� NASA Conference Publicationnumber ��

�Misue and Sugiyama� �� Misue� K� and Sugiyama� K� �� Multi�viewpoint per�spective display methods� Formulation and application to compound graphs� InBullinger� H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects

in Computing � Design and Use of Interactive Systems and Information Management�pages �� Elsevier� New York� v� ��B�

�Morris and Rouse� �� Morris� N� M� and Rouse� W� B� �� Human operatorresponse to error�likely situations in complex engineering systems� In Rouse� W� B��editor� Human�Technology Interaction in Complex Systems� pages �� JAI PressInc�� London� v� ��

�Muckler and Seven� �� Muckler� F� and Seven� S� �� Selecting performance mea�sures� Objective versus subjective measurement� Human Factors� � � � � � ��

�Nakamura� �� Nakamura� N� �� Human modifying knowledge acquisition forFMS scheduling� In Karwowsky� W� and Rahimi� M�� editors� Ergonomics of HybridAutomated Systems II� page �� Elsevier� New York�

�Nakamura� �� Nakamura� N� �� A human�supervised control architecture for a�exible manufacturing system� In Bullinger� H��J�� editor� Advances in Human Fac�tors�Ergonomics Human Aspects in Computing � Design and Use of Interactive Sys�

tems and Information Management� pages � � � �� Elsevier� New York� v� ��B�

�Ntuen et al�� Ntuen� C� A�� Park� E�� Wang� Y��M�� and Byrd� W� P� ��The TOP architecture for multiagent task planning and scheduling� Computers andIndustrial Engineering� ��

�Nygren� �� Nygren� T� �� Psychometric properties of subjective workload mea�surement techniques� Implications for their use in the assessment of perceived mentalworkload� Human Factors� ��

�Pejtersen and Nielsen� �� Pejtersen� A� and Nielsen� F� �� Iconic interface forinteractive �ction retrieval in libraries based on a cognitive task analysis� In Bullinger�H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects in Computing� Design and Use of Interactive Systems and Work with Terminals�� pages �� Elsevier� New York� v� ��A�

��

�Preece et al�� Preece� J�� Rogers� Y�� Sharp� H�� Benyon� D�� Holland� S�� andCarey� T� �� Human�Computer Interaction� Addison�Wesley� New York�

�Prothero� �� Prothero� J� �� A survey of interface goodness measures� Tech�nical Report HITL R�� Human Interface Technology Laboratory� University ofWashington�

�Ramadge and Wonham� �� Ramadge� P� J� and Wonham� W� M� �� The controlof discrete event systems� Proceedings of the IEEE� ��

�Rasmussen� �� Rasmussen� J� �� Strategies for state identi�cation and diagnosisin supervisory control tasks� and design of computer�based support systems� In Rouse�W� B�� editor� Advances in Man�Machine Systems Research� chapter �� pages �� JAI Press Inc�� London� England� v� ��

�Ravden and Johnson� �� Ravden� S� and Johnson� G� �� Evaluating Usability ofHuman�Computer Interfaces A Practical Method� Ellis Horwood Limited� Chichester�England�

�Regan and Pose� �� Regan� M� and Pose� R� �� An interactive graphics displayarchitecture� In Proceedings of IEEE Annual Virtual Reality International Symposium�pages �� IEEE�

�Rencken and Durrant�Whyte� �� Rencken� W� and Durrant�Whyte� H� �� Aquantitative model for adaptive task allocation in human�computer interfaces� IEEETransactions on Systems� Man� and Cybernetics� ��

�Rengger� �� Rengger� R� �� Indicators of usability based on performance� InBullinger� H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects in

Computing � Design and Use of Interactive Systems and Work with Terminals�� pages�� Elsevier� New York� v� ��A�

�Rocha and Ramos� �� Rocha� J� and Ramos� C� �� Task planning for �exibleand agile manufacturing systems� In Proceedings of the IEEE�RSJ International Con�ference on Intelligent Robots and Systems� pages ��

�Rouse� �� Rouse� W� B� �� Human�computer interaction in the control of dy�namic systems� ACM Computing Surveys� ��

�Rouse and Hunt� �� Rouse� W� B� and Hunt� R� M� �� Human problem solvingin fault diagnosis tasks� In Rouse� W� B�� editor� Advances in Man�Machine Systems

Research� chapter � pages �� JAI Press Inc�� London� England� v� ��

�Sato and Hirai� �� Sato� T� and Hirai� S� �� MEISTER� A model enhancedintelligent and skillful teleoperational robot system� In Bolles� R� and Roth� B�� editors�Proceedings of the fourth International Symposium on Robotics Research� pages ��

�Sayers� �� Sayers� C� �� PENGUIN� PENN Graphical User Interface� Universityof Pennsylvania� Personal Communication�

��

�Sayers and Paul� �� Sayers� C� and Paul� R� �� An operator interface for telepro�gramming employing synthetic �xtures� Presence� ��

�Schneider� �� Schneider� M� L� �� Ergonomic considerations in the design ofcommand languages� In Vassiliou� Y�� editor� Human Factors and Interactive ComputerSystems� chapter �� pages � � � �� Ablex Publishing Corp�� Norwood� N�J�

�Sheridan� �� Sheridan� T� �� Human supervisory control of robot systems� InProceedings of the IEEE International Conference on Robotics and Automation� pages��

�Sheridan� �� Sheridan� T� �� Telerobotics� Automation� and Human SupervisoryControl� MIT Press� Cambridge� Mass�

�Shneiderman� �� Shneiderman� B� �� Direct manipulation� A step beyond pro�gramming languages� IEEE Computer� ��

�Shneiderman� �� Shneiderman� B� �� The future of interactive systems and theemergence of direct manipulation� In Vassiliou� Y�� editor� Human Factors and Inter�

active Computer Systems� chapter �� pages � � �� Ablex Publishing Corp�� Norwood�N�J�

�Shneiderman� �� Shneiderman� B� �� Designing the User Interface Strategies

for E�ective Human�Computer Interaction� Addison�Wesley� New York�

�Stans�eld� �� Stans�eld� S� �� Lab review� A distributed virtual reality simu�lation system for situational training� Presence� ��

�Stein� �� Stein� M� �� Behavior�Based Control for Time�Delayed Teleoperation�PhD thesis� University of Pennsylvania� GRASP Laboratory�

�Stramler Jr�� Stramler Jr�� J� �� The Dictionary for Human Fac�

tors�Ergonomics� CRC Press� London�

�Suzuki et al�� Suzuki� T�� Yokota� K�� Asama� H�� Kaetsu� H�� and Endo� I� ��Cooperation between the human operator and the multi�agent robotic system� Eval�uation of agent monitoring methods for the human interface system� In Proceedingsof the International Conference on Intelligent Robots and Systems� pages �� IEEE�RSJ�

�Suzuki et al�� Suzuki� T�� Yokota� K�� Asama� H�� Matsumoto� A�� Ishida� Y��Kaetsu� H�� and Endo� I� �� A human interface for interacting with and monitoringthe multi agent robotic system� In Proceedings of the nd International Symposium on

Distributed Autonomous Robotic Systems� pages �� IEEE�RSJ�JSME�SICE�

�Taipale and Hirai� �� Taipale� T� and Hirai� S� �� A behavior�based controlsystem applied over multi�robot system� In Proceedings of the IEEE�RSJ InternationalConference on Intelligent Robots and Systems� pages ��

��

�Tarn et al�� Tarn� T��J�� Bejczy� A� K�� Guo� C�� and Xi� N� �� Intelligentplanning and control for telerobotic operations� In Proceedings of the IEEE�RSJ

International Conference on Intelligent Robots and Systems� pages ��

�Thomas and Goss� �� Thomas� P� and Goss� K� �� dimensional visual repre�sentations for graphical user interfaces� The art of user�involvement� In Proceedings

of the Vienna Conference on Human Computer Interaction� pages ��

�Virzi� �� Virzi� R� �� Re�ning the test phase of usability evaluation� How manysubjects is enough� Human Factors� � � � ��

�Wang� �� Wang� C��C� �� Local path planner� University of Pennsylvania�Personal Communication�

�Wang� �� Wang� C��C� �� The Performance of Repeatable Control Schemes forRedundant Manipulators� PhD thesis� University of Pennsylvania� GRASP Laboratory�

�Wang and Kumar� �� Wang� C��C� and Kumar� V� �� The performance of re�peatable control schemes for redundant robots� Submitted to IEEE Transactions onRobotics and Automation�

�Weir� �� Weir� G� R� S� �� Living with complex interactive systems� In Weir�G� R� S� and Alty� J� L�� editors� Computers and People Series Human�ComputerInteraction and Complex Systems� chapter �� pages � � �� Academic Press� NewYork�

�Whalley� �� Whalley� S� �� The human computer interface � designing for pro�cess control� In Proceedings of the IEEE International Conference on Systems� Man�and Cybernetics� pages �� IEEE�

�Whiteside et al�� Whiteside� J�� Wixon� D�� and Jones� S� �� User perfor�mance with command� menu� and iconic interfaces� In Hartson� H� R� and Hix� D��editors� Advances in Human�Computer Interaction� chapter �� pages �� AblexPublishing Corporation� Norwood� N�J� v� ��

�Widdel and Kaster� �� Widdel� H� and Kaster� J� �� Comparison of interactiontechniques� In Bullinger� H��J�� editor� Advances in Human Factors�Ergonomics Hu�man Aspects in Computing � Design and Use of Interactive Systems and Work with

Terminals�� pages � � � �� Elsevier� New York� v� ��A�

�Wietho� et al�� Wietho�� M�� Arnold� A�� and Houwing� E� �� The valueof psychophysiological measures in human�computer interaction� In Bullinger� H��J�� editor� Advances in Human Factors�Ergonomics Human Aspects in Computing �Design and Use of Interactive Systems and Work with Terminals�� pages �� Elsevier� New York� v� ��A�

�Woods� �� Woods� D� D� �� The cognitive engineering of problem represen�tations� In Weir� G� R� S� and Alty� J� L�� editors� Computers and People SeriesHuman�Computer Interaction and Complex Systems� chapter �� pages �� Aca�demic Press� New York�

��

�Yamamoto� �� Yamamoto� Y� �� Control and Coordination of Locomotion andManipulation for a Wheeled Mobile Manipulator� PhD thesis� University of Pennsyl�vania� GRASP Laboratory�

�Yamamoto and Yun� �� Yamamoto� Y� and Yun� X� �� Coordinating locomo�tion and manipulation of a mobile manipulator� In Proceedings of the ��st Conferenceon Decision and Control� pages �� IEEE�

�Yamamoto and Yun� �� Yamamoto� Y� and Yun� X� �� Coordinating locomo�tion and manipulation of a mobile manipulator� IEEE Transactions on AutomaticControl� ��

�Yokota et al�� Yokota� K�� Suzuki� T�� Asama� H�� Matsumoto� A�� and Endo� I�� A human interface system for the multi�agent robotic system� In Proceedings

of the IEEE International Conference on Robotics and Automation� pages ��

�Zyda et al�� Zyda� M�� Pratt� D�� Falby� J�� Barham� P�� and Kelleher� K� ��Lab review� NPSNET and the naval postgraduate school graphics and video labora�tory� Presence� ��

��

Documents

jaa/Papers/dissertation.pdf · HUMAN MANA GEMENT OF A HIERAR CHICAL SYSTEM F OR THE CONTR OL OF MUL TIPLE MOBILE R OBOTS JULIE A AD AMS A DISSER T A TION in COMPUTER AND INF ORMA