Case of Study in Human Factors Evaluation (1).pdf

7/28/2019 Case of Study in Human Factors Evaluation (1).pdf

1/9

C ase s tud y in hum an fac tors eva luat ionA W h i t e f ie l d a n d A S u t c l i ff e *

A human f ac t o r s ( H F) e va lua ti on , c a r r ie d ou t a s par t o f t hede v e l opme n t o f a s e t o f c ompu t e r - a i de d s o f twar e e ng i nee r i ng( C AS E) t oo ls , i s p r e s e n t e d and is u s e d a s an e x amp l e o f t hepr oc e s se s and p r oduc t s o f t y p i c a l H F e v a l uat i on p r ac t ic e . Ther o le o f H F e v a lua t ion a s a par t o f s o f t war e qua l i t y a s s uranc e i sident if ied , and typ ical current pract ice o f H F evaluat ion i s char-acter ized . T he deta i l s o f the par t icular evaluat ion are thenreported. First, i ts proc esses are described; the se are deter min edby r e l a t ing f e a t u r e s o f t he s y s t e m unde r de v e l opme n t t o t hede s i r e d oc us , ac t ua l c on t e x t , a nd pos s ib l e me t hods o f t he e va lu -at ion . Then the products o f the evaluat ion are descr ibed; thesepr oduc t s o r ou t c ome s a r e f o r mu l a t e d a s t he u s e r - c ompu t e r i nt er -action dif ficulties that w ere identif ied, grou ped into three types( ter me d task , presentat ion , and device d i ff icul ties) . The charac-teristics o f each typ e of diff iculty are discussed, in terms o f theirease of identification, their gene rality across application dom ains,the HF k now ledge that th ey draw on, and the ir rela tionship toredesign. The conclusion considers the usefulness o f the evalu-at ion , the inadequacies o f sys tem developm ent pract ice i t impl ies ,and how t o i nc orpor a te H F e v a l ua ti on i n t o an i mpr ov e d s y s t e mdevelopment pract ice .hum an-c om pute r in teraction , h uma n fac tors , sys tem develop-me nt, evaluation, usabili ty, softw are quality assurance, cases t udy

This paper reports a human factors (HF) evaluation of aset of computer-aided software engineering (CASE)tools. The purpose of the paper is to illustrate, for systemdevelopers who are not knowledgeable about HF, theprocesses and products o f a fairly typical HF evaluation,and to identify some important features of HF evalu-ation practice. The intention is, therefore, to talk wherepossible about evaluation in general rather than solely inthe particular context of the application domain ofsystem developers using CASE tools.Well engineered human--computer interfaces are ack-nowledged to be a critical determinant of productsuccess, and the importance of HF input into softwareengineering has been demonstrated in industrial prac-tice t. One of the major forms of HF input has been asevaluation, although HF evaluation has not yet beenincorporated into standard software engineering prac-tice, as illustrated by Nielsen's report of design flaws inseveral commercial software products2. HF evaluation

Ergonomics Unit, UniversityCollege London. 26 BedfordWay, Lon-don WCIH 0AP, UK.*Department of Business Systems Analysis, City University, North-ampton Square. LondonECIV 0HB, UK

should be regarded as pa rt of software quality assurance;it augments the accepted concerns of testing software forreliability, integration, and safety, with an assessment ofthe effectiveness and efficiency of the interaction betweenthe user and the computer. I n common with the generalmotivation for quality assurance, HF evaluation aims toimprove design by diagnosing failures and by ensuringsoftware meets acceptable levels of performance.This paper uses the application domain o f CASE toolsto illustrate HF evaluation. This choice is pertinent in thelight of reported problems of current CASE-tooltechnology and its inability to support software deve-lopers' working practices 3.An introduction to HF evaluation is given in the nextsection. The third section presents the background andmethods for the particular CASE-tools evaluation. Theoutcomes of this evaluation are then given, and the paperconcludes with a discussion of important aspects of theevaluation.H U M A N F A C T O R S E V A L U A T I O NT y p i c a l c u r r e n t pr a c t i c e in h u m a n f a c t o r se v a l u a t i o nIn common with much HF activity, the current practiceof HF evaluation of interactive computer systems can bedescribed as a craft4,5. That is, it relies heavily on theexperience and intuitions of the HF specialist, who maydraw on a body of largely undocumented or implicitknowledge and practice.It should not be surprising, therefore, that currentpractice varies widely. Practitioners have different moti-vations for conducting evaluations (e.g., diagnosis ofproblems, testing against standards, performance com-parison of two or more products), they use variousapproaches (e.g., controlled experiments, laboratoryobservations, field observations), and they use numeroustechniques for data recording and analysis (e.g., videorecording, questionnaires, interviews, system logs).There is no agreed framework for relating these variouspurposes, approaches, and techniques that could be usedto construct reliably sound HF evaluations.Given this variability, it is hard to identify unequivo-cally any typical form o f HF evaluation practice. How-ever, evidence from a number of sources suggests thatthere is a typical form, at least in the sense of the type ofHF evaluation most commonly practised. (Note thatthere is no suggestion that this is an ideal form.) Thisevidence includes published reports and discussions ofevaluations, personal reports from system developers

Vol 34 No 7 July 1992 0950-5849/92/070443-09 1992 Butterworth-Heinemann Ltd 443


2/9

Cas e s t udy i n human f ac t o r s e v a lua t iona n d f r o m H F c o n s u l t a n t s , a n d t h e e m p h a s i s p l a c e d b ym a j o r c o m p u t e r s y s te m m a n u f a c t u r e r s o n H F e v a l u a ti o nl a b o r a t o r i e s 6 a n d o n u s a b i l it y e n g i n e e r i n g a n d i ts v a r i-a n t s 7 .

T h e t y p i c a l f o r m o f H F e v a l u a t i o n t h a t t h e s e s o u rc e ss u g g e s t c a n b e c h a r a c t e r i z e d a s f o l l o w s :, I t o c c u r s la t e in s y s t e m d e v e l o p m e n t ( u s u a ll y w h e n

s o m e w o r k i n g v e r s i o n o f t h e s y s t em u n d e r d e v e l o p -m e n t e x i s t s ).

I t i s c o n d u c t e d b y H F p e r s o n n e l . I t i n v o lv e s e it h e r o r b o t h o f t w o m e t h o d s : ' w a l k -

t h r o u g h s ' b y H F s p e ci a li s ts , a n d o b s e r v a t i o n a l s t u d y( o f t e n i n c lu d i n g v i d e o a n d a u d i o r e c o r d i n g ) o f i n te r -a c t i o n s b e t w e e n t h e u s e r a n d t h e c o m p u t e r . ( A u s e f u lc o m p a r i s o n o f t he s e m e t h o d s i s p re s e n te d b y H a m -m o n d et al .8. )

I t te n d s t o h a v e a b r o a d f o c u s r a t h e r t h a n c o n c e n -t r a t i n g o n s p e c if i c i s s ue s .

I t d o e s n o t i n v o l v e f o r m a l a n a l y t i c a l a n d s t a t i s t ic a lp r o c e d u r e s .A l t h o u g h t h i s t y p ic a l f o r m o f H F e v a l u a t i o n d o e s h a v e an u m b e r o f d r a w b a c k s 9, i t s e em s l ik e l y t o c o n t i n u e a s t h em o s t c o m m o n f o r m o f H F e v a l u a ti o n f o r th e f o rs e e a bl ef u t u r e , f o r a n u m b e r o f r e as o n s . F i r s t, s u c h e v a l u a t i o n sc a n b e v e r y p r o d u c t i v e . T h e r e i s a g o o d d e a l o f e x p e r -i e n c e i n t h e i r c o n d u c t , s u p p o r t e d b y u s e f u l p u b l i s h e dr e c o m m e n d a t i o n s ~0. F u r t h e r , r e s o u r c e c o n s t r a i n t s i ns y s t e m d e v e l o p m e n t w i l l o n l y u n u s u a l l y a l l o w f o r m o r ed e t a i le d o r f o r m a l a n a ly s e s . T h i s f o r m o f e v a l u a t i o n i sa l s o w e l l s u i t e d t o i n c o r p o r a t i o n w i t h i n a n i t e r a t i v e p r o -t o t y p i n g a p p r o a c h t o s y s t e m d e v e l o p m e n t , w h i c h i s i t se l fw i d e l y r e c o m m e n d e d a s a n a p p r o a c h t o u s e r - i n t e rf a c ede s ign 11,~2. F ina l ly , t he re a re a s y e t n o r e a l ly r e l i a b l e a n de f f ec t iv e a l t e r n a ti v e f o r m s o f e v a l u a t io n . T h u s a t t h em o m e n t t h e r e a r e n o g o o d m e a n s o f e v a l u a t in g e a r l y i nd e v e l o p m e n t 13, o f e v a l u a t i n g w i t h o u t s o m e f o r m o f u s e rb e i n g d i re c t l y in v o l v e d , o r o f i n t e g r a t in g H F e v a l u a t i o ni n t o s o f t w a r e e n g i n e e r i n g m e t h o d s s o t h a t i t c a n b ec a r r i e d o u t b y n o n - H F p e r s o n n e l . O n t h i s la s t p o i n t ,t h e r e is e v id e n c e t h a t s y s t e m d e v e l o p e r s c a n m a k e g o o du s e o f s o m e H F e v a l u a t i o n t e c h n i q u e s t4,15, b u t i t i s a l soc l e a r t h a t H F e v a l u a t i o n s b y s y s t e m d e v e l o p e r s a r e l i ke l yt o b e i n c o m p l e t e 16.T h e r e i s, t h e r e f o r e , a g o o d c a s e f o r s e e k in g t o d e v e l o pa n d i m p r o v e ( r a t h e r t h a n t o r e p l a c e ) t y p i c a l c u r r e n tp r a c t i c e in H F e v a l u a t i o n . I t is a n e v a l u a t i o n o f th i s f o r mt h a t w i l l b e p r e s e n t e d l a t e r i n t h i s p a p e r .

C h a r a c t e r i za t i o n o f h u m a n f a c t o r s e v a l u a t io nT h i s p a p e r a i m s b o t h t o i l l u s t r a t e t h e p r o c e s s e s a n dp r o d u c t s o f a ty p i c a l H F e v a l u a t i o n a n d t o i d e n t i f y s o m ei m p o r t a n t f e a t u r e s o f t hi s f o r m o f e v a l u a t i o n p r a c t ic e .T o m e e t b o t h t h e s e a i m s , i t i s n e c e s s a r y t o d i s c u s s t h eg e n e r a l n a t u r e o f H F e v a l u a t i o n . U n f o r t u n a t e l y , t h e r e isn o c o n s e n s u s v i e w o n t h i s i n t h e H F l i t e r a t u r e . T h u st h e r e i s n o a g r e e d d e f i n i t i o n o f e v a l u a t i o n o r o f it s re l e -v a n t c o n c e p t s . W h i t e f i e l d et a l . 9 h a v e p u t f o r w a r d a

f r a m e w o r k f o r H F e v a l u a t i o n t h a t i s i n t e n d e d t o m a k ee x p l ic i t t h e c o n c e p t s a n d r e l a ti o n s o f e v a l u a t i o n a n d t h a ti d e n t if i e s a w i d e r a n g e o f e v a l u a t i o n m e t h o d s . T h e d i s-c u s s i o n o f H F e v a l u a t i o n t h a t f o l l o w s in t h is s e c t i o n isb a s e d o n t h is f r a m e w o r k . T h e i n t e n t i o n o f t h i s di s c u ss i o ni s to e n a b l e a n i n f o r m e d a n d p r i n c ip l e d a s s e ss m e n t o f t h ee v a l u a t i o n c a s e s t u d y t o f o l lo w .

T h e f r a m e w o r k d e f i n e s H F e v a l u a t i o n a s a n a s s e s s -m e n t o f th e c o n f o r m i t y b e t w e e n a sy s t e m ' s p e r f o r m a n c ea n d i t s d e s i r e d p e r f o r m a n c e . S y s t e m i s u s e d h e r e i n t h es e n s e o f a u s e r a n d a c o m p u t e r e n g a g e d o n s o m e t a s kw i t h i n a n e n v i r o n m e n t . A s e t o f h a r d w a r e a n d s o f t w a r ec o m p o n e n t s i s t h e r e f o r e n o t a s y s t e m i n t h is s e n s e b u ts i m p l y a c o m p u t e r .

T h e n o t i o n o f p e r f o r m a n c e is t ak e n f r o m D o w e l l a n dL o n g 17, w h o d e f i n e p e r f o r m a n c e i n t e r m s o f t w o f a c t o r s :t h e q u a l i t y o f t a s k p r o d u c t ( i .e . , h o w w e l l d o e s t h e w o r kd o n e m e e t i t s g o a l ) a n d t h e i n c u r r e d r e s o u r c e c o s t s ( i . e . ,t h e r e s o u r c e s e m p l o y e d b y b o t h t h e u s e r a n d t h ec o m p u t e r i n a c c o m p l i s h i n g t h e t a sk ) . D i f f e r e n t ty p e s o fu s e r a n d c o m p u t e r r e s o u r c e c o st s c a n b e i d e n ti f ie d . Am o s t e f f e c t iv e s y s t e m w o u l d m i n i m i z e t h e r e s o u r c e c o s t si n p e r f o r m i n g a s p e c i f i e d t a s k w i t h a g i v e n q u a l i t y o fp r o d u c t .

T h e p e r f o r m a n c e o f a s y st e m i s d e t e r m i n e d b y i tsb e h a v i o u r . T h e s y s t e m ' s b e h a v i o u r c o m p r i s e s t h e i n t e r -a c t in g b e h a v i o u r s o f t h e u s er a n d t h e c o m p u t e r . B o t hu s e r a n d c o m p u t e r w i l l h a v e i m p o r t a n t b e h a v i o u r a ll i m i t a t i o n s t h a t c o n s t r a i n p e r f o r m a n c e . T h i s n o t i o n o fp e r f o r m a n c e a n d b e h a v i o u r t h e r e f o r e d i s ti n g u is h e s w h a ti s a c h i e v e d f r o m h o w i t i s a c h i e v e d , a n d a l s o t h e q u a l i t yo f t h e t a s k p r o d u c t f r o m h o w e f fe c ti v e ly i t is p r o d u c e d .

I t f o l l o w s f r o m t h i s v ie w t h a t t o s p e c i f y f u ll y a s y s t e m ' sd e s i re d p e r f o r m a n c e m e a n s t o i d e n t i fy b o t h t h e r e q u i r e dq u a l i t y o f t h e t a s k p r o d u c t a n d t h e a c c e p t a b le l e v el s o fr e s o u r c e c o s t s t o b e i n c u r r e d i n a c c o m p l i s h i n g t h e t a s k .T h e c u r r e n t s t a t e o f th e a r t i n H F m e a n s t h a t b o t h o ft h e s e w i l l r a r e l y b e s t r a i g h t f o r w a r d . T h e e x p r e s s i o n s o fb o t h r e q u i r e d t a s k q u a l i t y a n d a c c e p t a b l e r e s o u r c e c o s t sc o u l d t a k e m a n y f o r m s a n d a r e n o t r e s t ri c t ed t o a p a r t i-c u l a r f o r m a t . I m p o r t a n t l y , t h is a l l o w s t h e a s p e c ts o f p e r-f o r m a n c e t h a t a r e o f p a r t i c u l a r i n t e re s t f o r t h e e v a l u a t i o n( e .g . , e as e o f l e a r n i n g , p r o b l e m s w i t h t h e i n t e r a c t i v e d i a -l o g u e, a t t i t u d e s o f u s er s , a p p r o p r i a t e n e s s o f f u n c t i o n -a l it y , a n d s o o n ) t o b e c o n c e n t r a t e d o n .A n e v a l u a t i o n i n v o lv e s b o t h a m e t h o d ( t h e p r o c e s s b yw h i c h i t i s d o n e ) a n d a s t a t e m e n t ( t h e r e s u l t i n g p r o d u c t ) .E v a l u a t i o n s t a t e m e n t s a r e o f t w o k i n d s: t h o s e th a t s i m p l y r e p o r t p e r f o r m a n c e a n d c o n f o r m i t y

a n d t h e r e f o r e t e n d t o b e q u a n t i t a t i v e ( h e r e c al l ed m e a -s u r e m e n t s t a t e m e n t s )

t h o s e t h a t r e l a t e th e b e h a v i o u r a l c a u s e s o f th e p e r f o r -m a n c e a n d t h e r e f o r e t e n d t o b e q u a l i t a t i v e ( h e re c a l le dd i a g n o s t i c s t a t e m e n t s )

T h u s a t y p i c a l m e a s u r e m e n t s t a t e m e n t r e p o r t s h o w c l o -s e ly o b s e r v e d p e r f o r m a n c e m a t c h e s s o m e m e a s u r a b l e c r i -t e r i o n , b u t i t r e p o r t s n o t h i n g a b o u t h o w t h a t p e r f o r -m a n c e i s a c h i e v e d . D i a g n o s t i c s t a t e m e n t s , o n t h e o t h e r

444 In fo rma t ion a nd So f tw a re Te c hno logy


3/9

h a n d , a t t e m p t t o i d e n t i f y t h e c r it i ca l b e h a v i o u r t h a tu n d e r l ie s t h e p e r f o r m a n c e .

W h i t e f i e l d e t a l . 9 d i sc u s s f o u r c l a ss e s o f H F e v a l u a t i o nm e t h o d , d i s t i n g u is h e d a c c o r d i n g t o w h e t h e r t h e u s e r a n dt h e c o m p u t e r a r e e a c h r e a l o r r e p r e s e n t a t i o n a l p r e s e n c e si n t h e e v a lu a t i o n . T h e f o u r c l a ss e s o f m e t h o d a r e: A n a l y t i c m e t h o d s , i n v o l v i n g th e u s e o f r e p r e s e n t a t i o n so f b o t h t h e u s e r a n d t h e c o m p u t e r ; n o r m a l l y t h i s

i n v o lv e s th e m a n i p u l a t i o n o f m o d e l s o f t h e s y s t em t op r e d i c t p e r f o r m a n c e .

S p e c i a l is t r e p o r t s , in v o l v i n g a n H F o r o t h e r s p e c i a li s t( i . e . , n o t t h e i n t e n d e d u s e r s ) u s i n g t h e c o m p u t e r t oa s se s s a n i m p l e m e n t e d o r p r o t o t y p e v e r s i o n .

U s e r r e p o r t s , i n v o l v i n g s u r v e y m e t h o d s ( q u e s t i o n -n a i r e s , i n t e rv i e w s , r a t i n g s c a l e s , et c . ) t o o b t a i n d a t a o ro p i n i o n s f r o m u s e rs w h e n t h e y a r e n o t d i r e c t l y i n t e r-a c t in g w i t h t h e c o m p u t e r .

O b s e r v a t i o n a l m e t h o d s , i n v o l v i n g re a l u se r s i n te r a c t -i n g w i th t h e c o m p u t e r ; t h e s e t o f s u c h m e t h o d s i s l a rg e ,r a n g i n g f r o m i n f o r m a l o b s e r v a t i o n o f a s in g le u s e r tof u l l - s c a l e e x p e r i m e n t a t i o n w i t h a p p r o p r i a t e n u m b e r so f s u b j e ct s a n d c o n t r o l o f v a r i a bl e s .

T h e f r a m e w o r k f o r H F e v a l u a ti o n j u s t o u t l in e d h a s an u m b e r o f i m p l i c a ti o n s a n d c o n s e q u e n c e s , s o m e o fw h i c h h a v e b e e n m e n t i o n e d a l r e a d y . S o m e f u r t h e r i m p l i-c a t i o n s a r e a s f o l l o w s . F i r s t , e v a l u a t i o n c a n b e d o n ed u r i n g s y s t e m d e v e l o p m e n t a t a n y t i m e a f t e r t h e f i r s td e s c r i p t i o n o f t h e p r o p o s e d s y s t e m i s p r o d u c e d ( w h e t h e rt h i s b e a s p e c i f i c a t i o n o r a p r o t o t y p e o r w h a t e v e r ) . I tn e e d n o t w a i t u n t i l s o m e w o r k i n g v e r s i o n e x i s t s . R e l a t e dt o t h is i s t h e s e c o n d p o i n t t h a t m o d e l s o r o t h e r r e p r e s e n t -a t i o n s o f u s e r s c a n s e r v e a s a m e a n s o f i n c l u d i n g u s e r s i nd e v e l o p m e n t . T h e a c t u a l p h y s i c a l p r e s e n c e o f u s er s i s n o tm a n d a t o r y . F i n a l l y , e v a l u a t i o n s w i l l a d d r e s s p a r t i c u l a ra s p e c t s o f s y st e m p e r f o r m a n c e a n d n o t a l l as p e c ts . A n ye v a l u a t i o n r e q u i r e s a f o c u s o n c e r t a i n s y s te m b e h a v i o u r sa n d c e r t a i n t a s k p r o d u c t s - - i t m u s t b e ta i l o r e d t o th eg i v e n s y s t e m a n d t o t h e i n t e n d e d g o a l s o f t h a t s y s t e m . I tis t h e r e f o r e n o t t h e c a s e t h a t a n e v a l u a t i o n c a n a d d r e s sa ll H F a s p e c t s o f a s y s t e m ( a t l e a s t n o t w i t h o u t u n l i m i t e dr e s o u r c e s w i t h w h i c h t o c o n d u c t a m u l t i - p a r t e v a l u -a t i o n ) .

T h e f r a m e w o r k i s i n t e n d e d t o e n c o m p a s s a l l f o r m s o fH F e v a l u a t i o n a c t i v i ty . I t s h o u l d t h e r e f o r e b e p o s s i b le t od e s c r i b e t y p i c a l c u r r e n t p r a c t i c e ( a s o u t l i n e d e a r l i e r ) i nt h e s e t e r m s . V i e w e d i n t h i s w a y , t y p i c a l c u r r e n t p r a c t i c ec a n b e s e e n t o i n v o l v e H F p e r s o n n e l i n s p e c i a l i s t r e p o r t sa n d i n o b s e r v a t i o n a l m e t h o d s o f a r e la t iv e l y i n f o r m a lk i n d ( l i t t l e e x p e r i m e n t a l c o n t r o l a n d l i t t l e f o r m a l o r s t a -t i st i c al d a t a a n a l y s i s ) . I t r a r e l y h a s a c o m p l e t e a n d w e l ls p e c i fi e d v ie w o f d e s i r e d p e r f o r m a n c e : i t u s u a l l y i g n o r e st h e t as k q u a l i t y c o m p o n e n t ( h o w w e l l t h e w o r k is d o n e ) ,p r e f e r r i n g t o c o n c e n t r a t e o n t h e r e s o u r c e c o s t c o m p o -n e n t ( u s u a l l y i n t e r m s o f u s e r s ' e r r o r s a n d d i f fi c u lt i es ) ,a n d i t t e n d s t o h a v e a b r o a d f o c u s r a t h e r t h a n a s p e c i f i co n e t h a t a d d r e s s e s p a r t i c u l a r p e r f o r m a n c e c r i t er i a . I t c a np r o d u c e e i t h e r m e a s u r e m e n t o r d i a g n o s t i c s t a t e m e n t s ,o f t e n d e p e n d i n g o n w h e t h e r t h e e v a lu a t i o n i s a c o m p a r i -

A WHITEFIELD AND A SUTCLIFFE

s o n o f a l te r n a t i v e p r o d u c t s o r p a r t o f a d e t a il e d r e d e s i g no f a s i n gl e p r o d u c t . T h e l a t e o c c u r r e n c e o f t h e t y p i c a l H Fe v a l u a t i o n w i t h i n t h e s y s t e m d e v e l o p m e n t l i fe - cy c le p r o -b a b l y r e fl e ct s th e c u r r e n t w e a k n e s s o f a n a l y t i c e v a l u a t i o nm e t h o d s , a s w e l l a s t h e r e q u i r e m e n t f o r s o m e f o r m o fi m p l e m e n t e d s o f t w a r e w i t h w h i c h t o c o n d u c t t h e p r e -f e r r e d m e t h o d s ( o b s e r v a t i o n a l m e t h o d s a n d s p e c i al is tr e p o r t s ) .N o w t h a t a f r a m e w o r k f o r c h a r a c t e r i zi n g H F e v a l u -a t i o n , a n d a l s o t y p i c a l c u r r e n t p r a c t i c e w i t h i n t h i s f r a m e -w o r k , h a s b e e n d e s c r i b e d , i t i s p o s s i b l e t o p r e s e n t t h ep a r t i c u l a r e v a l u a t i o n t h a t c o n s t i t u t e s t h e c a s e s tu d y .

C A S E - T O O L S E V A L U A T I O N :B A C K G R O U N D A N D M E T H O D ST h e s y s t e m f o r e v a l u a t i o n w a s a t o o l s e t f o r d e v e l o p m e n tu s i n g th e S t r u c t u r e d S y s t e m s A n a l y s is a n d D e s i g n M e t h -o d o l o g y ( S S A D M ) ~8. T h e t o o l s e t w a s d e v e l o p e d a s p a r to f a s o f t w a r e e n g i n e e r i n g r e s e a r ch p r o j e c t f u n d e d b y t h eU K G o v e r n m e n t ' s A lv e y P r o g r a m m e . F i r st , i m p o r t a n ta s p e ct s o f t h e v a r io u s s y s t e m c o m p o n e n t s a t t h e t i m e o ft h e e v a l u a t i o n a r e d e s c r i b e d . T h i s i s f o l l o w e d b y d e t a i l so f t h e e v a l u a t i o n f o c u s . T h e c o n t e x t o f t h e e v a l u a t i o n i st h e n d e s c r i b e d , w h i l e t h e m e t h o d s u s e d a r e o u t l i n e d i nt h e f i n a l s u b s e c t i o n .

S y s t e m f o r e v a lu a t i o nH a r d w a r e a n d s o f tw a r eT h e s y s te m r a n o n a m o n o c h r o m e S u n - 3 w o r k s t a ti o n . Av e r s i o n f o r I B M P C s o r c o m p a t i b l e s w a s u n d e r d e v e l o p -m e n t .T h e s o f t w a r e c o m p r i s e d t h r e e d i a g r a m e d i t o r s , a d a t ad i c t i o n a r y , a n d l i nk s b e t w e e n t h e m . T h e d a t a d i c t i o n a r yu s e d a p r o p r i e t a r y d a t a b a s e p r o d u c t a n d t h e r e f o r e c o u l dn o t b e c h a n g e d i n a n y s u b s t a n t ia l w a y . A t t h e t i m e o f t h ee v a l u a t i o n , t h e r e w e r e p l a n s t o r e p l a c e i t w i t h a n o t h e rp r o p r i e t a r y p r o d u c t . T h e d i a g r a m e d i t o r s ( D E s ) w e r e a llp r o j e c t d e v e lo p m e n t s . T h e y w e r e c o n c e r n e d w i t h e d i t i n gd a t a f l o w d i a g r a m s , l o g i c al d a t a s t r u c t u r e d i a g r a m s , a n de n t i t y l i f e h i s t o r y d i a g r a m s .

A l l t h r e e D E s h a d a g o o d d e a l in c o m m o n , i n c l u d i n g as i m i l a r s c r e e n l a y o u t . T h e b a s i c f e a t u r e s o f t h i s l a y o u ta r e s h o w n i n F i g u r e 1 . O f p a r t i c u l a r i n t e r e s t a r e th ed r a w i n g a r e a , t h e p a l e t t e ( i n d i c a t i n g t h e it e m s t h a t c a n b ed r a w n ) , a n d t h e m e n u b a r ( c o n t a i n i n g p u l l - d o w n m e n u su n d e r a n u m b e r o f h e a di n g s ).

A l l th r e e D E s w e r e te s t e d u s in g a d v a n c e d p r o t o t y p e s .T h e s o f t w a r e p r o v e d g e n e r a l l y r e l i a b l e , a l t h o u g h t h e r ew e r e s e v e r a l m i n o r b u g s ; t h e m o s t s e r i o u s o f th e s e c o n -c e r n e d t h e d i s p la y h a n d l i n g i n t h e D E s , w h i c h o c c a s i o n -a l l y c r a s h e d o u t .

T h e f i r s t i s s u e o f t h e t o o l s e t ' s u s e r g u i d e w a s a l s os u p p l i e d . T h i s w a s s u b s t a n t i a l l y , b u t n o t e n t i r e l y , c o m -p le t e .U s e r sT h e i n t e n d e d u s e rs o f t h e to o l s e t w e r e S S A D M a n a l y s tsa n d d e s i g n e r s . I t w a s a s s u m e d t h a t t h e u s e r s w o u l d

Vol 34 No 7 July 1992 445


4/9

Case study in human actors evaluationwindow\

pale~e

, / w i n d o w h ea de rba r

J

0 - - - - 7 1 - -- - " 6 1 i f I I ' " - - I ~ , . / - - m e n u b a r

.~mw~g area

Figure 1. Basic screen layout of diagram editorsa l r e a d y b e f a m i l i a r b o t h w i t h a n a l y s i s a n d d e s i g n i ng e n e r a l a n d w i t h t h e S S A D M m e t h o d i n p a r t i c u l a r .T a s k sT h e t o o l s e t w a s n o t i n t e n d e d t o c o v e r e v e r y a s p e c t o fS S A D M d e v e l o p m e n t . It w a s a im e d a t t h e t h re e m a j o rt y p e s o f d i a g r a m i n S S A D M : d a t a f l o w a n d l o g i ca l d a t as t r u c t u r e d i a g r a m s , w h i c h a r e u s e d t h r o u g h o u t t h e t h r e ep h a s es o f S S A D M d e v e l o p m e n t , an d t h e l es s i m p o r t a n te n t i t y li f e h i s t o r y d i a g r a m s . T h e t o o l s e t w a s a l s oi n t e n d e d t o i n c l u d e t h e a c t iv i t i e s a s s o c i a t e d w i t h t h e d a t ad i c t i o n a r y , i .e ., w i t h m a i n t a i n i n g a d a t a b a s e o f i n f o r -m a t i o n o n t h e s y s te m d e si gn . E x a m p l e s o f S S A D M d e v e-l o p m e n t a c t iv i ti e s n o t c o v e r e d b y t h e t o o l s e t a r e t h es p e ci fi c t e c h n i q u e s o f d i a l o g u e d e s i g n a n d p r o c e s s o u t -l i n es a n d t h e g e n e r a l a c t i v i ti e s o f p l a n n i n g a n d q u a l i t ya s s u r a n c e .

T h e t a s k s f o r w h i c h t h e t o o l s e t w a s i n t e n d e d , t h e r e -f o r e , a r e : c r e a t i n g a n d e d i ti n g d a t a fl o w d i a g r a m s c r e a t i n g a n d e d i t i n g lo g i c al d a t a s t r u c t u r e d i a g r a m s c r e a t i n g a n d e d i t i n g e n t i t y l i fe h i s t o r y d i a g r a m s d e v e l o p i n g a n d m a i n t a i n i n g a d a t a d i c t i o n a r y o f

d e s i g n i n f o r m a t i o n , i n p a r t i c u l a r w i t h r e s p e c t t o t h ei n f o r m a t i o n i n t h e t h r e e s o r ts o f d i a g r a m sE n v i r o n m e n t sT h e s y s t e m w a s i n t e n d e d f o r i n s t a l l a ti o n i n s t a n d a r do f f i c e e n v i r o n m e n t s .

F o c u sI t w a s s t a t e d e a r l i e r t h a t a n H F e v a l u a t i o n n e e d s t o b ef o c u s e d o n c e r t a i n a s p e c ts o f s y s t e m p e r f o r m a n c e a n dn o t o n p e r f o r m a n c e i n g e n er a l . T h e p a r t i c u l a r f o c i n e e dt o b e e x p r e s s e d a s c r i t e r ia f o r d e s i r e d p e r f o r m a n c e . D i s -c u s s i o n s w i t h t h e s y s t e m d e v e l o p e r s a b o u t t h e i s su e s t h a tm i g h t b e a d d r e s s e d b y t h e e v a l u a t i o n i d e n t if i e d tw o f o c io f p a r t i c u l a r i n t e r e s t t o t h e d e v e l o p e r s .

T h e f i r s t f o c u s w a s t h a t t h e D E s s h o u l d b e a s e as y t ol e a r n a s p o s s i b l e . E a s e o f l e a r n i n g i s in f l u e n c e d b y m u l t i -p l e d e t e r m i n a n t s , a n d t h is f o c u s t h e r e f o r e n e e d e d t oc o n s i d e r a w i d e r a n g e o f s y s t e m b e h a v i o u r ( b u t n o t a l ls y s t e m b e h a v i o u r ) . W i t h s u c h a b r o a d f o c u s , t h e r e l e v a n tc r i t e r io n w a s a l s o e x p r e s s e d r a t h e r b r o a d l y :

N a i v e u s e r s s h o u l d b e a b l e t o u s e e a c h D E t o c o m p l e t eb a s i c e d i t i n g t a s k s w i t h o u t e n c o u n t e r i n g s e r i o u s d i f f i -c u l t i e s o r m a k i n g s e r i o u s e r r o r s .

T h e s e c o n d f o c u s w a s t h a t u s er s s h o u l d u n d e r s t a n d t h er e l a t i o n s h i p s b e t w e e n t h e v a r i o u s t o o l s , a n d e s p e c i a l l yb e t w e e n t h e d a t a d i c t i o n a r y a n d t h e D E s . T h e y s h o u l d b ea b l e t o m a k e u s e o f a ll a p p r o p r i a t e f a c i li t ie s a n d t oc h a n g e t o o l s a s n e c e s s a r y . T h e r e l e v a n t c r i t e r i a w e r e : U s e r s s h o u l d u n d e r s t a n d w h a t i n f o r m a t i o n a b o u t

d i a g r a m e l e m e n t s is s t o r e d i n t h e d a t a d i c t i o n a r y . U s e r s s h o u l d u n d e r s t a n d h o w t o e n t e r th e d i a g r ame l e m e n t s i n t h e d a t a d i c t i o n a r y . U s e r s s h o u l d k n o w h o w t o a c ce s s a n d e d i t s p e ci f i ed

i n f o r m a t i o n r e l a t e d t o t h e d i a g r a m s i n th e d a t a d i c t i o n -a r y .

T h e s e f o u r w e r e t h e p r i n c i p a l c r i t e r i a a g a i n s t w h i c h t h es y s t e m w a s t o b e e v a l u a t e d . I n a d d i t i o n , o t h e r a s p e c ts o fp e r f o r m a n c e a n d b e h a v i o u r w o u l d b e e v a l u a t e d i n p a s s -i n g ( i . e . w i t h o u t s t a t e d c r i t e r i a a n d n o t i n d e p t h ) .

C o n t e x t o f e v a l u a t i o nT o d e c id e o n t h e a p p r o p r ia t e m e t h o d o r m e t h o d s t o b eu s e d , it is n e c e s s ar y t o k n o w s o m e t h i n g a b o u t t h e c o n t e x to f t h e e v a l u a ti o n . I n p a r t i c u l a r i t n e e d s t o b e k n o w n h o wt h e s y s t e m d e v e l o p m e n t is b e in g d o n e , a b o u t t h e f o r m o rf o r m s o f t h e s y s t e m t h a t w i l l b e a v a il a b l e f o r e v a l u a t i o n ,a n d a b o u t t h e r e s o u r c e s a v a i la b l e f o r t h e e v a lu a t i o n . T h ei m p o r t a n t a s p e c t s o f t h es e i s su e s f o r t h e c a s e i n q u e s t i o nw e r e a s f o l l o w s .

W i t h r e g a r d s t o t h e s y s t e m d e v e l o p m e n t , t h i s w a sb e i n g c o n d u c t e d u s i n g s o m e s e m i f o r m a l n o t a t i o n s ( e . g. ,d a t a f l o w d i a g r a m s ) w i t h in a g e n e r a l l y t o p - d o w na p p r o a c h . T h e d e v e l o p e r s h a d r e a c h e d t h e s t ag e o f p r o -d u c i n g a n a d v a n c e d p r o t o t y p e . N o n e o f t h e d e v e lo p e r sw a s k n o w l e d ge a b l e a b o u t H F .

T h e s y s t e m a c t u a l l y u s e d f o r t h e e v a l u a t i o n w a s d e s -c r i b e d e a r l i e r . T h e m a i n p o i n t s t o n o t e h e r e a r e a sfo l low s : N o c o m p l e t e s et o f s p e c i f ic a t i o n s e x i s t e d , s o a n y a n a l y -

t i c e v a l u a t i o n m e t h o d w o u l d h a v e t o h a v e c r e a t e dt h o s e t o s t a r t w i t h .

T h e r e w e r e n o e x p e r i e n c e d u s e r s , a s t h e r e w a s n oe x t a n t v e r s i o n i n u s e .

T h e t a s k s f o r t h e e v a l u a t io n n e e d e d t o b e d e v e l o p e d ,c o n s i s t e n t b o t h w i t h t h e t a r g e t t a s k s f o r t h e t o o l s e ta n d w i t h t h e f o c i o f t h e e v a l u a t i o n . V e r s i o n s o f th e a c t u a l h a r d w a r e a n d e a r l y s o f t w a r ew e r e a v a i l a b l e .



5/9

T h e a v a i la b l e r e s o u r c e s i n c l u d e d b a s i c r e c o r d i n g e q u i p -m e n t ( v i d e o a n d a u d i o ) , a c c e ss t o a sm a l l n u m b e r o f H Fs p e ci a li s ts a n d t w o S S A D M s p e ci a li s ts , a n d a m a x i m u me l a p s e d ti m e p e r i o d o f t w o m o n t h s .

M e t h o d sT h e e v a l u a t i o n f o c u s r e q u e s t e d b y t h e c l i e n t c o n c e n -t r a t e d o n a s s e s s m e n t o f t h e l e a r n a b i li t y a n d t h e f u n c t i o n -a l it y o f th e e d i t o r s a n d d a t a d i c t i o n a r y . T h e n e e d t oa s s e ss l e a r n a b i l i t y s u g g e s t e d t h e u s e o f o b s e r v a t i o n a lm e t h o d s , w h i l e t h e n e e d t o a s s e ss f u n c t i o n a l i t y i n d i c a t e dt h e u s e o f s p e c i a li s t r e p o r t s i n a d d i t i o n . T a k i n g t h e s ec o n c e r n s i n t o a c c o u n t , a l o n g w i t h t h e c o n t e x t d e s c r i b e di n t h e l as t s u b s e c t io n , l e d t o t h e e v a l u a t i o n m a k i n g u s e o ft h r e e f o r m s o f t h e e v a l u a t i o n m e t h o d s m e n t i o n e d e a r l ie r .( S u c h d ec i s io n s a b o u t m e t h o d s b a s e d o n l o c i a n d c o n t e x ta r e b y n o m e a n s f i x e d a n d a r e p o o r l y u n d e r s t o o d ; i n i ti a ld i sc uss ions o f t he se de c i s ions ha ve be e n pub l i she d l9 ,2 . )

S p e c i a l i s t r e p o r t s b y t h r e e H F e n g i n e e r s , i n t w o s e s -s i o n s l a s t i n g a p p r o x i m a t e l y o n e h o u r e a c h .S p e c i a l i s t r e p o r t s b y t w o d o m a i n ( i . e . , S S A D M ) s p e -c i al is t s, i n o n e s e s si o n o f a p p r o x i m a t e l y o n e h o u r .O b s e r v a t i o n a l m e t h o d s i n v o l v i n g f o u r u s e r s u n t r a i n e do n t h e s y s t e m . T h e u s e r s w e r e c o m p u t e r - s c i e n c e s t u -d e n t s f a m i l i a r w i t h s t r u c t u r e d m e t h o d s . E a c h u s e rw o r k e d i n d i v i d u a ll y f o r u p t o t w o h o u r s o n s e t ta s k s .T h e t a s k s i n v o l v e d t h e c r e a t i o n a n d e d i t in g o f a p p r o -p r i a t e d i a g r a m s , b a s e d i n s o m e c a s e s o n t h e t u t o r i a le x e r c i s e s i n t h e u s e r g u i d e . O n e o f t h e e v a l u a t o r s w a sp r e s e n t a t a l l ti m e s t o e l i c i t f r o m , a n d d i s c u s s w i t h , t h eu s e r s t h e i r v i e w o n t h e i n t e r a c t i o n a s i t p r o g r e s s e d . T h es e s s i o n s w e r e a u d i o t a p e d a n d t h e e v a l u a t o r m a d en o t e s i n a d d i t i o n . T h e t a p e s a n d n o t e s w e r e s u b s e -q u e n t l y i n f o r m a l l y a n a l y s e d t o i d e n t i f y t h e d i f f i c u l t i e se n c o u n t e r e d b y t h e u s e rs .

E xa m p l e o f t y p i c a l c u r r e n t p r a c t i c eT h e e v a l u a t i o n a s d e s c r i b e d i n t h is s e c t i o n c l e a r l y c o r r e s -p o n d s w i t h t y p i c a l c u r r e n t p r a c t i c e a s o u t l i n e d e a r l i e r : i ti n v o lv e s a n a d v a n c e d p r o t o t y p e a n d i s t h e r e f o r e l at e i ns y s t e m d e v e l o p m e n t , i t i s c o n d u c t e d b y H F p e r s o n n e l , i ti n v o l v es b o t h o b s e r v a t i o n a l m e t h o d s a n d s p e c i a li s tr e p o r t s , a n d i t d o e s n o t i n v o l v e f o r m a l a n a l y t i c a l a n ds t a t is t i c a l p r o c e d u r e s . T h e m a j o r d i f f e r e n c e i s t h a t t h i se v a l u a t i o n i s m o r e f o c u s e d t h a n m u c h c u r r e n t e v a l u a t i o np r a c t i c e , a l t h o u g h i t i s c e r t a i n l y n o t a s f o c u s e d a s i tm i g h t b e .

C A S E - T O O L S E V A L U A T I O N :O U T C O M E SO u t c o m e s t o b e p r e s e n t e dG i v e n t h e p r i n c ip a l f o c u s o f th e e v a l u a t i o n o n i d e n t i f y i n gt h e d i f fi c u l ti e s a n d e r r o r s e n c o u n t e r e d b y t h e u s e r s , i t isa p p r o p r i a t e t o p r e s e n t a n d d i sc u s s t h e o u t c o m e s o f th ee v a l u a t i o n i n th e s a m e t e r m s . B u t r a t h e r t h a n p r e s e n t t h e

A WHITEFIELD AND A SUTCLIFFE

f u l l l i st o f d i f fi c u l ti e s ( 7 5 f o r t h e D E s , p l u s t h o s e f o r t h ed a t a d i c t i o n a r y ) , t h e a u t h o r s h a v e c h o s e n t o p r e s e n t as e l e c t i o n o f d if f i cu l t ie s w i t h t h e D E s t h a t i s i n t e n d e d t oi l l u s t r a t e t h e v a r i o u s t y p e s i d e n t i fi e d . ( T h e t e r m d i f f i c u lt yw i l l b e e m p l o y e d f r o m h e r e o n a s a g e n e r a l t e r m t oi n c l ud e e r r o r s a n d p r o b l e m s . )

B e f o r e p r e s e n t i n g t h e d i f fi c u lt i es , i t s h o u l d b e s a i d t h a tt h e s y s t e m ( e x c e p t f o r t h e d a t a - d i c t i o n a r y c o m p o n e n t )g e n e r a l l y p e r f o r m e d w e l l i n t h e e v a l u a t i o n a n d w a s a tl e a st a d e q u a t e f o r t h e d i a g r a m - e d i t i n g t a s k s f o r w h i c h i tw a s d e s i g n e d . T h e d i f fi c u l ti e s i d e n t i f i e d r e l a t e t o a r e a s i nw h i c h i t c o u l d b e i m p r o v e d .

T h e r e i s n o s i n g l e f i x e d w a y i n w h i c h t o g r o u p a n dp r e s e n t t h e d i f fi c u lt i e s i d e n t i fi e d . T h e m o s t a p p r o p r i a t ew a y w i ll d e p e n d o n m a n y f a c t o r s , n o t l e as t o n t h e e v a l u a -t o r , t h e s y s t e m u n d e r e v a l u a t i o n , a n d t h e e v a l u a t i o nf o c u s . I n t h i s c a s e , t h e d i f f ic u l ti e s w e r e d i v i d e d i n t o t h r e ek i n d s , t e r m e d d e v i c e d i f fi c u lt i es , p r e s e n t a t i o n d i f f ic u l t ie sa n d t a s k d i f f i c u l t i e s . T h i s d i v i s i o n i s n o t f o r m a l l yd e f i n e d , b u t i t i s b a s e d o n , a n d h a s m a n y p a r a l l e l s w i t h ,m o r e f o r m a l l a y e re d d e s c ri p ti o n s o f t h e h u m a n -c o m p u t e r i n t e r f a c e 21,22.

D e v i c e d i f f i c u l t i e sT h e d e v i c e d if f i cu l t ie s a r e t h o s e e n c o u n t e r e d i n t h e d i r e c tu s e o f t h e i n p u t - o u t p u t d e v i c e s ( in t h is i n s t a n c e a k e y -b o a r d , a m o u s e , a n d a b i t - m a p p e d m o n o c h r o m ed i s p l a y ) . T h e r e w e r e n o t m a n y s u c h d i f f ic u l ti e s , b u t t h o s et h a t d i d o c c u r w e r e c e r t a i n l y f r e q u e n t a n d p r o b l e m a t i cf o r u s er s n e w t o t h e p a r t i c u l a r h a r d w a r e p l a t f o r m ,e s p e c i a l ly e a r l y in t h e t e s t i n g s e s s i o n . E x a m p l e d e v i c ed i f f i c u l t i e s i nc lude d : A l l u s e rs e x p e r i en c e s o m e p r o b l e m s w i t h w h i c h b u t t o n

t o u s e o n t h e ( t h r e e - b u t t o n ) m o u s e . T h e d i f f e r e n c e b e t w e e n b o l d a n d p l a i n t e x t is sm a l l a n d

e a s il y o v e r l o o k e d .U s e r s w o u l d b e e x p e c t e d t o o v e r c o m e d i ff i cu l ti e s o f th i st y p e r e l a t i v e l y e a s i l y a s t h e y g a i n e d e x p e r i e n c e w i t h t h ep a r t i c u l a r d e v i c e s i n t h i s s y s te m , a n d s o m e i n i ti a l tr a i n -i n g w o u l d m a k e t h e m l es s f r e q u e n t a n d p r o b l e m a t i c f r o mt h e s t a r t .

P r e s e n t a t i o n d i f fi c u l ti e sM u c h m o r e c o m m o n a n d v a r i e d w e r e t h e p r e s e n t a t i o nd i f fi c u lt i es . T h e s e c o n c e r n t h e d i a l o g u e a s p e c t s o f th ei n t e r a c ti o n , s u c h as c o m m a n d n a m e s , s c r e e n a p p e a r a n c e ,a n d f e e d b a c k . S o m e e x a m p l e s o f ea c h t y p e a r e a sf o l l o w s .C o m m a n d s T h e d i s t in c t i o n s b e t w e e n s o m e m e n u o p t i o n s a r e

u n c l e a r (e .g . , b e t w e e n S A V E a n d S T O R E o r b e tw e e nP R I N T a nd P R I N T D F D ) .

T h e D E L E T E a n d B A C K S P A C E k e y s o n th e k e y-b o a r d a r e u s e d i n c o n s i s t en t l y . S o m e c o m m a n d n a m e s a r e m i s l e ad i n g ; f o r e x a m p l e ,

Vol 34 No 7 July 1992 447


6/9

Case study in human fac tors evaluationA D D F I L E T O D A T A D I C T I O N A R Y o n l y a dd s t hef il e n a m e , w h e r e a s u s e r s a s s u m e d i t m e a n t t h e f i lec o n t e n t s .

T h e s in g l e g r a p h i c a l m e n u ( f o r s e l e c t in g b e t w e e n h o r i -z o n t a l o r v e r t i c a l a l i g n m e n t ) i s a m b i g u o u s a n d t h e r e -f o r e c o n f u s i n g .

U s e r s s o m e t i m e s t y p e i n t o t h e w r o n g w i n d o w ,e s p e c i a l l y w h e n u s i n g d i a l o g u e b o x e s , b e c a u s e t h e y f a i lt o r e m e m b e r t h e i m p o r t a n c e o f t h e p o i n t e r p o s i ti o nw i t h r e s p e c t t o t h e c u r r e n t w i n d o w .

S c r e e n a p p e a r a n c e T h e b u t t o n s i n s o m e d i a l o g u e b o x e s a r e e x t r e m e l y

s m a l l a n d t h e r e f o r e h a r d t o s e l ec t . T h e i t e m s o n t h e m e n u b a r a r e m a d e t o s t a n d o u t b y

u s e o f a g r e y b a c k g r o u n d , b u t s o m e u s e r s ( b a s e d o nt h e i r e x p e r i e n c e w i t h o t h e r s y s t e m s ) i n i ti a l l y t a k e t h i st o m e a n t h a t t h e i t e m s a r e n o t a v a i l a b l e f o r s e l e c t i o n .

W i t h i n a p u l l - d o w n m e n u , c h o i c e s t h a t a r e u n a v a il -a b l e , a n d a r e t h e r e f o r e g r e y e d - o u t , a r e v i r t u a l l y il le g -i b l e . T h i s m e a n s u s e r s w h o a r e s e e k i n g a p a r t i c u l a rm e n u c h o i c e c a n n o t i d e n t i f y it i f it i s u n a v a i la b l e ,w h i c h l e a d s t o u n n e c e s s a r y s e a r c h i n g t h r o u g h t h em e n u s .

W h e n d a t a f l o w l i n e s a r e s e l e c t e d , t h e s e l e c t i o n i n d i -c a t o r s ( s m a l l b l a c k b o x e s o n t h e l i n e ) o b s c u r e t h ea r r o w h e a d s . T h i s m e a n s u s e r s c a n n o t t e l l w h i c h w a yt h e a r r o w s a r e p o i n t in g , w h i c h c a n l e a d t o u n c e r t a i n t yi n p e r f o r m i n g s o m e o p e r a t i o n s (e . g ., r e v e r s i n g a r r o wd i r e c t i o n ) .

F e e d b a c k M a n y m e n u o p t i o n s g i v e l it tl e o r n o f e e d b a c k a b o u t

t h e o p e r a t io n o f th e c o m m a n d a n d u s er s t h e re f o r eo f t e n t r i e d t o r e s e l e c t t h e m .

S o m e c h a n g e s t o o b j e c t n a m e s d i s p l a y e d o n t h e s c r ee nw e r e m a d e w i t h o u t i n d i c a ti n g t o t h e u s e r t h a t t h i s h a dh a p p e n e d .

O n s e le c ti n g o n e p a r t i c u l a r i te m o n t h e m e n u b a r , t h ep u l l - d o w n m e n u ( b e c a u s e o f i ts d y n a m i c c o n t e n t ) o n l ya p p e a r e d a f t e r a d el a y . W i t h o u t a n y f e e d b a c k t h a t t h em e n u w o u l d s o o n a p p e a r , u s e r s f o u n d t h i s d is c o n c e r t -i ng .

T h e s e p r e s e n t a t i o n d i f f ic u l t ie s a r e a l l c o n s e q u e n c e s o fp o o r H F d e si gn . G e n e r a l p r in c ip l es o f g o o d h u m a n -c o m p u t e r i n t e r f a c e d e s ig n h a v e b e e n p r o p o s e d b y an u m b e r o f a u t h o r s 23, w h i l e o t h e r s h a v e p r o p o s e d l a rg en u m b e r s o f d e t a i l e d d e s i g n g u i d e l i n e s 24,25. E x a m i n a t i o no f t h e d i f f i c u lt i e s f o u n d i n t h i s s y s t e m i n t h e l i g h t o f t h eg e n e r a l p r i n c i p l e s s h o u l d e n a b l e t h e i d e n t i f i c a t i o n o f t h ek i n d s o f H F c o n c e r n s t h a t t h e s y s t e m h a s i g n o r e d ( e . g. ,p r i n c i p l e s a b o u t c o n s i s t e n c y r e l a t e to t h e u s e o f t h eD e l e t e a n d B a c k s p a c e k e y s ; p r i n c ip l e s a b o u t c o m p a t i b i -l it y r e l a te t o t h e g r e y b a c k g r o u n d f o r t h e m e n u - b a ri t e m s ) . F u r t h e r , e x a m i n a t i o n o f t h e d i f fi c u l ti e s w i t h r e f e r -e n c e t o t h e a v a i l a b l e g u i d e l i n e s w i ll a l m o s t c e r t a i n l y s u g -g e s t s o m e d e s i g n s o l u t i o n s .T h e r e e x i st s , t h e r e f o r e , a g o o d d e a l o f H F k n o w l e d g er e l a t i n g t o p r e s e n t a t i o n d i f f i c u l t i e s . T h i s i s i n p a r t

b e c a u s e t h e s a m e d i f fi c u l ti e s r e c u r i n a v a r i e t y o f a p p l i-c a t i o n d o m a i n s - - t h e r e is n o t h i n g a b o u t t h e a b o v e d i ff i-c u l ti e s t h a t i s e x c lu s i v e t o C A S E t o o l s . F u r t h e r m o r e ,p r e s e n t a t i o n d i f f i c u lt i e s a r e r e l a t i v e l y e a s y t o i d e n t i f y b yH F e v a l u a t i o n a n d a r e o f t e n s i m p l y t o r e c ti f y ( p a r t ic u -l a r l y i f t h e i n t e r f a c e i s w e ll s e p a r a t e d f r o m t h e a p p l i -c a t i o n ) .

T a s k d i f f i c u l t i e sTh e f ina l g ro up o f d i f f i c u l t ie s i s t he t a sk d i f f i c u lt i e s . Th e sec o n c e r n t h e f u n c t i o n a l a s p e c t s o f t h e s y s t e m , o r th e k i n do f s u p p o r t o f f e r e d t o h e l p t h e u s e r s a c h i e v e t h e i r t a s kg o a l s . U n l i k e t h e p r e s e n t a t i o n d i f f i c u l t i e s , t h e r e f o r e , t h et a s k d i f fi c u l ti e s w i l l t e n d t o b e s p e c i f ic t o a n a p p l i c a t i o nd o m a i n , a s t a s k s v a r y b e t w e e n d o m a i n s . D i f f i c u l ti e sf o u n d w i t h t h i s s y s t em a r e t h a t t h e s u p p o r t o f f e r ed ca nb e u n n e c e s s a r y , i n a p p r o p r i a t e , i n a d e q u a t e , o r m i s s i n g .E x a m p l e s o f e a c h t y p e a r e a s f o l lo w s .U n n e c e s s a r y s u p p o r tC e r t a i n m e n u i t e m s ( e . g ., f o r d r a w i n g p r o c e s s b o x e s ) a r er e d u n d a n t b e c a u s e t h e f u n c t i o n a l i ty i s o f f e r ed b y o t h e rm e a n s t h a t a r e e a s i e r t o u s e ( e. g ., o n t h e p a l e t t e , o r b y u s eo f t h e m o u s e ) .I n a p p r o p r i a t e s u p p o r t T h e c o n c e p t o f a n A 4 s h e e t a s s u m e s a n e x a g g e r a t e d

i m p o r t a n c e i n t h e w a y t h e s o f t w a r e r e p r e s e n t s a n dm a n i p u l a t e s a d i a g r a m , a s i t c o n s t r a i n s t h e d i a g r a ml a y o u t a n d c a n n o t a l w a y s b e c h a n g e d e a s il y .

W h e n a l o w e r - l e v e l c h i ld p r o c e s s ( i .e . , a d e t a i le de x p a n s i o n o f a s i n g l e h ig h e r - l e v e l p r o c e s s ) i s c r e a t e d , an u m b e r o f a s s u m p t i o n s i s m a d e a b o u t h o w i t wi llr e l a t e t o i t s p a r e n t p r o c e s s ( e . g . , t h a t e a c h d a t a f l o w i nt h e p a r e n t p r o c e s s w i l l r e l a t e t o a s e p a r a t e s u b p r o c e s si n th e c h i ld ) ; s o m e o f t h e s e a s s u m p t i o n s a r e t o o s t r o n ga n d t h e n c a n n o t b e c h a n g e d .

I n a d e q u a t e s u p p o r t A r r a n g e m e n t o f a n u m b e r o f d ia g r a m o b j e c ts in t o

a r e c t a n g u l a r l a y o u t ( s o m e t h i n g u s e r s f r e q u e n t l ya t t e m p t ) i s a n e x t r e m e l y c u m b e r s o m e a n d e r r o r - p r o n ep r o c e d u r e a n d i s w i t h o u t s o m e f o r m o f ' T i d y ' f a c il it y .W he n loa d ing a f i l e , i t i s no t poss ib l e t o ob ta in a l i s t o fn a m e s o f a v a i l a b l e f il es f r o m w i t h i n t h e a p p l i c a t i o n .E n t i t y l i f e h i s t o r y d i a g r a m s a r e a l w a y s a r r a n g e d i nr o w s . A l t h o u g h h o r i z o n t a l a l i g n m e n t i s p o s s i b l e , i t i sn o t p r o v i d e d a u t o m a t i c a l l y i n th i s D E .

M i s s i n g s u p p o r t I t is n o t p o s s i b l e t o h a v e t w o f il es l o a d e d a t t h e s a m e

t im e , f o r t he p u r p o s e s o f c o m p a r i n g t h e m o r c o p y i n gb e t w e e n t h e m .

U s e r s o c c a s i o n a l l y w a n t t o d i v i d e a d i a g r a m t h e y h a v ea l r e a d y s t a r t e d i n t o t w o s e p a r a t e d i a g r a m s , w h i c h c a n -n o t b e a c c o m p l i s h e d e a s i ly .

I n h i e r a r c h i c a l l y r e l a t e d s et s o f d i a g r a m s , c h a n g i n gl o w e r- l ev e l d i a g r a m s i n w a y s t h a t m a k e s t h e m i n c o n -s i s t e n t w i t h h i g h e r - l e v e l o n e s i s s t r i c t l y d i s a l l o w e d .



7/9

Although keeping the diagrams consistent is obviouslyvaluable, users will often want to change lower-leveldiagrams first and have them temporarily inconsistentwith the higher-level ones.

While some of these difficulties (e.g., obtaining a list offile names) could be dealt with by the kinds of HF princi-ples discussed for the presentation difficulties, most ofthem concern the way the software does, or does not,support the user's task activity. The origin of such diffi-culties usually lies in inadequate task analysis duringrequirements definition. For instance, better observationof software engineers' working practices would haveidentified the occasional needs to divide diagrams in twoand to change lower-level diagrams before changinghigher-level ones. It is clear from observation t hat soft-ware engineers use dataflow diagrams iteratively, thatthey sometimes make mistakes in levelling, and t hat theymay need to repart ition the analysis. Similarly, the inade-quate support offered to rectangular layout and to hori-zontal alignment in the diagrams stems from a failure toanalyse in detail what support users require most toachieve their task goals.

The DEs give the impression of working from ananalysis of the diagrams themselves rather than of howpeople produce the diagrams. Thus the general pattern isthat the DEs are well able to support the copying ofexisting paper diagrams, but are less able to support thenormal process of producing diagrams. That is, design ischaracterized by frequent major amendments, by bor-rowing from previous designs, and by being both a bot-tom-up and top-down process. It is these aspects ofdiagram production that the DEs do not support as wellas they might.

In general, task difficulties will often be hard toidentify, because they require detailed unders tanding o fusers' task behaviours and thus will vary between appli-cation domains. There is, therefore, no large database ofHF knowledge to consult as there is with the present-ation difficulties. Moreover, HF has not so far been ableto develop a range o f clearly effective methods, tools, ortechniques for understanding users' tasks. All this makesthe accurate identification of task difficulties the majorproblem in HF evaluation. In addition, because the diffi-culties relate to the funct ional ity of the system, they tendto be ha rder to rectify than the presentation difficulties.C r i t e r ia l a s s e s s m e n t o f D E sThe relevant criterion given earlier was that 'naive usersshould be able to use each DE to complete basic editingtasks without encountering serious difficulties or makingserious errors.' Against this criterion, the DEs performedadequately, although not optimally. All users testedcould complete the basic diagram-editing tasks withoutserious difficulty or error. The worst problems encoun-tered were in 'tidying up' the diagrams to improve theirappearance, the lack of feedback from some commands,and the mismatches of task support with respect tonormal working practices.

A W H I T E F I E L D A N D A S U T C L I F F E

D i f f i c u l t i e s w i t h d a t a d i c t i o n a r yIt is not the intention to present difficulties associatedwith the dat a dictionary, for two main reasons. First, it ispossible to illustrate the evaluation without discussingthe details of the data dictionary. Second, the dat a dic-tionary was not evaluated to the same extent as the DEssince, as already stated, it was a proprie tary product andtherefore not easily changed, and it was in any case liableto be replaced by another product.

Suffice to say here th at there were no classes o f difficul-ties identified in the use of the data dictionary that werenot also found with the DEs, and the data dictionarygenerally performed poorly in the evaluation, in largepart due to the contrast between the database-orientedterminology and structure of the data dictionary (row,column, master, detail, etc.) and the domain-orientedterminology and structure of the DEs (process, dataflow,entity, etc.). Performance with the dat a dic tionary essen-tially failed to meet all three of the relevant criteriamentioned earlier.

C O N C L U S I O N SWhat can the case study illustrate about typical currentpractice in HF evaluation? First, that such evaluationscan succeed in identifying a wide range o f user-interfaceproblems while consuming modest resources. The r e p o r tof this evaluation was well received by the system deve-lopers, who found it contributed to the improvement ofthe design in an area in which they themselves wereinexperienced. The total effort involved for all personnel(including HF specialists, domain specialists, and users)was between three and four person-weeks, of whicharound one third was spent on the written report.

There is no fixed level of effort required for an HFevaluation. In any instance, the chosen level will be deter-mined by a number of factors. However, the less effort isinvolved, then the less confidence there can be in theoutcomes in terms of their accuracy, completeness, andconsistency. The authors would suggest that, for thissystem, the level of effort for the evaluation was small asa part of the overall development and was worth while interms of its consequences for the quality of the finalsystem.As far as the range of difficulties identified is con-cerned, there is clearly an important distinction to bemade between what have been called here presentationand task difficulties. They differ in a number of ways,including their identifiability, their generality acrossapplication domains, the HF knowledge to which theyrelate, and the ease with which they can be remedied.They also tend to differ in the aspects of performance towhich they relate. While both will adversely affect theresource costs required of a user in carrying out a task,the task difficulties are much more likely also to affecttask product quality. Identification of task difficultiesrequires either detailed knowledge about the applicationdomain (in this case, SSADM design) or good techniquesfor task analysis and representation, or probably both.

Vol 34 No 7 July 1992 449


8/9

Case study in human fac tor s evaluationIn this instance the knowledge of the domain was contri-buted in the form of the tasks the users were asked toperform, and by the specialist repor ts from the domainspecialists. Both o f these contr ibuted heavily to the iden-tification of task difficulties.It is worth pointing out that there is at least one type ofdifficulty that has been missed (or rather ignored) in thisevaluation -- organizational difficulties. These concernhow the system impacts the working structures and prac-tices o f the organization rather than those of the indivi-dual. They have been ignored here because they were notincluded in the focus of the evaluation as negotiated withthe client. Had they been included in the focus, therewould have been changes to the methods used and to thetypes of specialist knowledge recruited.Wha t consequences for system development arise fromconsidering the different types of difficulties? The taskdifficulties highlight the need for better requirementsanalysis in the software engineering sense, and the needfor an HF input to requirements analysis in the form oftask analysis. Requirements analysis is acknowledged tobe the source of many errors in system development26,and the importance of tracing and validating require-ments in specifications and products has been empha-sized 27,2s. Either a better appreciat ion of empirical studiesof software design29 or simple observation and analysisof SSADM developers' activities would have indicatedthe need for support of actual work practices rather thanof the top-down procedures implied by the structuredmethods. Many CASE tools fail to adopt a flexibleapproach to supporting working practices, and conse-quently they tend to be used as documentation toolsrather than as genuine support for the software engineer-ing process 3.The presentation difficulties, on the other hand, sug-gest the need for the incorporation of known HF princi-ples and guidelines into software engineering practice.There is a good deal of relevant HF knowledge which,for one reason or another, had not been used in thedevelopment of these tools. Clearly, ways need to befound to ensure that what is known can be recruited tosoftware engineering practice as easily and effectively aspossible.Improvements to requirements analysis, and the re-cruitment of extant HF knowledge, would both reducethe reliance on evaluation as a form of HF input tosystem development and enable HF to contribute inother ways.

The framework for evaluation summarized earlierindicated that evaluation can take place at any timeduring system development, although it has been symp-tomatic of HF evaluation that it has tended to occur verylate, and often only once, in development. A typical HFevaluation, as described here, clearly requires somephysical form of the computer to be present. For a typi-cal evaluation to have the potential for maximum benefitby occurring early in system development thereforerequires the adoption of an iterative approach, if not aprototyp ing approach. Indeed, evaluation is an essentialcomponent of prototyping, for without a methodical

approach to testing, the value of prototyping for trap-ping errors early may be nullified by cursory examin-ation. Methods for HF evaluation can systematize pro-totype testing so that testing scenarios are correctlyconstructed and testing is thorough. The need for amethodical approach to prototyping and requirementsvalidation has been emphasized27, and scenarios haveproved effective in requirements testing3.

The framework and practice reported in this papershould enable readers to take some initial steps in sys-tematizing and refining HF evaluation practice. Specify-ing the focus of the evaluation, and selecting methodsappropriate for the focus, given the context, are criticalactivities whose importance the authors have tried toemphasize (while acknowledging that they are currentlypoorly understood and underspecified). The client-centric focus reported here represents a departure fromprevious practice, which has viewed evaluation from avery general HF brief. It is important that the focus isclear and that, while it may express the client's concerns,it is expressed in user-oriented terms. The authors believethat their approach, of attempting to systematize andrefine current HF evaluation practice, offers greaterpromise for improving evaluation than more radicalalternatives, such as increasing the reliance on users'subjective experiences or the development o f quantita tiveevaluation metrics with doubtful construct validity.The final point concerns the integration of HF evalu-ation into the quality-assurance role within softwareengineering. HF evaluation is capable of making a signi-ficant contribution to quality computer systems. Just assoftware engineering methodology has evolved to addressother quali ty-assurance concerns (e.g., reliability), itneeds to evolve further to encompass HF aspects ofquality assurance. Incorporation of the kinds of evalu-ation practice described in this paper would be one movein that direction.

A C K N O W L E D G E M E N T SThis work was done while the first author was workingon a project funded by the Department of Trade andIndustry, UK. The authors thank personnel on theproject that developed the tools for their assistance, andcolleagues at the Ergonomics Unit for discussions andfor comments on an earlier draft.

R E F E R E N C E SI Gould,J D 'How to design usable systems' in Bullinger,H-Ja n d S h a c k e l , B (eds) Human-Computer Interaction - -INTERA CT'87 North-Holland (1987)2 N i e l s e n , J 'Traditional dialogue design applied to modernuser interfaces' Commun. ACM Vol 33 No 10 (1990) pp109-1183 C h i k o f s k y , E J a n d R u b e n s te i n , B L 'Reliability engineeringfor information systems' IEEE Software (March 1988)(1988) pp 11-164 L o n g , J B a n d D o w e l l , J 'Conceptions of the discipline ofHCI: craft, applied science and engineering' in Sutcliffe,Aa n d M a c a u l e y , L ( e d s ) People and computers V. Proc.HCI89 Cambridge University Press (1989)

450 Information and Software Technology


9/9

A WHITEFIELDAND A SUTCLIFFE5 Den ley , I a n d L o n g , J B ' A f r am ew o rk fo r ev a lu a t i o n p rac -t ice ' in Lovesey , E J (ed ) Contemporary Ergonomics 1990T ay lo r an d F ran c i s (1 99 0)6 Di l lon , A P 'T he ro le o f usab i l i ty labs in system design ' inM eg aw , E D ( ed ) Contemporary Ergonomics 1988 T a y l o ran d F ran c i s (1 9 8 8 )7 Whites ide, J , Bennet t , J and Hoi tzbat t , K 'Usab i l i ty eng i-n ee r ing : o u r ex p e r i en ce an d ev o lu t i o n ' i n H e lan d e r , M (ed )

Handbook o f human-computer interaction Elsevier (1988)8 H a m m o n d , N , H i n t o n , G , B a r n a r d , P , M a e L e a n , A , L o n g , Ja n d W h i t e f ie l d , A ' E v a lu a t in g t h e i n t e r f ace o f a d o cu m en tp ro cesso r : a co m p ar i so n o f ex p e r t j u d g em en t an d u se ro b se rv a t io n ' i n S h ack e l , B ( ed ) Human-Computer Interac-tion -- IN TERA CT'84 N o r t h - H o l l a n d ( i 9 85 )9 Whitef ie ld , A D, Wilson , F and Dowel l , J 'A f ramework fo rh u m a n f a c t o rs e v a l u a t i o n ' Behav. Inf. Technol. V o l 1 0 N o 1(1991) pp 65-7910 Ravden , S a n d J o h n s o n , G Evaluating usability of human-computer interfaces: a practical method E l l i s - H o r w o o d(1989)11 H ar tson , H R a n d S m i t h , E C ' R a p i d p r o t o t y p i n g i n h um a n -co m p u te r i n t e r f ace d ev e lo p m en t ' Interact. Comput. Vol 3N o 1 (1991) pp 51-9112 Go u ld , J D a n d L ew is , C 'Des ign ing fo r usab i l i ty : key p r inci -p l e s an d w h a t d es ig n e rs t h in k ' Commun. ACM V o l 2 8 N o 3(1985) pp 300-31113 Dowell, J a n d L o n g , J B ' T h e ' l a t e ' ev a lu a t i o n o f a m essag -in g sy s t em d es ig n an d th e t a rg e t fo r " ea r ly " ev a lu a t i o nmethods ' in Su tel i f fe , A a n d M a c a u l e y , L ( e d s) People andcomputers V. Proc. HC189 C am b r id g e U n iv e r s i t y P res s(1989)14 Wrig h t , P C and Monk , A F 'A cost -effect ive evalu at ionm eth o d fo r u se b y d es ig n e r s ' Int. J. Man-Mach. Stud. (i np ress)15 Jo rgeusen , A H 'Th ink ing-aloud in user in terface design : am e t h o d p r o m o t i n g c o g n it i ve e r g o n o m i c s ' Ergonomics V ol33 No 4 (1990) pp 501-50716 Molich , R a n d N i e l s e n , J ' I m p r o v i n g a h u m a n - c o m p u t e r

d i a lo g u e ' Commun. ACM Vol 33 No 3 (1990)17 Dowell, J a n d L o n g , J B ' T o w ard s a co n cep t io n fo r anen g in ee r in g d i sc ip l in e o f h u m a n f ac to r s ' Ergonomics Vol 32N o ! 1 (1 9 8 9 )p p 1 5 1 3 -1 5 3 5

18 D owns, E , C lare , P and Coe, I Structured Systems AnalysisAnd Design Method, application and context P ren t i ce H a l l(1988)19 Hi l l , R , Den ley , I a n d L o n g , J ' T o w ard s an ev a lu a t i o np lan n in g a id : c l a s s i fy in g an d se l ect i n g ev a lu a t i o n m e th o d s 'in Lovesey , E J (ed ) Contemporary Ergonomics 1991 T a y l o rand Francis (1991)20 Wilson , F a n d W h i t e f i e ld , A D ' In t e r ac t i v e sy s t em s ev a lu -a t i o n : m ap p in g m e th o d s t o co n tex t s ' i n M eg aw , E ( ed )Contemporary Ergonomics 1989 T ay lo r an d F ran c i s (1 9 89 )2 1 M o ran , T P ' T h e co m m a n d l an g u ag e g ram m ar : a rep resen -t a t i o n fo r t h e u se r i n te r f ace o f i n te r ac t i ve co m p u te rsy s t em s ' Int. J. Man-Mach. Stud. Vol 15 (1981) pp 3-502 2 T a y l o r , M M ' L a y e r e d p r o to c o l s f o r c o m p u t e r - h u m a n d i a -logue. I : p r incip les ' Int. J. Man-Mach. Stud. Vol 28 (1988)p p 1 7 5 -2 1 823 Su t c l i f f e , A G Human computer interface design M a c m i l l a n(1988)24 Smith , S L a n d M o s i e r , J N 'Gu idel ines fo r design ing userin t e r f ace so f tw are ' MITRE Corporation report ESD-TR-86-278 (1986)25 Gard iner , M M a n d C h r i s t i e , B (eds) Applying cognitivepsychology to user-interface design Jo h n W i l ey (19 8 7 )

26 van Assche, F , Layzel l , P J , L oucopou los , P a nd Spel tinex , G' R U B R I C : a r u l e b a s ed r e p r e s e n ta t i on o f i n f o rm a t i o nsy s t em co n s t ru c t s ' i n Proe. 5th Esprit Conf. N o r t h - H o l l a n d(1988) pp 438-4 5227 R o m a n , C C ' A t ax o n o m y o f cu r r en t is su es i n r eq u ir em en t sen g in ee r in g ' Computer (Apri l 1985) pp 14-222 8 S a i l o r , J D ' S y s t em s en g inee r in g: an i n t ro d u c t io n ' i nT h a y e s , R H a n d D o r fm a n , M ( e d s ) System and softwarerequirements engineering I E E E C o m p u t e r S o c i e t y P r e s s(1990)29 Curt is , W 'Em pir ical s tud ies o f the so f tware design p roce ss 'in Diaper , D , Gi lmore, D, Cock ton , G and Shackel , B (eds)Human-Computer Interaction -- INTERACT'90 N o r t h -Hol land (1990) pp 35-5030 Benner , K M a n d J o h n s o n , W L ' T h e u se o f s cen a rio s fo r t h ed ev e lo p m en t an d v a l i d a t io n o f sp ec i fi ca t io n s ' i n Proc.A1AA Conf. Computers In Aerospace M o n t e r e y , C A , U S A(1989)

Vol 34 No 7 Ju ly 1992 451

Documents

Case of Study in Human Factors Evaluation (1).pdf