View
222
Download
1
Category
Preview:
Citation preview
copy UNLIMITED"--) TR 90019
0Y)
AD-A227 864
ROYAL AEROSPACE ESTABLISHMENT
DTICIELECTE fl
0 D U Technical Report TR 90019
March 1990
A Subjective Rating Scale for Assessing
Pilot Workload in Flight:A Decade of Practical Use
by
A. H. RoscoeG. A. Ellis
Procurement Executive, Ministry of Defence
Farnborough, Hampshire
UNLIMITED
90 10 .22 164
UNLIMITED
ROYAL AEROSPACE ESTABLISHMENT
Technical Report 90019
Received for printing 5 March 1990
A SUBJECTIVE RATING SCALE FOR ASSESSING PILOT WORKLOAD IN rLIGIET:
A DECADE oF PRACTICAL USE
by
A. Hi. Roscoe*
G. A. Ellis**
SUMAARY
Llo~pite the many techniques deveýloped for tvaluati~ig pilot workload in
fliqht siubjectivv aa e.,wient by excporience-4 p;.lots is atill, the rtoost relitable
mtehod kby far. This report doscribe4 the deoign and developmert --with tha nelp
of practising test pilots - of a ten-pe..nt catini qca.'e. The s'cilo uses a
dociaic~n tree similar to that aosd by the Cooper-Harptoz handling Qualities seale,
dnQ1, bda~ed on tne concept of space cap~acity. ExawapIes are given of its use by
a larqu numbor of pilots in variou.a flight triala and vykload atudie3..I t
DoparttwntalReene F1 6
c~opyrivhm
C'anitroJ2@r UNSO Londonb1990
Foermly ur RAE, Wedfod#apd now with Pkrr~!anna Airway**Fortmetty of M~E, flodford, and now with British Aerospace~
MUtA ZHL=
'Ar
2
LIST OF CONTENTS
Page
1 INTRODUCTION 3
2 THE 'BEDFORD' WORKLOAD SCALE 4
2.1 Design and development of the 'Bedford' scale 4
2.2 Description of the scale (Fig 1) 6
3 PRACTICAL EXPERIENCE WITH THE 'BEDFORD' SCALE 7
'. DISCUSSION 11
5 CONCLUSIONS 13
Refr~re ives 14
I.~lustration .Figure 1 (in text)
Report documentation page inside Lak cover
j-,
A-1k
tI-
3
1 INTRODUCTION
In 1969 the increasing importance to flight safety of changes in levels of
pilot workload generated by new operating techniques such as vertical take-off
and landing (VTOL), low visibility landings, and reduced noise approaches, then
being evaluated at the Royal Aircraft Establishment at Bedford, resulted in a
greater interest in assessing workload during flight testing. Unstructured pilot
opinion recorded during flight, or, more often, after flight was the accepted
method of evaluating workload at this time. The possibility of obtaining
misleading information because of bias or of pre-conceived notions about workload
levels - a recognised problem associated with subjective techniques - resulted in
a programme aimed at developing a complementary but more independent measure.
Following a detailed survey of available techniques, monitoring of pilot's heart
rates was selected for further evaluation.
After some 5 years, during which considerable experience was gained of
using test pilot's heart rates to support their opinions of workload, it was
decided to improve the method of obtaining subjective assessments by employing a
specially designed rating scale. A search for a suitable scale for use in flight
was unsuccessful: most research on subjective ossessment by pilots had been
concerned with aircraft handling qualities'1 3 . Although some scales such as that
designed by Cooper and Harper3 have somotimos been used to rate workload they are
not ideal for this purposo. As Gerathowhol4 pointed out, ". . . subjectivo pilot
ratings of handling qualities, as accurate as they may be in rgoard to control
desiability or difficulty, do not contribute to workload determinations, aineo
they are only loosely connected to task dmatnds and pilot rosponseO. The
decision was made, therefore, to design and develop a workload rating acale at
RAE Bedford 'on the back' of current flight teatinq and with the help of
praetiainq tost pilots.
As well a3 dosigning a pilot workload rating scale it esoomd 3ensible to
defino workload. A toview of the literature revealed A plathord of doe(nltina
based mostly en workload as a 4et of flight usk domands, a* the oftott roquird
to satisfy those demands, ir as th4 rosult* of that effort - porotman. . M#4y
of the d4flnitioen apdafw espvlicatod And/or unrealistie in thd 2onteox of ral
Utlttg a Is and fto a*o& obtained the Vlows of sum* 3O
tilitaty and kAirlino pilot3 and conciud*J thaL mtr~r thian 001 of Pro easioual
pAl*tn think of workload in totrm of effort. this is also an inr4protatioa that
agreod Woll wiuth the 1nfluee of sueh individual facotos as natural ability.
. ., ..... , . .
4
training, and experience on the piloting task. There is evidence that the
failure of pilots to perceive the demands of the flight task correctly has been a
causative factor in several accidents; it also seems likely that workload levels
tend to be determined by how a pilot assesses the flight task. With these
findings in mind Ellis and Roscoe proposed that a slight modification to the
definition of workload used by Cooper and Harper in the introduction to their
Handling Qualities Rating Scale3 would be appropriate, namely: "Pilot workload is
the integrated mental and physical effort required to satisfy the perceived
demands of a specified flight task".
2 THE '5KD10RD' WORKLOAD SCALX
2.1 Design and development of the 'Bedford' scale
The initial ob jective was to design an interval scale that could be used to'
give ratings in flight during - or immediately following - highly demanding
piloting tasks, and that would result in absolute values of workload.
As Ellisa' observed. "The use of rating scales results in the allocation (:4
a numerical value to the quantity that is being muasured. Not unnaturally,
researchers Wish to use statistical and mathematical proce.sses on the nube.ýS 1-
obtained, and so moot of the rating scales that have been devised hAve b~een
intended to be linear". However, McDonnell'. in discu!2sinq the rating of
aircraft handling qualities. referred to the difficulty of ac:hiiwinq linoarity
with ordinal and adjectival scales. A~nd 11e55g later cq=-i tod: "The rmajority
rating scales in existence have two thinqs in coqr=ncn tho'y a~eiio boh or~ia
adjectival~ in natuWo. Futhormero, the results of vaticmý *thior a*rty
3tedI31'ý' aiAed at developing non-adjectival lineat rating vsleoo woto not
encouraging.
It beestee obvious that an ordinal, atijgctival tratingq Ocalf) wjull 12.0 r-4r
41PProPriate for dovvlotwant at, a flight teot cintro 14cknq dtciies? !ot
laboratory exptients. An ordinal scalo tot Aircraft handlinq qqhltiog tnp
Cooper-Harpor scale'. wag allready dovolopod and Ostahblsioed; it. was e~aay *t4 ;A'k-
andi vas widqlv accoptod awangst tout pjl.i~ta and onqlnootm. at h t tad
goo6d adense to tt~y to design thO watkiad scalet usiti a gichla. di~siV. t.,de~
tree vith appropriate deoeriptott;.
The first daeaign, a nineo-point fieale. used doriptott baged- *tn litfott
such as: Pilot effort not a tactor tot doý!re4 pezort*Anca - rating 14l n~-&
partottaaneo requiros ~eoaopilot effort - ratinq I., adequate mac
requiros exrenaive 010t at fott -rating Sý intousivo pilot eflert is rV4,Ltred t"5
L~
5
retain control - rating 8; and finally, control will be lost during some portions
of required operation - rating 9.
At first it was not obvious whether the scale should bc absolute or
comparative. In other words, should it try to cover all possible workload levels
in all flying tasks? Or should it have the more limited aim of acting as a
comparative measure between the workload experienced and that which could be
considered normal or reasonable for the task in hand? It was therefore decided to
construct both types of scale, and then to decide which of them would be the more
appropriate in various circumstances.
An interesting finding from the questionnaire study by Ellis and Roscoe 5
was that pilots find it convenient to think in terms of 'spare capacity' when
considering their levels of workload. What other relevant but secondary tasks
can be taken on in addition to the primary flight task? For example, when the
primary task is an instrument approach in bad weather how much spare capacity
does one consider is available for monicoring the actions of the other pilot,
looking outside the cockpit, listening to the radio etC? The higher the workload
generated by the primary task the less capacity there is for these secondary
tasks. Pilots seen to find it a relatively simple matter to judge how much more
they could do even it there is no requirement to do so,
These findings Auggested that descriptora incorporating the concept of
spare capacity wo'uld be of greater value than roeerence to effort. The scales
were alao extended from nine to ten ratings.
In addition to the concept of spare capacity the 'absolute' scale also
referred to arousal, time, and fatigrue. Those now descriptors Included;
Low workload, plenty of tioe and capacity to cotwleto all tasks At a
moderate arousal satet, level of of fort could be Maintaine~d for severial hours.,-
voting Z,
* Modertetc workload, all pritmary And secondAry taksks within pilot capacity.
* but fairly high arousal state nee~ded tiring and fatigue likely after
1 -2 houra; - vating 3. Very hig;h work-load, only more irvortant secondary tasks
co:Pploted and then only infrequently; - rating 6.
In view of the present concernt about uindoratdusal and uftddrload it is worth
noting that the lowest wborkload de3riptor In this scalA read: Very low workload,
few taoks for the time available, dooe risk of boredom: rating 1. ThIS WAD
* shortly atended to., Workload teo low, too much spAre capacity# danger of
r,.
. A! Vt
6
During construction of the scales it quickly became clear that any
'absolute' scale that attempted to include the whole spectrum of workload levels
experienced by pilots would be too coarse to be practical. Certain tasks such as
gun aiming or landing in adverse weather were always concentrated within a few
seconds and were always high workload. Others, such as en-route flying, could be
sustained for many howirs. To place them on the same scale was of very limited
value. Also, pilots comments and ratings of workload made during various flight
tests in different types of aircraft, showed the difficulty of obtaining absolute
values. It was found that pilots liked to compare their workload to some form of
baseline, usually to previous experience.
Effort was therefore concentrated upon developing a comparative scale that
would help to answer the important and practical question of whether the workload
is appropriate for the primary task under consideration. Subsequent experience
of using the scale h4s proved this decision to have been correct (see later).
The need for concise descriptors in an adjectival rating scale was high-
lighted during the evaluation of the absolute scale in fight. it soon became
apparent that the introduction of the additional factots of aranal and ratgue
complicated the scale unduly. After same development in flilht and furthe-r
discussiOn3 with pilots the present descriptors were introduted (Fig 1). these
were readily acoepted and in a short time considered by Ovdford test pilots tQ be
quite adequate for the purpoae of rating worklead.
a.2 a ecrpto of the scale (Fig 2)
The pilot starts his dei~-al@process4 at tho bettýs. ý5f
the decision tree, which consists of tncee quo-,stions roquirinq Yk*A VOi .1in order to proeend to the dQcaiptionz of different lavejz 0 w .d, ThR
d*soriptora or* of increasing levels of workload aesoiaiated with rotiftqt of to,
10. Half ratings Art alloved thereby incret;ng the gsonitivity cd tho eug,
this bee"*e particu~larly dasivablo at tho lower voitkhvAd lov,*11. et in~.4!y t~l
ratings between the 'decision' groups voto not sought but aa mAny pilotsý sgoqt--
to find it difficult to decida betwevile # 'y anid *tbol o tar t~ voWlo5 Aakigad t -
tocy without t•ductlon?* A rating of V. bacat accptablo.
It is most imtortant that the flight t~ak to bo Pdtod 4hould tv uell.
defined And/or the period of time oveir which the Isseatoent Imad@ be nade
with reasotuable precision. The workload Weing assaesed is that involved In the
esecution of the prieury task; any additional ta~ks - such as tsanitozit other
aet *boee s - must ba Includad as part of the pitot's spare cacity.
7
Workload Description RatingDecision Morklood Insignificant.
Treeg 1 zz:::capcity for all desirable saditional 3yes tss
Insufficient spare capacity for easy attentionwa orla to additional tasks. 4L.wokodNo irReduced spare capacityv. Additional tests cannotsatisfactory ~I be given the desired &mus~t of attention.
without i ___________________ ___
reduction,? sp: are capacity. ravel of effort allows
yes
Very itutlesire a pain t y.in but Satatesanc, of7
effrtto hepr n~r asklo to quelpstf~ien. efot.1
toleabl fo Veryt Atuworload wansat Iit als mpefe spire~n capact)
ma doubts as to ttio ta to Ra1@ van ?#Vol ofneffort.a9
, V41 it ,,,cib, N ?o III rest abocadow. pilo attc U4410 ts @y fcituanta
Th e noa noa .. wraesio o thertadtin Scal va3ft oirt~ again.412daPi~
ThO n -theHS11rror acao i.-jumo tao-af triat" handin q~trial an~dsrod
tho AVrdqm .aof Imin a att itiltoad rooamp to vtv he atebit ake-ff otsratm" of
idtw itto piat t $hoaltu tyiatle01.*t~a-Vt atta oaedtaQT
At th o ftdtCP h f h on fbcfii ibce h odto tr"41iý, donw-Ads o apro-et.anq4. s tovooionl flingspod. i aptdah~ath onaA*gaulyrtto otoatPsto gii
Th s~o d ud t sss hndi% uliit ~d h
Eleven pilots rated their workload and had their heart rates recorded
during ramp take-offs, the ramp angle was increased in steps from 60 to 150 over
the period of the trial. Workload ratings and heart rates showed good agreement;
and both ratings and heart rates confirmed that workload levels were not
increased for greater ramp ar.gles nor for night take-off s. These workload
indicators also demonstrated that levels of workload are higher during the more
conventional short take-of fs (for this aircraft) from a runway.
The workload rating scale was used extensively during a trial to evaluate
Economic Category 3 approach and landings 12. Pilots' heart rates were again
recorded to complement their ratings of workload. The technique involved an
autopilot coupled approach to a decision height of 50 ft for the HiS 748 and 60 ft
for the SAC 1-11 aircraft at which height the autopilot was discoimectod for a
manual landing if the runway lights were seen. If the lights were not seen by
decinion height a go-around Was made.
In addition to rating the final approach with autopilot. and tr.e manual
landing, ratings were given for the very short term workload a aouiatoc with
making the docision.
It, late 1902 the Bedford scale and pilots' heart rate reapon~oe, wore u-e'd
tO a330ss workload in flight during crew eomplomat ce ~~nof thf&MAe 1413 Post flight quoastionnaixeaoa3~ne tho in-fligit. klatak ,ýnw4m
1rom the three toami (of twio pilots) who *eAch flew t~hro etqý,L at ~ e l~'
achqdeeao around a teireuit of threet hiqh initenoity pr tin-
Paris Charles do Gaulle, and Amserdam -W I).
Wrkload tatingsA vot's 2bohind ifvt Itth pilokp An4 t ar,
flight ob*Qrvor, an vtirbaol roquesz tiad llight xlqt~.a4 re kttý ý r~
biy 0eants of steall keybv!4rit fitted to tho rottrýl "lu". afS to tito
clip board. Mkiinga, wh!ich vere plKt-44 auot alyo~c hq-ari tftý
at tho tio* of the roquoet. were ated ttinj kio a o"1aktL'_1r40-
tequiests were woro froquotit dehnqi high rgikloki fgeat9 4>5f fli~hit 9Aýh RA "
takq-of I and Iinlt-loi Olv* the prcoeah a#4i lakndiul4. and vhen iie d !§*1rcltý
flight iallutos and eteoe aur4
pilots Vore DnsttuctO4 int tne ugo of tho relined zgal' beoloe 1'hp - r~s
atirted And, in partittalar, were askiod to eohi'i'sr theit utkowdfit the
pitevioul )0 A. All six pilott And mott Ot the !1lqht ýU10tvet!A fUnýUt thoesr;1
#6asy to irtot thero was no ovitiepee that qiving riatlvq;a ih fli-hI iitua r i4ed
t4e piloting task and only rarely was A iratitil ddlayed tý.-Y the fiqb dqemah4S.
Wi.t kationge uosa not used durlhq this trial aud thai~h t.ho seur.1tivity o~the
7 9
scale was consequently reduced it did not appear to influence its value for
certification purposes. In fact, the ratings obtained during the trial were
considered, overall, tc be of considerable value; and there was also surprisingly
good agreement between ratings given by pilots and those given by most of the
flight observers.
There was a reasonably good relationship between pilot's ratings and their
heart rate responses; disagreements seemed to be due mostly to the failure if the
pilot to rate the entire period under review.
Lidderdale'4 used the Bedford scale and heart rate recordings to assess
crew workload (pilot and navigator) during the evaluation of low-level high-speed
flight in a supersonic tactical fighter aircraft. He reported that "... the
aircrew understood the scale readily and whilst it was sufficiently comprehensive
to cover 411 circumstances it was easy to remember and small enough to be carried
on the flying suit knee-pad"t At pro-planned times the navigator, having
recorded his own rating of workload, would request a r4tir4 trowa the pilot. Beth
pilots And navigators veported little. difficulty in giving ratings3 in flight. A
pos,,t-flight.esat technique. basedof on the Analytical llierarchy PrnCess3'
was U40d to analyys paired compaisons, Liddardalo. in reporting a high
correot.'ion twetvon in-flight ratinls and the Post-flight s snt.oevd
that the farmert ttchnilque was eaieri to useo And mere p;4ctieal for u4O in Aft
;VPert4iOnAl ttial tpertrnA~l etahcto)
Thet Pedfttir'4l(cal w43 uNe" tth succes by Ha Mir an4 RluWelV t
r12l0Kad in arty hlelicpkte pilmts 01n94ge4 in vAtWr4nttflight taskalt 4n4 by
durinr f-.0iAvo rkgatiii i intot tho level a trkoa andI Operstnhq
n.by n4lte.ptoc pilott iValv an Nrth sa 4l1pltfre
W&Aggos atin revq ugv/d the t-9izJVQ -aslo 1c§t~hqt with heart tqr'-
"§~ ýtvo 'ce ho 110voe Mi o~Pilot workloamd 40onerated by toW adflhtC;*d
tecnolgy oetvc~ 141 with th4Oe qgenerlatd by the earlier lkeln" W)t Thit.
liritatolie Airways6 gtasy Vgv~afW -ariout duritng routine pastooener flights in%
rop drig hich lltftepiltsu fOUnd no !diffic-uity in qivintq tatings. at the Oh4
Of 14WR pe 't~iUlat figIht Oik-0 Or sgbs-Pha*e Orfnees. & IWsprienxed fih
ak*,±etver, Usingq 11W Ledfod sesleq. galgo atod the 4iffe'lent flight taskS. Viye
piats rtiipsedin tho *tu4y. three where etftitoted' on tho 0131StAa then,
afto e to type. on the #'G1 tea Pio ve Mnitered of both aircr:taf,
""a they auattlthtd every 4kM nrthiz botween 97) and (061, Aoth workload ringsa~d v~t tay, -etp~lw*SWO44tat~. tlw:Wbftttft ''' levels .at wvrkl'td jto
VII
* 4
A-,
10
lower on the 767 than on the 737. These measures have also distinguished quite
clearly between the different levels of workload associated with flying the 767
in different modes: hand-flying with raw information, hand-flying with flight
director integrated with the flight management system (Ft4S), and with autopilot
and autothrottle.
An extension to this study is presently under way in which workload levels
generated by flight failures, emergencies, and abnormal operating conditions in
the 767 are being assessed in a flight simulator. Both ratings and heart rates
for normal flight in the aircraft and simulator were used successfully to
demonstrate the value of the simulator for this type of investigatiort '.
Practical experience of using the workload rating scale in several t'.ght
trials showed it to be markedly better than previous methods of obtaining pilot
opinion, and though it lack'd some sensitivity - otipecially at the lower leveig
it was becoming well accepted by pilota and by tesearch scientists 414ke, never-
theless, in 19B3 it wa-i decided to carry out a series of *.lights to domn~tstrýt'
the ability of thit rating scale and of heart rate tespons to d~~is
between four short flight tasks having. theckreticdaly. throo dittornt lv-vtýýZ
difficulty. A N81S twin 'businass-3etl was use0d for 'ne! tza ittntv
*#tporioncod pilots flow a total of is 3equoncvs. 94ch voquomonee f:5 -e1tv
ta A 3 6 0 " turn in 2 min at cvngtant altitudo, :A$. and Ygtb oq tuu?%.
(bl A 360" turh in ý min with 4 a ixutno~u4 t~ao v* '00ft.n
tt a constant XAS and Cato 0f tutu.
(C) A 36V~ tufn io : twi Vith 4 04Litoatt4 O t' 011twao.l'4
folowed by 4 tvfae 40 turrn In Z nib Vitih a f
2000 ft at A eofistaft IMS and tato of turn.
id 1%30 tiurn in wi with A ziwk~lahltun 'W Iue 9c~
tach Soqeene was fliswn 4t -A 4ato tigeht in til*at A'tvpte m~ fktsl
pefeimne ws anteedby tmefý -in $' "'Ut ut a§ it 9'1," ~ ei'~
that @"dh pilot was deterMined to "Cet tioll im, the eoea
this vat discootinu~d. It vas tfhe iotwention to. Vaty th ore it% 6hi0% tthe k46'Aik
"ter flOW6 - e~t Oatatiotally, it %oat ouch poreQ ctnvouient Io fl-Y oath ~$Lit the bcdar 1-4 with tht? I itt task beIA4 reV@Ate4 ikk t49 000~ 4-f tMe t~h
6"ert MOMw wore ooeotdad throughout Oat~h 044UPftV 4 an wserk1nKad sativ4-1 vOte
t#4u~ead Atter oath tlilqht task. fthre pilot* Clow the .s1~uthro# tvito. 'thOtSr
Pilots not curtear e the 126 veoe give" at. boat 30 ai f~aillativatien0 beIO-t
being asked to rate the tasks; similarly, those pilots unfamiliar with the rating
scale were given a full briefing bef.'jrehand.
Results were highly encouraging and demonstrated that for 110 of the
12 pilots both workload ratings and heart rate responses were able reliably to
distinguish between the different tasks. In addition there wasa very good
agreement between ratings and heart rates for six of the 10 pilots and reasonably
good agreement for the other four.
The meAn ratings and range for each flight task were as follows:
Task Mean Rating Range
1 4.8 3-6
2 6.1 4-7
3 S. -0
47,0 4-a
1 5.03-6
0n thezortical grounds it wa.s ques4tiotiabt~ whether t~alk -31, which lasted
twiceo as long as the othter throe, would be rated lower Than taýýk - 4C In the
QvQ5It, task - 3 w4&. -4ctad higher thgn teak - 4 on thtoea occasions. lower on' fivo.
An-4 the agrw cst sevqn occasions. AlIheugh piloti were askted to rate the enttse
tastk, half of the pilots gave two. ratings tat task - 3.
Ovoza~l, kthere w4As no- ovide@n-tv of le~faq igoad roduced wothlee4d twe
t4ask - I flowft 4t tho #t4!.t end 4t the end "t the aeqtwsiee. Similerly, thtv*t
pia§ w04 fle6, the aewee oa n:f two separ~at ottvttna 'lid sn.j ow " on vidonce n3f
vorkto-gtl r-kd!Cti-o.n with iiOet 4txrailiarity.
4 DISCUSSION
th# lkd--ferd s'±atg haN now bieen int up 9#1 "toe tha%0y~k n t & h
4etiilt-a prinnaly !fo uge by ten11 PIl&t hj bQeen Vdsontll r lti
Pilts cvi hlloptot plota,. atd alirline0 r110411. The idea 04 epto
capaity apals tgg tst ilotsh eto eprt nlat it, helps thgo ttO arrive at at
fl~ltptiae r~in~jwithrelaive siath'is i stoaryvla~ewe fr
r~io th s*e~PttracttIte. stints a4.Th-it to bkecI0 Isalliaf sit\ý theb scalg
resattbiyq~ely to'. ilos'then s4ers tN think *ftly In tvtft Of ntU#6ers
witoutse~enc tothe Aactal taflsion Itre.
the a~ttia~e t fwng C t* use a rtto itdjSale diarfing or sbortty 4fter,
a Tk**a#d0t-j flight task hAt been decssssotratod 04.4%V tines durift the W0 yeata
especiilly durin g log flight sectors tgq~uliti# tony ratings.4 P4st (11gMt
tatiq4 -t ws even s th*m assistiktce ot video rocrtdibqS - Oe.Utb lossb tvllablve.
r' Rs;
4'. . .. ..
12
The authors are unaware of any other workload rating scale that has been
used as extensively for assessing workload in the 'real world'; nevertheless,
there are a number of possible shortcomings.
Several authors have underlined the importance of sensitivity and diagnos-
ticity, others have stressed the multidimensional nature of workload, and rating
scales having varying degrees of sophistication have been designed with these
issues in nirnd 2 0 - 2 2 . A subjective workload assessment technique (SWAT) developed
by Reid and his co-workers 2 3 , 2 4 considers workload in three dimensional terms -
time load, mental effort, and psychological stress. Hart and her colleagues25, 2 6
have developed and refined a multi-dimensional rating scale consisting finally of
six subscales, namely: physical demands, mental demands, time pressure, own
performance, effort and frustration.
Certainly, in many situations it may be quite important to be able to
analyse the reasons why workload has changed. The Bedford scale does not have
the power of diagnosis but in practice, on the rare occasions when diagnostic
information is required, post-flight discussions with the pilots - especially
when their beat-to-beat heart rate plots are used as an aide memoire - are
proving to be of considerable value. And, in most cases, assessment of overall
or global workload is all that is required - for example, during workload
assessments for the purpose of crew complement certification.
The lack of sensitivity at the lower end of the scale was at first thought
to be a disadvantage, but experience now suggests that it is unrealistic to
. strive for a high level of sensitivity. This is particularly so in view of the
variations in subjective evaluations between pilots and, occasionally, within the
same pilot from time to time. In addition, the cost effectiveness of modifying
systems or procedures to correct for small differences'in satisfactory 'otkload
-". .is questionable.
The nonlinear nature of the scale, although making it difficult to cnar:y
out statistical treatments on data, does not seem to cause any problem in
practice.
The decision to develop a scale giving relative - rather 1,han absolute
valu;es o.f workload has not appeared to pose any problem from the practical point
'o view, And it may be questionable whether absolute rating scale, are really
recessary. Lidderdale14 , in comparing in-flight with post-flight worlload
assessments, w!'*Le: "It is possible that all assessments of workload are made
trom a baseline of comparison with other elements in the flight and, if this is
the caoa, all ra~ing methods may be relative."
I
. . ' " .. . • • .
13
Examination of several hundred workload ratings given -luring a variety of
flight tasks or flight phases together with attempts to correlat~e katings with
heart rate responses have resulted in three important findings. Firstly, as
suggested by Ellis 6 in 1979, it was not possible to compare ratings for different
flight tasks; for instance, ratings given during the take-off could not be
related to those given during the approach and landing.
Secondly, although reasonably consistent, ratings were highly individual
for each pilot; unless some normalisation procedure was applied to the data - if
sufficient data were available -each subject pilot had to be considered as his
own control; a sirni±ar but more marked idiosyncrancy has already been identified
for heart rate responscs25.
Finally, a goo~d correlation be,.o;en worklead ratings and heart rate
responses for the name flight task tqý.s apparert in about S0% of pilots bk~t one in
five did no, show any agzeement. The veasot- for this lack of agreement has not
always been identified, s.c'meLimej it was bec-ause a pilot i4ss not integrated his
workload over the entire period of interest, but there is mnoie evidence that it
may be related t V~.- nature of that pa-licular indivietual's heart rate response.
5 COZICLUSXQNS
The B~edford workload rating scale was designed for uae in the 'real world'
of practical flight testing some 10 years ago. anrl pilots have, without
axception. found it easy to use in flight. During tho past decadde, despite the
shortcomings rotortted to above, its value has b~een demonstrated in several flight
triala - especially when ratings have been augmented by recording the pilots
heart rate.
Rlowovat, unlike the studies errtiod out an some other rating
scals ~the Vvdford scale hasi not boon subjected to a critical evalu-
otion in contrtilled labor~tory expoariinenta. Nevertholonot. the authors beliovo
tho #cale in ita present form is quito suitablo iot assessing workload in n~st
3rs~o ituAtionj.
41.
14
1EFZRZRNCES
No. Author Title, etc
1 R.P. Harper Pilot evaluation of handling qualities.
Ninth Annual Human Factors Society Meeting
(1965)
2 J.D. McDonnell An application of measurement methods to imrrovv
the quantitative nature of pilot rating scales.
IEEE Transactions on Man-Machine Systems
10, 81-92 (1969)
3 G.E. Cooper The use of pilot rating in the evaluation :,f
R.P. Harper air~craft handling qualities.
NASA Technical N6te TN-D-5153 Washington DC
(1969,1
4 S.J. Gerat~hewohl Definition and measurement of perceptual and mental
workload in aircrew and operators of Air Force
veapon 3ystemi.
In AGARD CF-181 Higner mental funetioning Ln
operational aystem5 AGAP.D, Paris (1976)
5 G.A. Ellis The airline pilot's view of flight dock werklal1:
A.R. Roscoe A preliminary study using a quostionnaire.
RAE Technical~ Memorandum~ FS(IBý 465 i)
6 G.A. Elisa Subjuctivo assa~sment. pilot apinion momiuroi,
In: AGARL'ograph No 233 Ao.5coe A 11(d
Aasfsosinq Pilot Wokload, A"AKD. V~rls ti9Th1
1 0. McDonnell Pilot rating tochniquots for thQ edtititaion Andi
ovi~luation of hvidingq i~gtito.4.s
AVFF)J - TA-60-"6 W;ri4Vatwraor1 Arlh, Oh;D tl(
8 I.A.A )I08s Ron adjaotiva4 ratinq ticalo.s In humtAn roapwi
MUwtAu Factors. 15, 275-280 (1973)
9 D.A. SPYkOr DQ~alopmot~n of tachniquot tot #*a urifiq piot
UK"A elk-k8$iF, wahingto" DC (19711
7 .>
15
REFERENCES (continued)
No. Author Title, etc
10 J.M. Rolf Evaluating measures of workload in a flight simulator.
et al In AGARD CP-146 Simulation and Study of high workload
operations.
AGARD, Paris (1974)
11 A.H. Roscoe Assessing pilot workload in flight.
In: Conference Proceedings No.373.
Flight Test Techniques, AGARD, Parin (1984)
12 A.H. Roscoe Pilot workload and economic Category 3 landings.
In: Conference Pre-prints Aerospace Medical
Association Annual Scientific Meeting,
Washington DC (1980)
1i W.A. Wai .right Flight test evaluation of crew workload.
In AGARDograph No 282 Roscoe A H (Ed)
The practical assessment of pilot workload.
AQARD. Paris (1987)
14 I.G. Lidderdale measurement tt aircrew workload during low-level
flIght.
!n AC'RDoqrs.n No 282 Roscoe A H $Ed)
The practical aass3mont oZ pilot workload,
AGAHD, Pa-ii (198')
"i5 T.L. $Sdty Th-i atialytical hierarchy prcess.
Heiraw-Hl1X (1980)
16 H.C. Muir The asts,=ýont of wuorkloari in helicopters.
R&. El-woll In., Procae4.ýngs of R~AE/AGARNDCIT Sympo~itz -
The prwical maeo-imnt of pi}o. workload.
Cranflic (198f)
17 P. |Harn'0 Pligqht d.ck environmont and lorkih.g in North Sea
A.C. reobor helicopter Opqcataona.
"La tonforenco Nbstracts Aerospace ModicAl
Agaoeiation Annual Scientific Hooting,
W4Ajhington DC (1188)
%%
: 19 A.fl. RoSceN The i.ct of nw technotoqy en pilot vo~k~oad.• U. •S, Or~eve ,,AE "Te.hnatcal Pap~er C.&•• (1986)
-Ii,•-••::.
16
REFERENCES (concluded)
No. Author Title, etc
19 A H. Roscoe Assessment of pilot workload during Boeing 767
B.S. Grieve normal and abnormal operating conditions.
SAE Technical Paper 881382 (1988)
20. J.H. Skipper Evaluation of decision-tree rating scales for
C.A. Rieger mental workload estimation.
W.W. Wierwille Ergonomics, 29, 585-599 (1986)
21 G.B. Reid Development of multi-dimensional subjective
et al measures of workload.
Proceedings of the International Conference on
Cybenetics and Society, 403-406, New York (1981)
22 S.G. Hart Development of NASA-TLX (NASA-Task Load Index).
L.E. Staveland Results of empirical and theoretical research
In: Human Mental Workload, Eds P.A. Hancock and
N. Meshkati, North Holland Press, Amsterdam
(1988)
23 G.B. Reid Application of conjoint measurement to wo-kload
C.A. Shinglodocker scale development.
F.T. Eggomeir In: Proceedings of the Human Factor3 Society -
25th Annual Meeting (1981)
24 F.V. Schick The use of subjective workload o3sessmnt to? nquv_
R.L. Hann in a complex flight task.
In AGARDograph No 282 The practical asseisimot iZ
pilot workload. Ed A.H. RoSCoe, AGAIAD, Vari3 (097)
25 H.R. Bortolussi Measuring pilot workload in a motion base traintr.
U.N. Kantowitz A Co0Parison of four tochniquos.
3.G. Hart Proceedings of the Third Biannual Symposlum on
Aviation Psychology, Columbua, Ohio (1988)
26 W.W. Wierwill* Deciaion-trao rating dcalo. for workload ostlmntid:
J.H. Skipper Thema and variations.
C.A. Riegat Proceedings of 20th Annual Contfoonco on Manual
Control, Sunnyvale (1984)
27 P. Tsang Cognitive demands in aUtonation Aviat,
W.W. Johnson Space &avi.on Hod, 60, 130-5 (.1989)
..I• ,. .4
REPORT DOCUMENTATION PAGEOverall security classification of this page
E - UINLIMITED
A\,: fir a, possible this page should contain only unclassified information. If it is necessary to entcr classified information, the boxaii' xc ni'st be ma.ked to indicate the classification, e.g. Restricted, Confidential or Secret.
I DRIC Refet~eicc 2. Originator's Reference 3. Agency 4. Report Security ClassificationiMarking(o be added by DRIC) Reference
RAE TR 90019 UNLIMITED
I S DRI( Code for Originator 6. Originator (Corporate Author) Name and Location7672000H Royal Aerospace Establishment, Bedford, UK
i '. Sponsoring, Agency's Code 6a. Sponsoring Agency (Contract Authority) Name and Location
fitle A subjective rating scale for assessing pilot workload in flight:
a decade of practical use
For. (ltI 1 ranslations) Title in Foreign Language
7b. ilFr (Conference Papers) Title, Place and Date of Conference
MA.\,,ix1. I Surname. Initials 9a. A'jthoa 2 9b. Authors3,4 .... 10. Date Pages Refs.March I IRoscoe, A.H. Ellis, G.A. 1990 16 27
S(,,trac{ .,mbei 12. Period 13. Project 14. Other Reference Nos.
Flight Management 6
I:1) Controlled by -
(b) Special limitations (if any) -It it is intended that a copy of this document shall be released overseas refer to RAE Leaflet No.3 to Supplement 6 ofMOD Manual 4.
o. Ih'w-uptor (Keywords) (Descriptors marked 0 are selected from TEST)
Workload. Flight. Pilot. Rating scale.
i 7 .Xbq•'trj
Despite the many techniques developed for evaluating pilot workload inflight subjective assessment by experienced pilots is still the most reliablemethod by far. This report describes the design and development - with the helpof practising tqst pilots - of a ten-point ratin.g scale. The scale uses adecision tree similar to that used by the Cooper-Harper Handling Qualities scale,and i8 based on the concept of spare capacity. Examples are given of its use bya large number of pilots in various flight trials and workload studies.
Best Available CopyRAI, AIi (tledQthe Y~
Recommended