Embed Size (px)
Citation preview
AD-RIGS W PROCEEDINGS OF TME WORKSHOP ON TIE RSSESSNENT O F IANOMR EM0" U IL I~tTC OFOINCCR N A BIFERNO ET AlL AUN67 RIL-TR-S?-3S43-VOL-1
mNUSSFEEEE3315hEC36hhhEI/ N
1.0.
% %%
AFWAL-TR-87-3043, Vol I -I t LL CCK
PROCEEDINGS OF THE WORKSHOP ON THE ASSESSMENT OFCREW WORKLOAD MEASUREMENT METHODS, TECHNIQUES ANDPROCEDURES
VOLUME I - Preliminary Selection of Measures
Dr M. A. BifernoDOUGLAS AIRCRAFT COMPANY
3855 LAKEWOOD BLVDLONG BEACH, CALIFORNIA 90846
George Boucek, JrBOEING COMMERCIAL AIRPLANE COMPANYP. 0. BOX 3707SEATTLE, WASHINGTON 98124-2207
JUNE 1987
' FINAL REPORT FOR PERIOD 24-25 FEBRUARY 1987
00
SAPPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED.
OTICQ ELECTE~DEC 1 1987
FLIGHT DYNAMICS LABORATORY %
AIR FORCE WRIGHT AERONAUTICAL LABORATORIESAIR FORCE SYSTEMS COMMANDWRIGHT-PATTERSON AIR FORCE BASE, OHIO 45433-6553
%
I'I2711,
. .. . -. - S . . . . ,S - - . -- .1- - - -' -- -. [4----
UNCLASSIFIED A d a a eSECURITY CLASSIFICATION OF THIS PAGE. ,// f ' g~
REPORT DOCUMENTATION PAGEis REPORT SECURITY CLASSIFICATION Ib RESTRICTIVE MARKINGS
Unclassified2. SECURITY CLASSIFICATION AUTHORITY 3. D!STRIBUTION/AVAI LABILITY OF REPORT
Approved for public release;2b. UECLASSiFiCATION/OWNGRAOING SCHEDULE distribution unlimited.
4. PERFORMING ORGANIZATION REPORT NUMBER(S) 5. MONITORING ORGANIZATION REPORT NUMBER(S)
AFWAL-TR-87-3043, Vol. I
46& NAME OF PERFORMING ORGANIZATION 5b. OFFICE SYMBOL 7a. NAME OF MONITORING ORGANIZATION
Douglas Aircraft Company ([(applicable) Flight Dynamics Laboratory (AFWAL/Fl6R)
*C. ADDRESS (city. State and ZIP Code) 7b. ADDRESS lCity. State and ZIP Code)
3855 Lakewood Boulevard 'PF 1 14465Long Beach, California 90846 ~PF HH4465
So. NAME OF FUNDING/SPONSORING 8 b. OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBERORGAN IZAT ION li aI pplicable) F33615-P6-C-3600
Sc. ADDRESS (City. State and ZIP Codel 1O. SOURCE OF FUNDING NOS.
PROGRAM PROJECT TASK WORK UNITELE ME NT NO. NO. NO. NO.
62?OlF ?403 04 6711. TITLE (include Security Ctauafication)( ak
12. PERSONAL AUTHOR(S) Douglas Aircraft and !Poeing Commercial Airplane13. TyXPE QF REPORT 13b. TIME COVERED 14. DATE OF REPORT (Yr. Mo., Day) 15. PAGE COUNT
rina FROM 24 Feb87 TD?5Feb871 87 June T37316. SUPPLEMENTARY NOTATION
Supported by FAA and UJSAF
17. COSATI CODES SpjJECT TERMS (Continue an reverse if neces~sam' and identify by block numbert
-O9fLD GROUP SUB. GR. Slbjective measures05 09Performance measures
Physiological measures10. ASSTRACT (Continue on rsuer,. it necessary and identify by btock numberl
Report Covers USAF/FAA review of workload measurement methods: validity,reliability, and applicability workshop.
20. DIST RI BUTION/A VAI LABI LIT Y OF ABSTRACT 21 ABSTRACT SECURITY CLASSIFICATION
UNCLASSIFIED/UNLIMITED ESAME AS RPT. D: TIC USERS 0 .'nclassified
22. NAME OF RESPONSIBLE INDIVIDUAL 22b TELEPHONE NUMBER 22c OFFICE SymnOL
.C.Britten-Austin 1 'FWAL/F1PR
DD FORM 1473, 83 APR EDITION OF I JAN 7I1 OBSOLETE. UNCLASSIFIEDSECURITY CLASSIFICATION OF THIS PAGE
UNCLASSIFIEDSECURITY CLASSIFICATION OF THIS PAGE
(11) Title -Proceedings of the !*orkshop on the Assessment of Crew Iforkload
measurement Methods, Techniques, and Procedures. Volume 1.
Preliminary selection of measures.
SEUIT CLA-UNL SSIFICAIO OF THIS PAGE
ACKNOWLE DGEMENTS
EDITED BY:
Siferno, M. A. Douglas Aircraft CompanyLong ',each, California
Boucek, G. P. Boeing Commercial Airplane CompanySeattle, Jashineton
"ayward, K. B. California State 'iniversityLong Reach, California
Williams, K. M. Douglas Aircraft CompanyLong Reach, California
Corwin, W. h. Douglas Aircraft CompanyLong Beach, California
Sandry-Garza, D. L. Poeing Commercial Airplane CompanySeattle, 1.Jashington
Barbato, J. California State UniversityLong Beach, California
Logan, A. Boeing Commercial Airplane CompanySeattle, W!ashington
Dolan, '1. California State UniversityLong Beach, California
ODluyvetter, K. California .State UniversityLong 'each, ,alifornia
SCopyAccession For
DTIC TABUnannounoed d3utitloatlon
Distribut i on/
Availability Codesvall and/or
Dist Special =
iiIA -
" " " ' ," , k W "i ' " "-" " " W " '. " -" " " .r"'"'" 4
,. - W.5 -
TA3LE OF CONTENTS
0
TOPIC PAGE
Attendee List ............................................. viiWorkshop Agenda ............................................. xiWorkshop Objectives ........................................... xiiiSubgroup Participation List ................................... xivFact Matrix Review Handout .................................... xv
VIEWGRAPH PRESENTATIONS e'' -
Background and Objectives A
Objectives of workshop. (9iferno) ........................ 1 .What is workload certification? (rahriel) ................ 5 % %
Workload assessment and certification. (Fadden) ........... 27 . .Validity and reliability issues concerning workload
measurement during certification. (!iferno) ............ 29Application issues concerning workload measurement
during certification. (Boucek) ............... ......... 53Panel Presentations
Subjective workload measurement panel (Hart) 69......-"(Reid) .......... 177 -"opher) 201
Performance workload measurement panel (Wickens) 207(rgqemeier) .... 221("Crlny) ....... ?41•
Physiological workload measurement panel. (Kramer) ........ 245
(1,ilson) ....... 263(Stern) . . . 1
REPORTS AND MINUTESWorking Groups
Review of criteria for Fact MatricesCategorization. (1Wil I iams) ............................ 299
Subjective subgroup presentation. (Gopher) ............... 313Performance subgroup presentation. ("cCloy) .............. 317 ..Physiological subgroun presentation. (Stern) ............. 3?5-
ConrlusionsBriefing on how measures will he implemented in
simulation scenario. (Corwin / Sandry-rarza). .......... 335 .,.
Questionnaire summary ..................................... 369
".. .-'
... ...... '.. :
V V%, V"".%
.", - ., ]
• '-U S~j-
,..' .. ',.'w ',% , , , % - "."w'-, "."". . "' "-,-' -" . " . - - , '.',,'-.'-
%.. -%. e.
e,.r .% o.f'
0.P%
10
USAF/FAA Review of Workload Measurement Methods:Validity, Reliahility and Applicability Workshop d . % '
February 24 and 25, 1987
Attendee List -_ d
Dr. Randall B. Aust George PoucekMcDonnell Douglas Helicopter Co. 0008 15?nd F.Building 531, c240 Payallup, WA 983735000 East McDowell RoadMesa, AZ 85205 •
Janet Barbato rqn Ldr 'larry Britten-AustinCalif. State University, Long Reach AFWAF/FIGR(Douglas Aircraft Co.) Wright-Patterson Air Force RaseLong Beach, CA 90802 Dayton, O1 45433 '.-.
Dr. R.M. Barnes Larry .. Putterauq r?-..CAA House FYA' ,' F I -Civil ',vi3tion Authority ':rit-Ir Force Rase45-49 Kingsway Deayton 011 45433London, WC?B 6 TE England
Dr. Robert P. Bateman .lr. Richard F. ChristBoeing Military Airplane Company ARI Field 'nitP.O. Box 7730, M/S 76-23 P.r. Box 6057 "Wichita, KS 67277-7730 Ft. 3liss, TX 7nq06
Mr. V. Battiste 'r. William CorwinNASA Ames Research Center McDonnell nouglas CorporationAttn: MS-239-3 F?1 'C 15-36Moffett Field, CA 94035 358,R Lakewood Boulevard
tong Reach, CA 90846
Dr. Michael A. Biferno Dr. William L. DerrickMcDonnell Douglas Corporation The Center for Creative Leadership -E21 MC 35-36 500 Laurindra Dr. PO Box P-l3855 Lakewood Boulevard frperr0+oro, '!C "'740?Long Peach, CA 9P46
Dr. Richard Blomberg nr. Michael BreselDunlap and Associates Fast Inc. Lockheef Ceorgia Company -17 Washington Street rmept. 7?-O9 7one 419 ,,..
Norwalk, CT 06854 flarietta, GA 30963
Michael R. Bortolussi "r. r on~nas qrqqemei(r?1ASA Ames Research Center r epartment of PsychologyMS 239-3 'inivnrsity of nayton -Moffett Field, CA 040 5 ")0 ,rollere Park
')avton, '' €' - -'" "
Vii
........ ......... . . .. ". .
Delmar M. Fadden Aileen LoganBoeing Commercial Airplane Co. Boeing Commercial Airplane Co.P.1. Box 3707, MS 77-70 P.O. .ox 37P-, -S 66-?Seattle, 1.A 99M124-2207 Seattle, '!A ')124-2?07
Dr. Richard F. Gabriel George LyddaneMcDonnell Douglas Corporation AN'1-lO4LDept E21 MC 35-36 4344 Ponald ' ouglas Drive3855 Lakewood oulevard Long "-Lon, .e , -
' r . F r' *, ier 0 r . T ,o 'as ' . ,C rln vTec'rnion-IIT 'Initerf States nir Force AcademyFaculty of Industrial Eng. & Mgmt. Pehavioral Science A Leadership Dept.11aifa 32000 Colorado Springs, CO 8084)Israel
Dr. Peter Hancock Dr. Henry '.W. IlertensUniversity of Southern California Human Performance Pesearch ')nitDept. of Safety Sciences FAA/CAMILos Angeles, CA 90089-0021 lox 2509? AAC-11I
Oklahoma City, OK 73125
Sandra G. Hart Dr. Sam 'etalisNIASA, .Ames Research Center 4cnonnell Douglas CorporationMS 239-3 Fl "r 35-3611offett Field, CA 94035 3855 Lakewood Boulevard
Long Peach, CA 90P46
Kathleen Hayward Dr. Thomas P. letzlerCalif. State University, Long Peach !!.S. Army 2esearch Institute(Douglas Aircraft Co.) API AVSCO" PERI-IRALong Beach, CA 90802 4300 ',codfellow Boulevard
St. Louis, *,n 631?0-17qP
Dr. Peter V. Hwoschinsky Dr. Neville "orayFAA APM-430 U!niversity of Toronto800 Independence Avenue South West Department of Industrial FngineeringWashington D.C. 20591 Toronto, nntario Canada 'i5S 1 A4
Dr. Pichard S. Jensen Dr. 0ohert n. O'DonnellOhio State University 1ITI CorpnrationDept. of Aviation 41in Linden Dr., Suite ??Rox 3022 rlaypool BuildingColumbus, OH 43210 Dayton, OH ' 43?
Dr. Arthur F. Kramer rharles OverheyUniversity of Illinois Science Advanced Technology OfficePsychology Department Federal Aviation Administration603 East Daniel Street Washington n.r. 2o5qlChampaign, IL 61,120
viii
NI*
Don~~~~~~~~ L.. Pak.r4lr A hnldc
Boeing~~~~~~~ Miitr Aipln Copn IIc
Drn .AlnPoke Dr. Kiar A. higldekeNAoeiLngilety RsairplaCner ompanyalfoni CTo.nP.O B52x 3707, R/S 33Sie 13
DDayton, OH 45432
DrighAla tro Popre ae Defr mance Si peatosergnNaSAtLnle Reeac Center55 FlockheeD-valifonia4B Co
Hamptn,3VA2366 Burank, c CA x France
Dr. AlayB Reido Dr. John ASmt r
Wrgtaters AirpFocerts Waustinto TX vrst 787311
John *Ryeisn GJenincuseyeAivilAviato Atort FAru Indstie 21Crgh-atero Hruore Base Performance veainse nin.IDayton H 543-55 Flsigt Dviio P.C P o 3
31707n Blagna Cedex Frince Kingdom
Dr.n AL.an droe z DIr. JohnA. Steinrn .gh -
Breitnni Aomriras Altd. Prfeso of PiiDvsycoloy,
SedtlW 8?-?7Hrfordshire ALU2 9NDL UntdnigdmStlausndO33
John RyeadShife r Laic y Tinleyat
Crivil Avatieon Ath ory FasA AFSn 2107MS1
Da-y9ton sa OHashingtO. on P.C ?16Londo LoCis 4TE United 0ingdom
Di. ae l G. Sandry-ar D~r. Chisope !in r.qh WicenBoein Com/mecial Aipan1on!A ivril ofIvisio s
P.O.UN Bo sAF1 3707 78? 66-2 ae Dail tre
4:0
Dr. Ken Williams Dr. Glenn F. WilsonMcDonnell Douglas Corporation 1I'IAFE21 M4C 35-36 A'4RL, IAMPL/fHEr3588 Lakewood Boulevard Wright-Patterson AFR, OH 45433-6573Long Beach, CA 90R46
x
,waJ':m 2I T
USAF/FAA Review of Workload Measurement Methods: VValidity, Reliability, and Applicability .
.%... -%
PROGRAM - WORKSHOP 1..February 24 and 25, 1987 ' ,
DAY 1 6
TIME EVENT . * ,%. "-.
BALLROOM "A"
8:00 Welcome from Douglas Aircraft Engineering.(Peterson)
8:10 Welcome from USAF/FAA sponsors.(Britten-Austin and Hwoschinsky) .
8:20 Objectives of workshop..(Biferno)
8:30 What is workload certification?(Gabriel)
9:00 Workload Assessment and certification. .(Fadden)
9:30 Methodological issues concerning workload measurement during -
certification.(Biferno, Boucek)
10:15 Break
10:30 Subjective workload measurement panel: A review of theevidence regarding validity and reliability.
Reviewer Hart QReviewer ReidDiscussant Gopher
12:00 Lunch - LOUNGE
1:00 Performance workload measurement panel: A review of theevidence regarding validity and reliability. ..
Reviewer WickensReviewer EggemeierDiscussant McCloy
2:30 Break
2:45 Physiological workload measurement panel: A review of theevidence regarding validity and reliability.
Reviewer KramerReviewer WilsonDiscussant Stern
xr
Z.
DAY 1 (continued)
TIME EVENT-----------------------------------------------------------
4:15 Review of criteria for FACT MATRICIES categorization.(Williams)
4:30 Review of participant Fact Matrices by workload panel.Three subgroups will be created with panel members as leaders.
Subjective subgroup in room # 106Performance subgroup in room # 136 (Wednesday in room # 121)Phystological subgroup in room #151
5:30 Adjourn.
7:00 Banquet - MONTEGO BAY ROOM
DAY 2
T -AE EVENT
8:00 Continue review of Fact Matrices by workload panels insubgroups. Cite evidence for additions or deletions tooriginal matrix.
12:00 Lunch - LOUNGE
BALLROOM NAN
1:30 Review Fact Matrices of Subjective Measures.
2:00 Review Fact Matricies of Performance Measures.
2:30 Review Fact Matricies of Physiological Measures.
3:00 Break
3:15 Briefing on how measures will be implemented in simulationscenario.
(Corwin, Sandry-Garza)
4:15 Concluding remarks.
(Boucek, Biferno)
4:30 Survey of attendees regarding the best workload measures.
4:45 Turn in survey and final version of Fact Matrices.
Cash bar - MONTEGO BAY ROOM
xii
, - ", " " - "-', " ' " - ,, , -, - . , .- - ...-. -° . . .A..& . .. A AZ .. ,-& . . . -.. . . -~ , ..
OBJECTIVES
OBJECTIVE-OF THE WORKSHOP
GATHER INFORMATION FROM WORKLOAD EXPERTS REGARDING WHICHMEASURES HAVE EVIDENCE SUPPORTING THEIR RELIABILITY ORVALIDITY.
OBJECTIVES OF PANEL DISCUSSIONS
PROVIDE AN INDEPENDENT REVIEW OF THE FACTS CONCERNING THE
VALIDITY AND RELIABILITY OF WORKLOAD MEASURES.
OBJECTIVE OF THE SUBGROUP SESSIONS
PROVIDE A MEANS FOR SYSTEMATICALLY REVIEWING AND MODIFYING ThE
FACT MATRICES. .
xiii
USAF/FAA Review of Workload Measurement Methods:Validity, Reliability and Applicability Workshop
SUBGROUP PARTICIPATION LIST
SUBJECTIVE PERFORMANCE(In room #106) (Tuesday in room #136)
(Wednesday in room #121)
Dr. Randall Aust Ms. Janet Barbato (assistant)Dr. R.M. Barnes Mr. Michael R. BortolussiDr. Robert P. Bateman Dr. George BoucekMr. V. Battiste Dr. William CorwinDr. M.A. Biferno Dr. William DerrickDr. Richard Blomberg Dr. Michael DreselDr. Larry C. Butterbaugh Dr. F. Thomas Eggemeier (moderator)Dr. Richard E. Christ Dr. Richard S. JensenDr. Daniel Gopher (moderator-chair) Dr. Thomas McCloy (moderator-chair)Ms. Sandra G. Hart (moderator) Mr. Don L. ParksMs. Aileen Logan (assistant) Mr. John M. ReisingMr. Thomas R. Metzler Dr. Clark A. ShingledeckerDr. Gary B. Reid (modearator) Dr. Kim SilerMr. John Rye Mr. W.A. WainwrightMs. Diane L. Sandry-Garza Dr. Christopher Wickens (moderator)
PHYSIOLOGICAL OBSERVERS(In room #151)
Dr. Peter Hancock Sqn Ldr Harry Britten-AustinMs. Kathleen Hayward (assistant) Mr. Delmar M. FaddenDr. Arthur F. Kramer (moderator) Dr. Richard F. GabrielDr. Sam Metalis Dr. Peter V. HwoschinskyDr. Neville Moray Mr. George LyddaneDr. Robert D. O'Donnell Dr. Henry W. MertensDr. Alan Pope Mr. Charles OverbeyDr. Alan Roscoe Mr. Don SmithDr. Richard Sohiffler Mr. Guice TinsleyDr. Samuel G. SchiflettMr. Jean-Jacques SpeyerDr. John A. STern (moderator-chair)Dr. Larry WalrathDr. Ken WilliamsDr. Glenn F. Wilson (moderator)
xiv I
WORKLOAD FACT MATRIX REVIEWPANEL AGENDA(suggested)
TIME ITEM ITEM DESCRIPTION
1630-1730 1. CRITERIA FOR FACT MATRIX REVIEW
a. Reliabilityb. Validity0. Empirical datad. Capability for flight usee. Rejection/Addition rationale and support
0800-1030 2. REVIEW OF EACH MEASURE ON FACT MATRIX
Process:1) Pick workload measure for discussion2) Identify empirical material which
provide evidence on validity andreliability of measure
3) Solicit change recommendations
4) Review evidence for changerecommendations
5) Speaker Panel form recommendation6) Repeat process for next measure
1030-1130 3. REVIEW MEASURES NOT PREVIOUSLY INCLUDED IN FACTMATRICES
Process:1) Identify a new measure for inclusion2) Review evidence for addition of measure3) Speaker Panel form recommendation4) Repeat process until no additional
measures identified
1130-1200 4. FORMULATE SUMMARY REPORT
Summarize:additional reference itemsdelete reference itemsadditional measures
VJ
R ite,, . . , ' .'....'., : .,: .'. ". .. e .'. .... -
WORKLOAD FACT MATRIX REVIEW
The purpose of the Subgroup reviews is to modify the FACT MATRICES ina systematic and orderly fashion. The FACT MATRICES are basicallylocators for Reliability and Validity information for each Workloadmeasurement type. We want you to add reference work to the FACTMATRICES if they contain Reliability and Validity information. Weare not asking your panel to judge the quality of the information,but determine whether the material addresses the measure'sReliability or Validity. The evidence can be weak and still beacceptable for inclusion in the FACT MATRIX. On the other hand, ifwe entered references into the FACT MATRICES which are notappropriate or correct, we ask you to delete those items. Regardlessof the modifications recommended by your subgroup, evidence must begiven for the addition or deletion of items. It is up to the SpeakerPanel to decide on the acceptability of the evidence and on thedecision to add or subtract from the subgroup sessions and direct thedecision process that the group employs to modify the FACT MATRICES.
We ask that each subgroup first employ the Anastasi and Guilforddefinitions of Validity and Reliability to the workload literature.We are aware that the workload field has not generally focused itsresources on demonstrations of reliability because there isdisagreement on the definitions and content areas of the workloadconstruct. Therefore, after addressing the literature using theAnastasi and and Guilford criteria, you are free to employ thevalidity and reliability definitions of your choice to support thecontention that a particular measure is valid and reliable. We onlyask that you explicitly define these definitions in yourjustification for that measure. Remember the studies suitable forsupporting validity or reliability must be empirical, not review ortheory articles.
0
x.v
0%
....................................................-......
- a-|t
VALIDITY DEFINITIONS
1. FACE VALIDITY: "... pertains to whether the test 'loods valid'to the subjects who take it, the administrative personnel who decideon its use, and other technically untrained observers."IMPORTANCE: "Certainly if a test appears irrelevant, inappropriate,silly, or childish, the result will be poor cooperation, regardlessof the actual validity of the test." ... "For example, it a test ofsimple arithmetic reasoning is constructed for use with machinists,the items should be worded in terms of machine operations ratherthan in terms of 'how many oranges can be purchased for 36 cents' ofother traditional schoolbook problems."REFERENCE: Anastasi, A. (1968). Psychological Testing. 3rdEdition, Macmillan, London, p. 104.
2. CONTENT VALIDITY: " ... involves essentially the systematicexamination of the test content to determine whether it covers arepresentative sample of the behavior domain to be measured." ..."The content area to be tested must be systematically analysed tomake certain that all major aspects are adequately covered by thetest items, and in the correct proportions."IMPORTANCE: " ... content validity depends on the relevance of theindividual's test responses to the behavior area underconsideration, rather than on the apparent relevance of itemcontent."REFERENCE: Anastasi, A. (1968). Psychological Testing. 3rdEdition, Macmillan, London, p. 100.
3. CONSTRUCT VALIDITY: " ... is the extent to which the test may besaid to measure a theoretical construct or trait." ... "requires thegradual accumulation of information from a variety of sources. Anydata throwing light on the nature of the trait under considerationand the conditions affecting its development and manifestations aregrist for this validity mill."IMPORTANCE: " ... construct validity is a comprehensive concept,which includes the other types. All specific techniques forestablishing content and criterion-related validity ... could belisted again under construct validity."REFERENCE: Anastasi, A. (1968). Psychological Testing. 3rdEdition, Macmillan, London, p. 114-5, 121.
4. PREDICTIVE VALIDITY: (CRITERION RELATED VALIDITY) "indicates the effectiveness of a test in predicting an individual'sbehavior in specific situations."IMPORTANCE: *For this purpose, performance on the test is checkedagainst a criterion, i.e., a direct and independent measure of thatwhich the test is designed to to predict."REFERENCE: Anastasi, A. (1968). Psychological Testing. 3rdEdition, Macmillan, London, p. 105.
xvii
1V.
RELIABILITY DEFINITIONS
1. TEST-RETEST RELIABILITY: 0 The key concept for this(test-retest) procedure is that of stability. It answers thequestion concerning how stable or dependable are the measurementsover a period of time."
IMPORTANCE: "High reliability of this kind tells us that theindividuals remain rather uniform, or maintain their rankpositions in spite of changes, in whatever psychological functionsthis test measures.3
REFERENCE: Guilford, J. P. (1954). Psychometric Methods, 2ndEdition, McGraw Hill, New York, p. 374.
2. SPLIT HALF RELIABILITY: "The information sought (fromfractionation of a test into two or more parts) concerns theequivalence of parts for measurement purposes, or the internalconsistency of the test."
IMPORTANCE: "In the case of the split-half method, theSpearman-Brown formula has usually been applied to estimate thereliability of the test of full length from the obtained estimateof correlation of a test of half length."
REFERENCE: Guilford, J. P. (1954). Psychometric Methods, 2ndEdition, McGraw Hill, New York, p. 373.
3. ALTERNATE FORMS RELIABILITY: "... method bears resemblances toboth the internal consistency approach and the retest approach.the end result is an index of how equivelant thepsychological-measurement content of one form of the test is withthe content of another."
IMPORTANCE: "... the alternate-forms method indicates both theequivelance of content and stability of performance." ... " Someinvestigators prefer the alternate-forms type to the internalconsistency type of coeffecient for the reason that they areinterested in how much stability to expect of scores over time."
REFERENCE: Guilford, J. P. (1954). Psychometric Methods, 2ndEdition, McGraw Hill, New York, p. 374,5.
4. INTER-RATER RELIABILITY: " ... rater intercorrelations, whichindicate the internal consistency among raters. Such correlationshave usually been regarded as indices of rating reliability ..
IMPORTANCE: If raters agree, demonstrate high intercorrelations,then the number of raters required to generate a significantresult cna be reduced.
REFERENCE Guilford, J. P. (1954). Psychometric Methods, 2ndEdition, McGraw Hill, New York, p. 286,7.
xviii
ABESSMEIT FORM Lib. No.CHECKA. Content (If none are checked QUIT) 1et Init.
1 Workload Measurement Primary.2 Workload Measurement Secondary. 2nd Init.
NUMBER OR MAKEB. Quality of Review
_ 1= Formal Review 2= Informal Review 3= No Reviewer
C. Quality of Data (If not 1-3 QUIT)1= Experiment(s) 2= Case Study(s) 3= Theory/Review
(Skip to F.) (Skip to M.)
D. Experiment Name if more than one (One per form)
E. Highest Fidelity of Experiment1= Actual Flight 2= Simulator 3 = Applied lab. 4= Basic lab.
CHECKF. Validities (If none are checked skip to G.)
1 Content.2 Construct. MEMO3 Predictive. Independent
Variables:G. Reliabilities
1 Test-retest.-2 Split half.
3 Alternate forms.4 Inter rater.
H. Measure Types DependentSubjective. Variables:
I NASA Bipolar Scale2 SWAT3 WCI/TE4 Modified Cooper Harper5 Interviews6 SurveysT Other Subjective Measures
Physiological.8 Body Fluid9 Brain Activity10 Heart11 Lung12 Muscle13 Skin14 Vision15 Voice16 Other Physiological Measures
Performance.17 Primary Task
17a Time17b Position
17c Event18 Normal Secondary Task
18a Time18b Position
18c Event19 Artificial Secondary Task
19a Time19b Position19c Event
. -.. ,-.. .. . . . X-
9.
I. Workload Types4a Degree of Mental.4b Duration of Mental.
- f4c Degree of Physical.4d Duration of Physical.
J. Workoad Functions1 Flight path control.2 Collision avoidance.3 Navigation.4 Communications.5 Operation and Monitoring.6 Command decisions.
K. Workload Factors (Task Demands)
1 Normal Control.2 Normal Display.3 Normal Procedure.5 Normal Monitoring.8a Normal Communication.8b Normal Navigation.6 Non-normal Crew unavailability. -"7 Non-normal Automation.9 Non-normal Procedure.10 Non-normal Crew incapacitation.
o.-
xx
% V9
FAR-25 O ILoAD FACTOR 4c: DEGREE OF PETSICAL
VALIDITY RELIABILITY
NZ.JUKE CONTENT CONSTRUCT PREDICTIVE TEST SPLIT ALTERNATE INTER APPLICA-RETEST HALF FORMS RATER BILITY
NASA 28 28 95Bipolar 95 268Scale 268 539
530
SWAT 28 28 340 34041 41 766 766339 339 773340 340773 766
773
WCI/TE 13 13 16 64 64 64 6416 16 53864 64243 243367 367538
Modified 13 13 15 225 183 183Cooper 15 15 16 583 283 225Harper 16 16 65 344 762
65 65 175 459130(1) 130(l) 183 496159 175 225 583175 183 283 762183 187 436 793225 225 459243 243 492344 283 496367 367 538436 436 583459 459 761492 492 762496 496 778509 509538 583540 735583 761669 762735 778762 793778793799
xxi
-
z c-
C4 LLa :3
ui Lu OCCOC Jm 4e 3f
cn N Q>- OC
L1
LA1
0
zz
I-I CC~WzL. L
00
CLL
LU~ C 0 cc~CU
F w CC-oLL
W 0 c3Z
o cUO( u 3Z MooL
wo~~0 xU4 ~~
WW Zu 00 c -
3: c0c =L0U>Z
LU0U >00
0 x0 04 0> 1i> 0 L > 0
0
0
-4
4 04
B ~ ~ caa
zoozi
~0
uviw-,rw- T jj I rV~r v7-- www-p
UV.
CL L2U)p
(z CL
o L
LL U
0CC 0
0 0
0J Z
U. ch
oz
0 (
o ULI
z
(LL
0 c,,0 owc0
z o-oww0 CjLLJW
0 u
LL LL.
U. MLL,
LL.4
LL..
6 z
0 .I.
z200
9- P40
a. 0z (
LU "7 4
0 ~40 -J LL
W l)m U .01 2 LU *
z Z -
awa
S LU cc C4~
0u U c . C CU t I.- 0. 4 z0I. -I UA 00 a.a c
O w I- 4c ) I-- .) C .
. C CLo
LU z
Z Co LLLI Co
LLo >m ca
UJLU Co 0 o . 4woCz LU(1 C>
U CLocc 0,
X 7
j 0 C qc L
Iuum To,
Le) Cv
CL CL L
w ~
C/) 0
C,, LUI- LU
0. cn 0 0
LU
000 =-J 00
L) -J
X
0~
z cl,
0 :0
0LU I-
L U l
LU w -0L 0
LU 0L 0 n
Wn MfO LL <
W CLw zCl, 04cm <EE
z -j z I.< w z LUI- LU (n -Lucn<z CL I cn 0
Z uj 0(jZ z -LLJ CfL LU >-o
(n 50 -i59 0 I 3 LL-
~ LL L CLZ0OaCL 4w Euz LU eooo 0ee
0 o0
LLI
UL
n z -
o I-
I-
wcn ML
o c ,,LL.J W
10
~ 0A
wu >U))
(A C L aQ
0. -JL
0 z.(*) LLJ U
a.~ ~ j LL
I- COLL
CC z. <oU m
= 0
0UccL L
LLJ11
z u
0 z 0
,.- -o 2 7= Z
0 Oj ~ Z LC C C0 m CC l -X
I. U . 0 (
CC 04c L AK ~000C a 0 W ) ZU .Z ~ w (a. Z I t L C
WIC (J Z CC Oc z
Z 0 0u
0 ~ Us z =a >. -mL = ZU- 0 C 0 L
W, 4( w: z 0w 3( z0U U- C *.j w0
(i u 03 3IL Om
A Z2 UA . ~ (A
e '- I.~ usB AlflU. in C C CCU201S LUZ 1- d
a. Z~ z - U
IL - cI j- C,Z 0 Z Z. P (5( LCC - om o - -
1. 0 5 .! a IL 1WZ0QIUJ CC (
U. a 0 (0 0 M
1?0 0- 0 'CC z I.>
INK iff- x .p A I -J -W-. T -. F -. -.
z0
cc -- - - -- - - - - --
0cr4
ILu - - -.I
z -61 (A
CL U)J 42
U.a
'C 4' Z"LUC-. C
g13
uq.
oLU L
Z/ (0CU), 0
> CJ
o aU <C 0 0
Om wL Q L PUU(Cl
LLI 0Im 0 < Lrz ~LU 4c mC.
L cU. a.L cn Ocr.m Im L LL C LU -<cw 0 n. C 0 ooa C/oj LU nLa Z>: (Lu wu WZ 0 0 W )
Cl) d LU LU - -c. J L>- ~ I Z OZw c maL
:5o ULU*L
14
Y.T
z LU
0 0 m
C) 0LLL
.00 z
00. LU
0Z 000U0 LU <) w, L3 0 LC)_5 z LU 0 L, z=0 -rza U
ulf Z L 0 W
0z OC0 w J
0 Z Z 7 n i-0 ~. ~e*0 U j
0 0 LU is
w LI.
LUU
ozw LU
LU cn
LL >>W LU
C/ O LUZLU Lu~ 0nF
% 00Oo 0*0-
LU00 jz Co0OO.L ZCC)
@0- Co@@ CL) x
0= I-w0o uU
16
I-L
LLUfLL Zn L
LL 0 Z LL
o LU
~LLLJ (fl Lcrw Ol)Zw
mj~ ~ =-0z0~ 0J J 0
>E Z
cr C uLU Z-4~ a>ILU> Cfit x
z , 0 C
0 i z LUJ0LL
4cJ ZJMLu-
0 = - < a. Zs17
C/)
Z -rn In z
oq - a.00 cn z
o ow 0CC~
0 Q)L LL -0
cazLULL J L 1
LL00 NWUJ
0 0- 0
-LI--L
n rui > <U00 Z Z a.
LUUJ C
LL.iULL LL 0l
UL. 0w Mw
LIu
z m0
UU
LU 2-0- M- LUI
Cn z a1 =0 WU z C
u <0 --
z Mc 0 wLUCn- LU c,,LU L I- z cfl2j OL U) M u LU
4c _c Cn3 (n zj.0 m'(
mL u: C5LULU - LU- 0
I- 0 0L
U. ~-J~ 0 i 0 zzi -J 0U 0o~~L LCC 2
0 > < 0 2 L U
0 LL ;0- w u W-L U - 30Lz X.0 " z-J U - o j .>0 cu
2-M -am qcUU 0
00- 04 o 4 02z 0 ZZ 0 3 0 ~LUCJ2 cc z < 0
2 jU. Z .- 4 0Q ~ZL4c I-.(I- - 0
.0 z -Q
CL u z U-0 -U
>. RR~V ~ ~ ~ j
w
inwwOw f w /CmL m w
LU LU LU VVL
0 00
3: C- mJ a. l
uj < LUU z _.. cn U
w 0
W <z U.m 0J0l- a.W W mI UL LU
LL I- 0-1
tn ~ z 0.0
-J 0<0 LL
Cfl~~~ LULLio~
w n w
w ZU 0- 00 LU 0 0>
20 Z n
W o W z
-JW LL
-j U-m 0
LU
CL 0.
Z z c/) L 00 a-
C/) LU
00
LU Ul0- 0
o.cLU 00
cnZLU LU
o UJLL. 0
< LLw z21
cc w . -- -t-
wuZ
-~ 0
xw 0
wz UZi
LMU
0.
LL.
LU
a .if
0-
LL4
mU
=l 4
LU
I-n
0)0
0~0
civoumo 0NVW CV~l3Cz
00
21)
LU I-
zz
LU 00 %UoL
LU LU L0 0 CD~ L&.
I, 0
F0
o LU
0LLc Z -j
0 CL, LL
wU LUpwo mA
0 0 m0
mzw a.0I > LL
Cl1 o n 0
L 0 LU0
~LU
25.O
W. mv v-~w -VV~qv-r v7 -W .w yV-V . .- - -
z0.
00 z
w ccLU 0 0
cn z~ LL0 zU) 0 0
.j. cn) 0 zo> z 0 L
F 0 LULLC.
U cm z >LI.
z 00 C/U Lu
0 C/) C. N0 U. >I
0 w 0 5C.) Cr' C
C.) o U.LU
26I
Intentionally left blank.
*7
'p
'a
Intentiooally left blank.
'a'a
N
'a
paa,
a,aa,
N
L
a.
28
z
j00
0
0
WLLL<
cn cnO ci n
(n c~
cc 0
0<
0 >
29
00
CC
03
___ C
0 C)
U) C..C,) C
o i). cs
00
c.E
oU W 4) U)M 5
I-
0.)
E Co0
E. E Q
C.0.
c . .-
Cu 1 E =
-M E r76-%6
I..) =05
o) a-'nca o.EC., CL
C1Cu
31
0
0C/
L
i0j ZU
CL'LU z
>1. LUZ >
C/) cnO 0 LLU Z
0uCC 0-Z EI q
0 I- n 2...
CLz~~LUULSL
32
pul -API-PTW.-PC-7V VW-7
a.
WI..
Z
E 0~~2 ,0- CL Oc >%
L 111a) m CU) ~ >U
m rc
.CA (f~-0 LU(no
*-(A5; A14%4b
C33
E . ' - ' V ~ ~ . -
Ca UU
r cc
a)< in X XX XX~
mL.
w~ >
IE-~ E
WO m
Qfl 0
0 CL~
xwi
34
LUI LUJ LU
UI =L =
LUI LUI LU
IL >
* 0LU uj LU
0U 0
-J LU
U 0
z F-
= C,,LU
0U LU
-A LU
5 Ia.
0 0
(n Cln U)
wj w w
ClF U U)
o z z zw wL wj
0 z z z
-LJ
00 0
0 000
M >xC3
ww
LU CL)
U) U)
N~jr w~w w -v- r w v- w v V- -Vwwx - V
LUI
o000
i. Zdz LLI -
000 0U
uZ z ozo ZfH 0
H LU
O<L z ~0 za-Wa 0 Z a- W L
=L -H a z< ) U-0 LL C%cn0 0
LU C/) LLOZ O u -J uV
C- * * * a- 0 ccr.
HL CLa U
~~0C* * f 4- --. 4'A:.
TV4
LUU
CL LU0
~C/ 0 w U
LU z~LL LL (, n
00 0 0Cl) W W
C13 LU u
cn 0 0>Z Fn Z Cc_
U) LU
LL zZ1
38
F LU LU
LUU CcnLU LU
LI> z 2 m Z
pw - 0Lj 0n 0 i
'Em- 0 n i c - -
LU 50. u
u 0.
z 39
0
I- w- W
z UN~. > >0 0
-0 ~ 000 0 Wufl
ul m 0
U)n w1 U 0 c
U) x0
ujCl Z LvC.
0U zoX-~ U. ZWW Q
~w0 LU < uc
0 z U.0 0I. LU
UJ z w
m cr *r
Z C).
LL CC z *U < .w
(I(-
0 _)
C_ z0L
Fn J
-0 -J. woz<LL jI-
0 Zn( 1= L i L L J - -
L LLI X
N LL J C1 M L0C) ( -j
M-- I <
0 z zz ~ U EDLLW W' > H- j LLLJ cn 0L C
=J a.w <~ Z L> Lj _>W -QL~ LL O < Z<W Cfl LL c
L.J 0IL -X
Z >-J aW cH 0
0 > 0j-< Z LUcc m < <
LL U -j = -a :
w 0* TO.
I-d/ (I)
e r-.C
00
Cl, LLLLUi LUJ
Lu zwL LU L
LI
4?
J1*
vv
colW m
LL2
LU
z
o4
I- (1)OL() U-u-
0 - i Cs a-
z I- LC) LO) 1o in r.:6in N _
~LLi
U)C/ - O _ _ _ _ _ _ _
LU Lu I0
0 _r r- .4 C
L(I,
,-mo LU 0 0CN 4
LU
44 1
ch f c 00
m_ _____
0
o 1chE
E E E d
coC O 0 c
m 0 doJ wo( E E E>- r- Ln 6= o tr 0:t r. LO
oWi c NL 0 co0
0 0 Lt~,0 00Cin~
45 C
CL ~0
00
CLI
*L 0
464
CVJ
co,C14
q.
Cln z
C4 0
zzOct-J
447
LO C! .o
a) 00
41 cv,
E-C,_L 0 M
CC x
.d~. om_ =
I 49
CXW WTTOW IP - -,7i W - F' ), Tx u -w w Rr Mf- Vr V.rU OUR
IL
* 1-
C)C
0 C)
0 4)
* I)LL0(14)
LL. :
CL 0
> L
U. X %)'-o LJ
cn a LU.0
9$0 x mEZ
w z0 .u.-
(n. 41'*w
0
aI.
U)00f cu
0.0u
U)-
I-s ~ o ufl w Lzz
0 0UL
VIE- ~ www~-J-. -v -V - .iV - - -. 'T .wV WY d'I V V-
M
z w
I 0
U) z
.Z
I-LCl)U
w z
0 o
.- 5
00 C
0 0
W w wT vW'vr w% 'V-wWW wvr w.V wv ~ ~ ~ ' ~ .- ' .- - - --. * -.
Omit
CL
4) 54
0
CL
4) 0ccm) m) Q U)w .
C -
ci) .2 .u
- ~ -.. '--'. -. -. .-. -. * -. '-*----------*--- U~ ~ - - -
0
4
4)
~9~4)
U- C.)U)
Lu. E _(U
C,)4)o 4)o qC
(mm) 0. 4)U)
o C.)o = 0C.) C.)
4)
'rn-rn 4)2o 0* 0 0
AJA'.FJPA.P ~F ~ *.~ - -. -* .1
00
4)4
C,)L
j j C 0
WIN
pm
otl /"
S n
-cu. C.
4:,
o 1
(A
I- C
59
Cl) 0
mm 0e
C) EE
h.LC)
U- so Cl
ci) _
CLo
0
a0C) CL
CL-E
o Co
C
Z 0
0 _ 2
0 __I
oIwoCl,
la
0
CLQ
0) 0
- cm)
72 -00
*11. Ac(n0, c
66 *rJ~ C CU ~ E r
77'
~ .9-0 47'E
-zr
a Gi
or4
0vJtjL;CD Q~~1
CL :ik CUP
= %t"
(A w~
CL. CL
I "4-
OtAM, 'liiID%
ccN' Pt
6i[
J-21
vr.
ILLI
64
-w-w-v w~rvw- a- v
- d
iv
-tu
-Sr
Ivw
itI
4. 4 . - . 4 4. - - % 4. . 4 . - - .k
131
*1'C-
cnA
1I67
-ir W " ~ -KY V- - w v r- w mw v w- , V. rw w V JVLqv--~r 'J Xs. J uw '-... - r. r
C
0
z0
0
z w U.
ml00 1
0
69
z
w
I-
w
-j-
-- Jw
z w w
ccn
-j
> -i (J
ww w c
700
LLCL
LUUz0
711
0zz
wz 0 00O3j
cn 'I-so.
0
0 .f - -K-
LLI
zzz > z 00 0 cr~
p- a. -J LJ >< Z
U) >: < L 0Z (n0 C :) < 0zC
ii z w
z CL. =z 0qU) wO o'W :
U) < 0C- Z <- OU. W
03 U) Z Z <
U) LL JOL U.W < -<o o00 U (z -0 0) LI < 4 W
(A <z 0 -o z La44~~~ ~ ZZ)z xw Zz =X
4~ < _ : < 0
U. LL W w iw L . 0 a. -j Li
-C 0L Li ~ ~
< x a< < L. 0 QU
w- Li 0LJ - Lm ia
a I -J <A LL
- =00 LA.. -
v*wxW 0 1. <
z 73
0
U0
z (n
U) <0
_j~ o~.
-<0
0 Z 0Dz I
gu z
LLO.<w
74
z
00
zi
N/ N
I--7-
~0
40j~a~z
000
C,6U)
X U
0 0
INA IS
w z
guiA . .
a: D- 0iw) 2c !*9
z LLZ
; ~ I zz 0L
I-l
LU -
CU- -wL.
U-nU
IA.ELw
6O IZOI6 P
4 --0 I
O t~v c-i
zf
x <
*w I
a..
& *~4~.?'j w u __
0 LL \L
<. L.J -i .
- 7
Li
< xn
zA LIJ U.
0 U0
ZZLoi <
- ccT
> ' z
w <-<
co < L
Co/ <LA.
Ui -
V)A . LJ
<La.
**.tL W-* . .-
FM 4 PIOCEDU OF THE NUUP ON TE NETU EUICKLMO ESI*EIEN. (U) DOUOLR armwy M LOWE MouCR R A DIFERNO ET M. JUN 9? WFIL-TR-07-3643-YOL-1
S~LSIFIE 65- C 3 3SIS46-C S V'9 .
CL E flfflfflfflfflffl l lEhmhhhmhshmhEIEhE.Ehhhhhmhuo
L:l 14. 1.0 ft
I'll' -Ii~ 1f8
'I'll-
4-04 I 1p4%~J I2
N-
44
z
)- I- 0
zc zz Z 0
ccZ>0 0 Z
m a
7q
aCC
Wa Z
.J~ -J -
0 z I Z
.0a -
o U 3 : - ->( w -
LU a
I ciZ -'
0Ow a c .1 a aa a a a
LU 0 2 [4- Uuoq a"g auAOW some0 1 6 Gwas"
J L&. 0
LU 0.C
0I wa. w
>- uj u
0. U. '0~0 0 W I iI g U'
0 l RE
LUU 0 ~
UU Z- z
I-
:5z _ _ _
IL 0Moo60 <z
CL ow
0< 1. 0< 2
o o > 0 -
i- ,0 w0 LC
>. Q ~ Q CI) U C) Z .) C 0 > i)
ALZ
uJi
0l
o w'I-< Ij loww
w .
wWw w -
wiI w wo w z> (5 z
CO I( i~ 82
MAMP~PN mFIWV .- EL W- WIXI 1%?r Vw ~1ww g- 7:
... . .......
0 I /
(A In '41- e4
00 w C,,zz w-_jIL fit IZm -j
Fwr W
< LL LWO-wU)mz ww>OL
< M a W
I w 0
83
A pp
0
w -U
Z1 9i 11 a(A A
LL. L Y 55
(I. Pru-vsv
0 i Wn0c
2. ww Tc<
0 0 0j U- U)U. w z
x 0 ouz.*~Uuo< .i( x <W T) w <
w -1- U mww
LM 0 < 2 <Z0 wo xWuo. w(
U. w~ CL~ -L
W Ww o wO
Cc w 0 u .5
< CL Z =cc*< (c
w w ww w I
< < x *g UUs z rx
w (1) < < < w L
-~ u 5; 5
* ltss~a .flU .WWW WW U w - Nw W S1. U W52 ,Jrvw vmuwwm w fL WUWLV V m' USblluS rsIS u 'sUnwsL r n mS.!* .k r 5 Lw ltwS.-
21p
a -
03 JA li A-10 .MON M.l
<- Lii U-
LU 0t :1 1A *
Z :Y.U
Wu < WjLU W LL
o : L.<u 0 -C L
I. 0I Lu 0 - U L : -1- LZaUt u r0 3wLj w 0zJ
wZ~~ 0i o~~C L 0
0" O< oo <=
-, = . a <
CIIU z u WQw W < L
-k* z. 0j~ CO -. -. LL -ra
- -n
z
zz
w wI.. e
LL NE V I
w JLL1
wx
WILJ
*. X -C
I-. -dC. __ ._ ____
C)C
cc zoc
w
III xU -8
zLU -
LU
LU
zmX < OR
Z Z -n
u0 -
orIIIJ
-vr~v rj -u . -
~VK.
U) a
c4 U J
0 -L L> CC LL
W ) ).u L
LL L
0 _0 0cz Cwu L4 LL (~ n L,
0 uJ88
<00
LU 6,
z LEE < 1) <
LU Z 3
H- ><LU ' U>-Li
0 u < .0
0 _0 0
<Z Lfl z ~
Z<OL
.'. a: <p
-. i
-2I
o 2_ L
cn -0
Z -1
LLI J
u~Ljr T O a
0 LII
-1i L.U
MOT wi M "~j.vv *FUNP'7 MR~ FOKI i rip FN -wr2)-- w Km ATK - -- wv- W %n '4V~1d ' P 'J .r Jd -J- ..-
>10
200
ILWL
00COZ
an Q w
*.III 0 wmW 0 w 0 w
SW LL U) Z 0
lo w w wo0
UU 2 >- )-g z
0 <U. wU) Cl)0 Wo>o0 WC 0
x .0a
91
-,A -W wr-vv -v.-- -;rwm -.p -J. -,pi~ ~ -
0 A-
z
ww
MJ MI fzr I t
zZ w~I
w - w ......... ......
w 00 L
L L U ) U ) C
-0. Q0 w. a
a <U co " e)U 7=1LL< c-
z w >-J3 =
Lu o F3_ _o
z< W~
Lz oZ sm se il m I
w ..
w -r_
I. - <a-
w U> 0
Z~< 0w*
A~,NJ. .,. .L
11%%. w <: ~*
~ww r~. .. JW. .' MI V". . - . W"6 WN-u -v
4C
00
(0 LUJU , U
zw - UJ 0W~ ~ U, a:a
cr-0 z (1 LL
S - nw -cn IL 0 L Q
W' W ccw-cr z < -z m0 0 c20' x -
93 0 < = U
LU ix__ _ __ _
ox -C wLL IS r
tw o M U -M
U(A -*. C
q 0
z z~01
u- C
z ~ ~ -- La z J
LU0
LU L
~0ac
= ____ ____94
LL~ ~ ~~ N-. N _ _ _
~Ywt~~MUW w1Jvv'JY1wvw WL'W .rwv~~w Ur V.- "w LIN V 'h ''UI . - .. w V- k V w" - "TU"m u P.)Y F --- - -P"Lw r~p np . ~w.wjw-J.. .
w ,mZ Z W
I- 00 0- __ __ __
LLZ~ 0
x cr wo LL~~C Ii~-..
zS 0
w C ,> 1 w w 0~
C/) ' I I
0 'NLLc
w w c4 nzU V
cr w~c~
W a. > w
wr w~ In~. I--s I
U., < -- -- -
w wZ<2 LL=<
Z Z 0 a. 0 £wI z __jo
o < FnX
CLe % .0-J 3s
z U)m i ~-.- J 0) .cncnw r
QJ ECL- j3 .Lu z w OO L'
< Z a c95
cr. 4c Z _% ..
lm
i
0z
IL~0
~iILcc Lam L
Wm LLL. w 5-r w
0ca fW Zw C). w U
LL. i U. M w
- W
LL w :w>LL a. (n w
-U-(U)U <w (
z 3>
00 C 0 0 0 0
U) :
-a 0
U) 97
0
>-~0
'2z
L~~i
L
LJ
o as0I
I Cd
LA- _ _ _ _ _ _ _
s~~u r w w -- . .v n .u J Vv-t.V S IU'.. t tWtVU f ~V.. - - ; - - -i - -
z uiu
wo c~
- 0Z~D
0 -
w - z ul-
W < 6
> L. w LL
o LL Z0~ ZZr<P:i 0- o ww 0cc) wJ w Q(1 W-
W M a0U) 0 0-_W
'0- wn a.. w w w
Qn. -L cr L CL
4.~~~~~~ ~~~ V- % n z~%.% % N ~ . -i .. - .N
cIOU
0 z
a.
0I 0
Id 0
IL Zu-J
u-II- z.'_ _ _ _ _ _-
Co >L I I I
Z
a. N A
x ~LL (
100
w
mILL.
0~ CL
S... 0
U.. CL~WI
w (
0I a. _ __ _0 LU 0 0 0 0 0 0 0 0
cc U. -
en 00 >
us <, - . z -J
Z ZJI CL >ZI
-~ ~~~~~ .Z X> w z L W ~ J ~ ~ J J 'I- co < 2 a
<-~ < L (
0* 1
Z 0- ZJ- w
o Li
- x w. o' wm A > co CDg a.
0 101
LIUU
-%7L%
0
- r 0-
0~ >
- U Q
U. <
Zw w 0
o z c- z
LL 0< -J
I- 1 z 0
z ~ < z
wwU) Qw 0 cc
w
U) 0 L
- 0
< NUiSd3A QZId3J.fLdWC2
<< C-1
Wi -10
0 2
10?
%
Z LLLij
> z wE- ZWW azW
~LL (00
~CL L r
<U o< m 2 oz z a) CoLUILUO co
-<O < o_ (
>- w~ W < . LL u)
W > Xw 5Z LU w
LU c
<w >- 0Cl r~. <oa .- Z
0n 0 0j>(nz w <,
W < \> Z
cr. > t.- Z
0
wwa. w w
w z
LL LU0 0 ozl wu Clc)L
4 ) < ~ 2 Da
U. <0
~-01 2LuL a w -
4R 0LU 1 aLL -Jj Lu
I 0- 0 omu Z0~~ w 0 <
LLLIL wa: Q Q 2 --
LLJ 0x Q f W L
Z IL U 5 0w (n> w w LL
0 m 0 m
0,~.J x m - w z
I-- < Hw0 ow 0 0
o 0 LUCl) WLw U)
wI ww <m
>~ W HjFz CC
Z n 0 OCI) Cl)
1=0 w<H U) (nl W I
O w w w>. )- C <<I.-~a) z~U
(1H U)w L < wD z: >
Z Oz (n) 00 0L < W
-n -H~Z
C0C < 0Z t_ a. La acca < >
0 0< 0H U- a.
LL 0w w W Z . _
0 0n F- H L~~ 00 HC-L-
= > i cn 0 wH-W- w w~ ir ZQiii W W WZ 0
LL. (nz J CO ~ <
z0 0 W z-<JOo co < zO zMI O.w n 0
HO c r < > 00r
~ZU)
1S5CL
'~~~C/ w - .-- LL'. - .-
Li
<<
z 0z
0
- Lm >-
>- Co0 w
WW- - 0
www
3: U) Z-
o W0 N
0 C 0 U -
- w
IP' L F--
Z< >-LL
F- 0<W
wI U) -
w ~ <V> LL
( r4
w ccz* m
w1 0-j -j FCL w w
x* .i**\W F- F-
00
> >- z
Lw a: -4 0
- - , o0 000
>w~ W , 0 0cc1 00~~ </-0
0 w~
LL0 0
0 zM LJ>0 zCw0C
Z 9: 0
< In
C. 0 0 0
z L~e < C- -
0 0
U-di3VUN 3
z I-
0 <
)m L.Li CL(nE T. W5
-0
- Iz
,LJ zz-
Z -~< cr ____
<W cr
- < w
W (
LZ 2 L u - Il
< = c
LU ~,C, (L
Un cn~
LU L, (M
-J _____<
<0 0 1
~~w
A .J ; -. ----- -
L
wC0 2
w z
Z 0 w>
00
z Luo L
z>
U 0
0Au
I-
~~10I.-
>- <- W
.J <I.
o nGA3~ .r
j ._j
U.J
z <L A
- i 0I- z
Z ,i~
z 0.
< :3 cr
Cw (D bw OO -SW CL
C0 m(Il)
< .4
x>C<
w > (n
TO -R 7 7. a
0 z - ___
cr 0 cr
- ~ LL a) IKca MLL K )4 wOT
0LJ~f~
..m <
\uL j
Il-r oI.
-e CLL L
Z U, W -
a- w
~~ ~ awWLL ' -
Z 0M
00
cfl w 3 2 2 2 x 32 2~
0'U,. .c .
x 0
w m
-j
- <
ml Li
> ~LL>0 LU LU
jI C- zi
3: LL L-i LnZ n 0n L
LLJ-W
<W L
0 ocn
0 w =I- >- ->
LiJ~O LL
LLI
C- .d .'V .'3 x > >-.
9000
CL 00
(fl 0
LU0
0 =
-U)
zz
0 a.
_U Li . 0 0~
LLW _j LL 0ooLL 0WLLLL-j-
113L 1 0 L
< . < Q I
Fr "
o C11C',
0- atm a Nii% 1
a :3 w
L3 r-- -
0C 00
I*W
in LL w
1 14
0
z
0Q CL < II(. ;
ZC0 0w 0) z 2 Lm Li 0 a.
Q 0d
w- LL _ 0aLL- 0 Z Cz L.J
0 5 < LL < =<W OC LL < wca < ~ 0 j
w w < C < <
< a ~Z Z -j <0Z~ L4 2 0 L
CC ) :c LLa a> LU < L2 a:Li 0c
w u < > -P < m Z0 0
z z x0.L
L .U
z 0 0
UU
0x(0w
LL115
cc* -
> -J .7-
-J C)
00
0
Z - - --
W U Z
LL -a:L ZZ - - - .-- - U - ~ .-.Qz i a:-0
-L 3: I. .W*a
w
4c >
0 z > >-
x 1, 1-
wU.
U) <I
z
'z
x- z
< <2 -j
Uz
m< Y.
<_____________ a
x1
ujS
WW1WW 0~ r ~ *~~~ , -u.
iu
- U
J 0 -'
I-. £
z -I Cc 10
LU w 0
LUL
LU LL 0 .2 I I .
>-w - -
-J LL -* 5 sC L
0 u,
C 04
,** C , - - C4nJ<'I~a ~- -
<L~ <
U. 0
LLO< Z
<LL
w :
LU :D r o0 -- -. 0
<) zI -- C/):
a. w.13 -
x z w
.4 117
wi0z
(00
-L F
r< 0
F- wC0 a.
4ZI- c
LLLL
0~ ccF-Z 0
9L(n z -J
0> - 0 0Z~ 0 w
0 0-LI
00 00 Zc=0 ) W 0 <(n F- (n
< o-
00 0wF-0 6- WJ
e w 0
- ~r'A'V~ ~ "A~ '.~~ ~ ~~ ~ ~ -' VV~~Y~VJ.' '.XJXX~J?.~'.~'"~
~ ..- ~'~r ~ ~U1L
(0zwnu
00 z
Cy LL J uLU
z c n, -ja -IU
C) z W cn < a I
-o LU w-< x 0 UJ
U. J 0. W l .
0 -
(n z I~,< WoL C .<c
Z U3 CC z zz
z w w 0x o 0 -ja 0z * *: * a . 4 w 0 - cn. L
I-a~ r = * )= ..
w_ o p -i a - x 0 m
o 0
um Si
v
P7WVV. f . W.W~.. - ~ ~VWJ~*'~ .. ~ ~* . w,.. -. - - -
U)w-I
Q p
U)
zI-s
a40U.'
0
I4 4
0Iz
U'U
i
N
-p
-4I
0 0
04 41 ) 0-4CO -4 4 .
.u 43 -.4 . 0 I d*4.
4Jlj..4
o L.S. 4 41 rz"d a
121
LLI
o~ 0a
zL Z
wm -Cl" 0 ? u
z 5.
"z~O
CC.,
Z fn< cr0
LL0.
0. (r> .
0 0xu~LAIwn
U.)
z
C.l 0 0.
CL <0a
Z U.1,>-0
LU a)
IN LU0. 0.. w r - _ _ _ _ _
~JI-..1
000
00
23J
0
ILII
I
LU
0c-
U0
I
LI
a
2 4
LU - . .
LU
*LU -
3 I
2 . *
* 0 I-I -K
.j 0-J C'o 0 ) 0i-Jai
*z
C.) 0
0 0 0
0 0 M
a 2 0 c02 ' 2 E E E
lu 0, CL 'a
ou a o 'o a m4 .. 4
LU oo 41 D
z1
4)~ 4431C~ 2~2 u4 In~E E32 2 ''
f 0-
0: CO-0 0 C
0 - z1 ~ ~-4
041 04 "0 4C41 1C U 4
41*I -
- ~ . CL O04 4
- UCL
ccU :. U U~
-~ 2~ C C C C26
ww
C,,
LUL c 'aC J O.
z 0
~~C -E -~oCC0 iuLu:1 c c
LU U~ 0 c C
cc E -- =2
I (CfL W a3 E0
-0 c
0.r-
C -)
Mo c 0-0 Cz ~~o,-
I-00C
1270 ;
C,,0
-
CoZwL L .C
zJLu-
0 >~IQ i..qwz
19
ZN1V 1011
<W0w a
I~w 5 .0
OC W 00.-
Ic 0o 0
Ow %
0z z <o
ZC , <C
CC 0 < >
0 0
> ZZU.
xoo
129
coo
0 C4.6AJC cc
>I - =
;3~W -0 0
z 0 .0 I-
6U~ ZZ Z0.4 I
UL
UC z
z a. I0uj~~ C4 7-1-
Q UZZ W
U. 0 -C-
LUU Q a
so o 0 2U
I-co 0O0 -K Z
U a" 00C0 '.6 0 T0 I I
2o 0- U '
13
Lo 1 3
w
0 C
- w a :: 0 0 - =
w z=
> szr - zh"'P-. S0 0' wo-
- - 0 0 o
0 ~ 4 0 12 02 0-' Zc~ 0'. c~ ca 0
0 u - -i=a~h-w o CC - 0
-z 3-' cc' W, -C0~~~~~ c3 o0l a~A-~0 s .. J - H.
17LC, ~ c w ~ 0.
(2 !2 C3 - a 0
0z
PUcc
pv-w--vwr~,m ' F WT ml Pon V% 'V'np,% M7 R F N,~J 5 X~ Ow l~ Wr- -r
CL 0.
Ak 0-0 In
LL w < .z a
wI LLC0AG
W 0a,. ~LL - -
z -I ,
0 0 L L U .-
< LI .
L ujcII
n~ pz
0 w 03 2 -
--
Ow 0 C1 aWj04
<. 0
>U C0CI - 4
z U)L..'-
Cwi<0
0
Z ZW z
CL 0- w x
-v-. PIK N, '.3W -A 12 -, Iu
w0C 0 4 h
4 0'- 61*-4 -0 C)
-~~ 4 al4
m4zL 4 Q 0 m 6
48 ~ ~~~ w 4 48C ~0 4
. C N 8.. rC 0... 04~c .4 17 0. I. - .
0~~~~~ 4 2 4 C 8. > '
.4~~ 0 r Z 4 ~-
z 0 0 t 4- I
C 0' 40 E 0 w4
- 0 0. 4 L) C 0 v 6.0
4a in20 4 -- w~6 2
2 0 0 40 4- 4 4.4E
.C A4 .0 Cr 0.. m. C)0 40
- 4 4 w 40 41 2COW 60 0 . 2 . .
2. .44 0 .0. 7 00 U.00 C C 44
~ . 4 0 00
4J 4J t- C C:.
m 2- 0s 2 0 w.0 1 8.
d8 - r- a4 c8
-0 40 CQ 0. 0 0Cw 0. 0. C6
0041 84
o zj,
I
U
a
0~~~ 3- - r-C -
-S.- . S" A'' A""
3 1 3-". "- .,-- I __ _ _
~~._.
,z
I0I
-33
F - 'T T
,j LU
_ L
03 F=--I-s
1, E 0 C
0 (n
Z -0
- C.
C - -0
0 u 1 - -
:U ITO~ -4
w
w owIW 0I ~ L~
0 ow:c a
_ z
10 a_ _ _ _ _
Em ~z
CdR
In =(0LLI cc w -
(z Ip wLI4r
s3oC.
U- (0
'i a a
w 12 -9!! ' fl - -
L!L
0
: qii i!£_-.
ism
cc
-I-
137
a cncr 6. "0~
o, .C 0a7 ,
@2 00 -Z - . c~
0J w-
C-- z @
-J LL o6- LL q)V 6
=5o LI a @. a .ca
l- 0 '1CL.
z _j"E:9< < Lu ;a dc 1 - Z
Z2 2 66
LU ~ ~ LL -
o~~~ Ia x 1 22
CLC
C. Z> > ---
w< w C) w- - - 2,, ~)(~~~XOW0. 93 U . ''
> w.. x U
> w <
z1380 r4
~ ' ~ ~ s. **~%** %*.****~<'
I
(0w3 ml
(U)
(0-I
z0
zwU0I
/
-I
A U
9 1 V?
w
LL
IL CL Q~4
z0
0
zw4 ~0 41
-- %
a. Ih 6 . 4
-0 w
0>.00 41
0 L0
0.
1. 0-
~j2. - 1 41
>0 0
a' D. 0.
OC 0 I..4.
w~~ 40 V) 0
c~~ ~ 0L 1 c0 t
eA -4 1a a I
UU Noc (
-~~ *4140 16 C
0 L
- I.)UC CD
CD >-C cu
CDd
- C
zl
0141
44
Uj
Z3
Ii.. - f. ~ 142
0Nzz0 .
Z LL
C13
CC CaD
0 7j r:
0~0
>-'
<
V 143
~'FT~FP~I' !~p '~F'~P'X. ? ~ ~ 7WV~ V'
I:
w
iz(U)
wI-
I-zwE00Dl00-4 4a
0C-I
0
-Dl
(I)"aU
0
~44
I
.a. .aP
zz
U, -4
LLA
-
_2 -3!>-
U, * a3C
ru
M cf
(n0
145
Z j 0
LU
L z
z~~~~ o..0~ no~~
LI g 7;i-
0
.. ,, ~~~~~ ~~ ~ ~ ~ ~~ ~".R A 'A"!-. :WI%7vrr~ l- f-7 Y WV W V VV~J J ~WV ~ ~. ~ -~*
>..
W CL
2 Zm LU 0-CL
0 Q)
a. >
0 0
U)Z0a
U) CD
< <
zL 0 *
w < w 4
P > <~
a. 0w <L
* 147
U) C M- cZ 0 - ( j L)J Lcc z LI Uuuc
_ -Lj ZZ- D -
(n -l LUZV) uiZ
2 - ~ l cc x - .__ _ _
Z - Iw 6- ,- -_ _ _ _ -n
LU j C - 1
zz (4 - _ju zc : C4' mt wx<1 "0 .-
A- ; -.. C C dC . .p ( C JccW W CC C3 - 2 (A
Cm M U.tbo' ' ( X0" 0cn -M LUZO w>-
- - ----I
ix cc
oC wo m ac-
- ... ~2-.t.** *2 = 7.c *.. . . z 2*'*~~~-£W I 0040 Z
z -z
~~~ I..-~ <
Z3 - L> _
W*x0 -Z Z z- -0 x ccI x X
LO 3~ .0 a Z~ a. 0
0-1J o- l c
cc cc'Z j z~ vi z -- X
<~ ~ =______________ w - < % Lj C Zc, C- X.-40zz c o. - = (IC _j Xwf .. ( 3- ~ . n Lj
a.) Xfr W1f x vyw - I
2<~ ~~~ ~ o-z '~ z ~zW.C a. L,,,-f7
Zma ZX_ X -
I.LI -x -nd
=zxJ wZ cc
- (A- r
xU'
I",I
vi -
L)
ud U
TM.
H-irU'
I-r
LL-A~
zoz0 C)
Az
<.
CLn
L) LIM
I-
LA.-
0/
wI
oz w.-
C, -J- 1:
o CU _ _ _
U-I
0 _ _ _ _ _ _ _ _
U-j
CD-
< -m-L.i--7
U, z__ _____
?v'7¥ WYPJ, w .. ; r U , U . - . k ' . '-' " -' . - .- ,, '. .'- .- ; V . '---a.. - & , . . . .
II.,
I-
U
I,
,I
-I0
15?
0n u w
Z <0 <0 - a n LL U m__4
-U cc <_
0i LLUl 1-4- -r4X < C uj W0 2
"-4 m0
A LU4 L0f z U) '-4L.
M_ _C _ _ _MCL_ )
z ~ -uU0Y-L .f)-) ~
uj~~ci -j C -
a_~ cu) CL < < )M L) L.miI- x >Z U
z Li~
M = --Lw 0 - 1
U) I(fll
p0 .LLJ
C, Lz0 CC'0
wz r
~- 0.0
U0 0 .
o >
0 0 .
> X
*~ I.
<-
C, 0
<. a. L
-w
> <0U 0 >
.0 0
WJ >.u 0i ~0- 0
C1 M< < 0)< ~ Z <0
154
4z LL
< U
z 00L<
OW a:
< x
0z <-0
CO !00
-J y-
zz
CD I-0~ 3:F- Fu w
0
- ~~ ~ ~ ~ F <4* ** ~ds. ~ S
<' d 1
.m
zw<C/3
< <zr -iC
I.-cr zI
w0 0 L c u
oI coC3 !0 U U I>- 0
0- m0 2 LLL
0 - W 0j
z LLCfH z < o 0w cW 0>01wL LL 0 0 0
LL W ccn 0 w0w
0 doWf
J0 in 0 ~ W 0
CD~~ -J z L0 -W
00 w <wF
LL Z- AL <0oU cc0 o< -)WCW003 0 CIO -l < W w
m.1 Un ar M
0 0,0 I0 0 0)
0 <- LL 00r1 -0Ci - -C 00 0~ 0M
D-o- r w CC') 3
< 0 LJ m - cr LL 1 6-i z0 LU CL Q w LI I
II-I
0
I-
I
[-157
x _
C)'
40 . z >->J
LU 2 >'
-j Lf.
a- Iu <
0 I
j 000 0 0-L 0 CC >o6 o
I- u
as LU i 1s
Z *' .'. ****
-JL
4wL
CLC
Lu-
LU 2.
0 wU-A
w U
L.
0
cm)
L,-wU-wfllr~~~~trW <- - r - r -~wy- - sLWvsc - -'*- . * ---- - -J
Qz
Z -L
0-
a..
-0 _ _ _ _ _ _ 0 -
-n u
= U- - -z
Nn -6* _ _ _ _ _ __z
wLAA '
0i
%w
0 L
Lo L
- cr
I--I -U a. D-
~o coCL'' z_ _ _ _ _ _ _3~ LL <~ _ _ _ _ _ _ _ _ _ _ -l
3l - _ _ _ _ _ -, -
> <4
o _ _ _ _ _
- --- - - - P ~-P ~ ,. -~J-P .P1 PP P.Pw- ~d
im
LAJJ
,A.,-<-
IC,
C/)
X WJ
N..
ca
Z -
C, -- 4
0LU)
_j ~ LLJ
ULL C.* 0.
i 0
LQ/ 0l (za L - RoL 4 * *x
- O ~ CO OCW ~ - - b
-> -- < R 0
44 UC/c/ 0. -
40
4 ~ ~ _ ( -:C)
LLU
LO <-2
> <
-. P-- 0
LU L
< ~ < LO~
- LA -j U
W(,) < )6
(fl(n
10
A,
z -
~-.0
r---X < U-L
00Z-3 x
-Cl,
(n -L -
~z r
C'
-AJU
LU
N - - *J
gJz<
z Cm
z 0U
<La
J)-
z -
--
-(0 c -
7-
z z
CAC
LU
>' Us)1
-w n-Co. 7. 7z
rKwuw..~m~-- ~ WTW~IF~. WXV~V P~ ~ W~ ~' - -~ - - -~ - - -
x-J
I.- 8~ - 44*Aw - -
_ / 4
X ~ -J 2
W ~ 4
~ *i<-< -,. -, / / /-~z :~ ~, -~
- Z-~-~I-o 4
4 X5~-4 / /~0 ~ U '~ 0 W
-J o~U z
4 ------ "---------
- <- 41 - 0 --
~u,4 ~ I~
U - -~ 44
4 ~tf - -4-
(I) ~'4 -~ ~zC~,Ez ~ - -4
-~~4 ~I4
IU- -~ -~
0 ~-4.. 7 -
U -
I- ~ -44- -,
-j ~
*1-~ - -
W ~- -..- -4
- -
-.4- 444-
Ow :~- --I- .4 ,-
4 -. . D
-~~~~~--W ~ ~ ~ -- WN~ -arw WV WWU-S A- z " . ~- V . -
LU .
lLU
II- -a
z
x Z2
o
cr.-
ccz
AZ
.mEL
0 6-
IJJ z
c- Z f
z
n~ - w -i
LO F
-4
vi DL"I -,
'0.
N s
Laiu
C42on;,C-'
GqN
ulu
xanal
- -t .- - - - - - - 43
4j2<2
~ >
LA. 1x -- _ -'<
W --
07
w ~ -- _ _ _
ILL
_ LU H
:DHZ 0?
0~ <I H Y
L.U IL 0 LIJ -Co- - ~ 0)
Cl) Z y0 C) 0UoZL 0 0
m -r z< < W H
Q) 0 0LLUD
-~~LL cD 0rL ZoO OLUOOLoo 0
0J CO LUH -J: >
0 LU < 0L Z H0
o L wc< <z i~>yJ < Z<OZL
LU LLJ 0 0LU0 "5 Hf zZ
* OL C 0
0 LL0z cr) > LLLi- 3
0
LL4
001
17-
~-' - v-.-~. ~ ~Y~* ~' *Y*' ~ ~** ~ j~ ~ . ~W ~ . ~ *4 ~) ~ ~J.l~4 4 4 *4 .~ -~ - -
i-J -
LU I:- --- -4.
a -~
2- -~,
1< - - ; 3 LiL~A
I>
LU .~---~- ~-.-- ,.-.
LU - b~ _2*~ H- -~ -
* ii., lit Ii liii,,, timimit ~ mliii,, mmliii 111111
7),o -- a. - - - -
4.
U) -
(.J 4.- -Z _
I-
* -s-. -a-
Li
- K'
LU , zr *~ ~
-, (-I-L..
('I,
-J
- *. . -, %.' i*,*.* .. *\ **/--- 4~'
MSflS 4 PROCEEDINGS OF THE WORKSHOP ON THE ASSESSMENT OF CUMWOKLOD HIEMMNI. (U) DOUGLAS AIRCRAT CO LOWSUNC~CA N A BIFERNO ET AL JUN607 ANAIL-TR-S7-3043-YOL-1
w LNXI SIFIED F33 15-06-C-36M F/ 5/9 N I
E:7hhEEEEmohmhhEEEmhEEEEEmhshEEEEmhhEohhEEEEEhEmhEEEshEEEEEhEEohEEEmhhEEEhhohmhmhohEEE
S2.S
10 L5 '11111 1112-
IEE.0
;I 11111 1
Alp w 1w l1
UP 10. w~~~~~~~~~w-%vrw~~~~~~~rVW~~.W 1. 9. wu V VWrUW~ ~U VWEM UW - h krtwrwtWr W WT W lr~ -r~ - .~ mp i
zU00 0 W
w~. 0 0/ wiEuL w0L
w 0w CCW
w 0 L0
-j .w- cr~w~ w -J
i0mh 0I LL WZ3 fl - 0 WQ W /I-
z w 0 j-w
0 t7 T I- w LL Z*Z <JOn~r0 W Q)A,
y0 zI 0j -j ,a w- 0 0 Wcc
> Zw w w
w ciC0 0 m
W 0
175z
4 ~( -'
w 04'...~ A
x: 42
~4 ~-z 4
AJ0
4Z
~t I~jU ~..3
':~- 'i:04 -~
444
0LU
Z 1 A. - Z
4 ~ AI ,
z &~'z ~'j~
:- :4; ~ -4: ~
< ~ ~ A~J~ -~ 4~I
'-~ I A - - -
0 Z~ ~ 4
LU ~. ~ ~Aj23 ~ 2-3- - - - '.r:- -I
a -- ~ - -
4 2 3
z - 3 eA
LU A -
z -~
~- - --4-4~jI ~ z
00 .
0A z - -~ ~3 AZ
~4A - 2 :~-ta
~ ~3A 4
o ~ ~
:a~J Z
~3~ij r 44
~1 54~i ~
- A ~!~~o~-4444,- I~ 3~ '
U-_ 4£
44 A-
-4.
-. Z
~ -
- -
'422 -Az ;::~-- 4.
0 A J01..
AA
4
4.0A -~
4 4~ --
124
-A ,., --'~4 -- --
A.-
176
4 *~* ~
C,,
a:
I- cn-L -
WL Z
2 I-
WW (n 0ca >O xu w -j w
cr W L (nO -J x
w Ccr w w 0 iw -J
-, m W / (. <L W.
0 0 z 0 0
ww
< 0
.77
EFFECT SIZE = LARGE100 ~ COVArMO LO
to0.
70-
s4o
40'N
30-
20-
t0o-.
0- p I17
EFFECT SIZE = LARGE100-- CO. 11O111H H.IGH
100
90
70
so
10
0-
179
W v
EFFECT SIZE100.
go-so-
70-
so
go-S40-
20,
10-
A aLO 4.79 S -. 41B 0 .00
180
wwwwwu~wWwWWWWr VOW. UW l iMTV MTV- vw'w v N -KT -i v g w'vj -.. ww.pw ,'..
0<
E I
LL w00U
U)
le 0 at
Z6 zz
LL- X
00
Lud0<
m
z In
.4 <. 4 44*.- ~.yV.-~ .*** *. ~U.,
.J LL-0 0-< c!)
z-2 0 '
0J > 0
w<cr wzc
m w
I-<
CA z
N z
II ui 0- 9
Z w LL
o XLI. c-
LU 0
wo I <(0 0 <
(A CC OS3 ~Z~liO(5(0
w p-
wmI- z
US -P<L < 0 a LU
cc18?
< s- -
(L < .~:&~V' .~i' .' '
.rftfnr~fVW u 'Uf W .~~A-zL,~,r'C~I ~U f ~v~-'V b~v ~ P i~.'h1,z s~_W 71j
x Lz m
w0mW1< __ _ _ _ __ _ _ _ _xx
W I,
CL0z -W
Cl < w c
o~ 0(U Lii -
20 2 U, wi~C~W Uoo - t) x
~~U3 <LWZZ
x P o
I.- <0W cn
-J x 3 1 1 1 x
5 U. afoo0Z
O~Z
UW> <z
m w .jwU) CCw
183
SPATIAL PROCESSING100
70-so- goo
140-
30-
20-
10- -
0 ,
a LMW " lof
I
I
184 "
I
PROBABILI-1-Y MONITORING
so.
70
o
oo
140.
20.
10 3'
a-:
10- -
,
Ii
-- 4,'
* ~ ,~ ~ w~T1P ,WrW Y W V. . .WJU-- .-
MATH PROCESSlNG100!
70-
so
401
30-1S
20-j
186
.1, C
w igU W 'U W W WV V U U 'W ~ VU W U 'W W rv wrvvrw .V. WV -- - IV .-.-
.- g.
-v -- Wv s.
- .-
LINGUISTIC PROCESSING100
0
go-
70-
140.30.
20.
10 p
0
187
GRAMMATICAL REASONING100
00
70
200
i.
10 MGM
U--
1"0
0-- I
p.,
188 -
CONTINUOUS RECALL100.
so-
70-
4 0 _ _ _
30-
20-
I0-
0-
a LOW *~~M* 1G
'INV . 16
MEMORY SEARCH100
40-
40.
30-
10 ___ _ __ _ _ _
0-
a6
a LAW MdEDANM No"i
no
plo"I
*.d~d -5*.~'~' - ~ V- ~ ~ -'
~ ' i,0,
B-52 TAIL GUNNERN 13
100-
DO -
go-
70
Go-
i4030-
* 20-
10
HIGH ALT LOW ALT LOW ALT MALFUNCTIONS
HO*=lL TARGETS ENCOUNTERE -- LEVE
'
i~
-C ~ C -* ~ .. ' W .
"V"V'.V2WP UL-w jrwvW LIU 1tmv 1T. I irw 7n - - .K - - 'Y.Y" 'V'J ri '1A7dVWJ
F- 16 DEFENSIVE
go -
70-
50-
40-
30-
10-2
N~0- -A
E3-52 ILSN - 1200-
S0-
70-
140
30-
20-
10 e
0-S- G TTL. 2 -NOR ILS 3- OECOLA
LLE
1 n4
'0- 'IF -I
I-
--
--r9 , -r -r
II
In. .. . . . * . % • % °% '. ,, ' ,, -
cox
0I
W~: C))
LI)J
Y'. 4-)
L-a
0L ) J H
(Ua
J~h
F.L
40 a.-4v
1~ q6
zz
aa
z,
cc w
44 IL
Er- - - c
LU N - mw40 4f
z3
X -o
*
0In8
'-C,
i-a 0
40fn
I n7
- F
ccz
w bw
ww*w
LL Wi
ZLL.
LL, -o
asZ a
w~00
cc CA _ ___ ___
-- Am AM _ _ _ _ _ _ _ _
c.
02
cr.l
S . --
rP WFTW ~~~~, K- 4WW W -l NO VL AMI. %- W-ZT"olw
CONTEXT STUDYS'T1.13Y I -- SWAT
100.
90-
Go-
70-
LI:20-
10-
1 2 3 4 5 6 7 8
PRESENTATION RATE0 LOW (RATES 1-8) * HIGH (RATES 3-6)
STUDY 2- SWAT1009-/70-
30
20
10
0 I I 7 a1 2 3 4 5 6I 7 8
I1
CONTEXT STU DY100-MrI
so-
70-
60-
10-
0-
PRCUVET11Ot MATTa LOW (RATES 1-6) * HIG (RATES 3- 4)
100- STUDOY 2-IPOLAR
*0-
801
40-
13020-
2( 4
N44
4j4
y'..
.4. N 911
S ~ ~ ~r ~ ~ .1' '.' t' ".P J ~ ~ ~ ~ - I
'3
Sf)
4%~J
KkNJ
N
4
*I~)a'-
N I
p q
p - ~ ~ P ~ *~ *. * ****~* ****~ ~ -. *p -- -.
44%
kXk
Zzlit,
201
1/41
IN...
"04.
ot
IIkA
205
V
*
- -~ ~ '~'~' '~ - .,W W UY ~rw ~r ~FU 1.U UU IPW .~4 V, IJ-v W ~ W. ~ ~ - ~ - - ~ ~m JJ-u~w w~qyu~.J ~ w~. mr~
~
r:z 4
NJ
K(44
4< KNK
Nb
B
IN
'iJ I~1 k
sz~
a-9
206
I.4
' - -. -. ~ *& *fl~4~ 4%'t~4 *~.pbm~~ 4t 4~ ~ 4.'44'4 I~.% 4 4 4~ ~~''~' *~~*w~* 4 * ~ 4b 4 ... d..,sJr.... . 4
"MENTAL WORKLOAD MAY BE VIEWED AS THE DIFFERENCE BETWEENCAPACITIES OF THE INFORMATION-PROCESSING SYSTEM THAT AREREQUIRED FOR TASK PERFORMANCE TO SATISFY PERFORMANCEEXPECTATION§ AND THE CAPACITY THAT I AVAILABLE AT ANYGIVEN TIME. GOPHER AND UONCHIN, 198h.
"THE CONSTRUCT OF SPARE CAPACITY, DERIVED FROM MODELS OFATTENTION, IS THE MOST IMPORTANT COMPONENT OF MENTALWORKLOAD.,,
'HOWEVER, M NTAL WORKLOAD IS MORE THAN JUST SPARECAPACITY. MDDITIONAL ASPECTS OF MENTAL WORKLOAD INCLUDESUBJECTIVE FEELINGS, EFFO T, INDIVIDUAL DIFFERENCES,STRATEGY AND PRACTICE." P&NTOWITZ, 906.
"WORKLOAD IS FUNDAMENTALLY DEFINED IN TERMS OF TH'S~ELATION 15 NEEN RESOURCE SUPPLY AND TASK DE.MAND. 1ICKENS, I4.
'WORKLOAD ASSESSMENT TECHNIGUES ARE PRINCIPALLY nESIGM.DTO MEASURE THE DEGREE OF OPERATOR PROCESSING CAPAC'fTyWHICH IS EXPENDEg !N PERFCRM!NS A PARTICULAR TASX ZYSTEM FUNCTION, c3GEMEISR, _mINGLEDECKER, AND
URABTREE, 8
,,T HE DEF!NITIIN CF WORKL3AD *,: TERMS OF THE AT70nl"REQU:RED 3Y A TASK, OR THE ADDITIONAL CAPACITY YETREMAINING TO PERFORM ANOTHER TASK, WITH POSSIBLEREFERENCg TO THE INTENSITY OF METAL OR PHYSICAL EP=-EXERTED. HART AND -HERIDAN, 194.
I 8, T KNOW!' NNOBODY ELSE KNOWS EITHER." KANTOWI
*THERE IS NO AGREED-UPON DEFINITION OF MENTAL WORKLOA7AND NO AGREEMENT ON HOW TO MEASURE IT. 'ORAY, 19S2.
?07
. . . . . .. . . . . . . , . . • - . , . . . , , • ,
,,= .,.,,,,= . .,.= ,,w,, .o... ., .,. , - ,.,.z .'. , ,.., -'.-'.• .'.-. - -' , .' • .- ,..- ,. .. .. " ",." " -" ," -.' .
FOt Pccvoij;t.l:
SCo nsTr k.et V&;A t,res OP e.,& I*,&
C r tP. ri Va
fr~itelofA f ~I )4C*m$
* 208
2 4.
CL4
22
209j
ItIilli
.1 I
3AI
NO,,/
u I
I I IF AA IR IN I 7TI FimM - - - R r 14 1fTw-S~ I -7 -ww~vwq
rA S 9
ploAT'e Ila~
COMPLCvoIV S12 12.
Thi9*u.wg/ I) f ,'
HAIA10681A& C4j1 I
&a&,~~u (%S 11.+,,
%$.*t (-*S.A, %Wei .S ~ ~ i. 0'o .33-6 w w7
wmk- It. C U1107)e
* (S.~~~~a4 L0I.~~~4 m.t. Z*,.
'7. 4b oft. 16. 3g~~~~wa(w
I0
S ~p* 4 .2 % 1~1 4 4 9' ~ ' ! A' V ' Vv .' J%
o ,, . ,,$
slot*
yV
'I
*1
I
i
~II I -
_____ _____ I -
- ~-
C. H* .*,~iI -
V0~ .~.'0 -
IN N
I -
N 0
I - 0'
~
- C I,~I ' - -,
- I '0 .q.
-~ I
41 M --o .1 i I
I. II II~ *
___ I ~K+Kii: .-*- I
C I ~ '
4~IN -,
'C m* I~I
- - I.'~ In j [ '0* 2 0 0
K -iC N
- I~ 0~Wi -
N' 0.
-I - *
WI~* ~. N 0
0. ' N
___ W~ '0~ N'0' -
In -'C 0 ~I -, In 7-
-C * '0 N
U - ~-'Il~~ ~,
I _ t- -
- I >1 > 'A
I I ~ -A
I~ -J 4~. -~ 2~ 4 'v
-~ 'A
?13
~ .. k~ .&A&.a .A -
-~ ~ ~ 7 - .I4
Ab
14I
3'~~~~-C A7.J e~F.
U ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ wwM Uw MwL wWUvi&WV MUM WMJWLW .'5 VW VWJ -~W V T w.. '( W 21%~ VUVW wV V VW. w. wv w2.-" .. VJ..W7"Nw-j V
I;.oj -' 4..
-J
4k'l-u IS-,.-
C~~ C~zC.~. FO t tal. 1 t
~ Iid
-cl-1"
1.... 4C LA 1 A
AI A
0.A 5 . *~0 IA .5 -A C A i . )....1 ~ A . - 31
CIO ~-.5 -.. - A. . C C
- -, A. I L..-n -
Il AL C -- -C - - .
o 0 5~ A
1jj
g -6
' :i#17
- 7-
SM 40.^Af a NPORO. - i
• i- M -- f
?Iture 6. Somnled or-or erfnmonce md faces@ controlcapacity statistics vitte he crlss-couyiLed task 4isglisad iAm the spiral divergence of a uimulated C-SI STOL aircraft. MW a'-w".The plan elevation of the intended approach profile are alsoshown. The aircraft was controllod amually by a pilot, .using raw situation data under instrumnt flight ruleswithout any flight director. tach symbol represents thetim-veraed man value and plus or minus one root-iman-square value betveen the numbered vaypoints on the approachprofile. The adaptive spiral divergence was cross-coupled w 'i,'f- , ..to a weighted linear combination of the three error pert- >oremace mwaures shoo., and the weighted error reference4ee ten per cent greater than the pilot's own baselineeor performance without the cross-coupled adaptiveloodiag tas. The miumm poesible value of the excesscontrol capacity nmaurmnt wee limited at 0.25. Thismaxim vae was reached between w ypoints 3 aid 7.
218
N N N
UwVw W U V Pt9M1 M/r W R4 RAW f PW U W WV MV WV W U WVwW' .W WVWXW~ W -W .... F r\P -tr - -%. -
?RM k Tb S
6% Or ~C CA 'TW.L ACT i v r r
WAX&.+t~ feu
LOAD
270.
z0 00
z >4
Z H 0 u
z ~ 4 o0wzE "zJ
0
Nu 0
00
0 10 02 o
gr4 00 E I-
o C E- N
> 00 a
0 a
221
4t:
b-4
z04
-P-
'-4 0 C
0
222
Z 00
'-4-rn< 0
u- -4
m z<
N >4-
>4P 4 0 Kc01 4W4 u Z p
I I >4
E-4
'-4
223
0 0 Lz-
Z H0~ 0z
0 Zz Uz
0 zz 00 <00 >-H
>: O1-4
l< :00
0~ 13 0: >
'-4
09
0>4 (:4
z.
224(:
cl)
:DC/)
z
E 00N-Z
zz
o '-4
10a-4 z
0< 0
"1-0 z/ 0- Zl <~E- > 1c- -
<0 c r-
z 0M,)
1-414Z Z 01-4 NH> 0 H
<Cl))
z E-- -I
-- 4
00
<00
H>
z zW0
U u
226
KKR
1.0-
05w
0.0
-. 5
-1.0
LOA 0
Figure 9. Mean normalized scores as a frlictiopi of
load for the control movements ,tmeasure.
SOURCE: WIERWILLE & CONNER (1983)
4O
LEVEL OF OPERATOR PERFORMANCE
0
0
0
0
L
I o
>4 0
0
z>1 0
MH Z U
0 t-4 o<
o< I
> Liu Hu o< > o<
9=1 o< o< 0 Ww
H>
229U~iZ~
z Zw 40
a4>4"
u 0 13Ao1< aZE
O 0- 00 0 E
O0. < 0
O - 0 Zi-"H-Z o<HE z i>,-~)0 ;- z>
r z L0Jo- H
> CH 00
0 Q
C/) >4 C/
<0 H>. > 0
~zJ
H ><
w :4
230
r-44
00
z z OD c0 w C
0 '-4
E-4 NNr-
J- FA- zA
.4
H POOR HANDLING OUALITY
GOOD HANDLING QUALITY ,*
0.20.
0.15 CONVENTIONAL KLOPFSTEINDISPLAY bISLAY
cc4
0.10
O BA SEL IN E".
1 2- 4 1 2 4
MEMORY SET SIZE (NO. OF LETTERS)
Figur~e 6 Mean Percent Secondar Task Error forNemro Set: Size (Number of lAtters) byDisplay rormat and Handling Quality
SOURCE: SCHIFFLET, LINTON, & SPICUZZA (1982)
?3"Io
".,
>4Q0
z z 0C0 w
0
z -4
0 0 %SH~C 0* -
0A> z z
E-4 E-4 H
z 0 - 0 3E-4 0-
a Z90 0- DzOH = H
E-4 '4-
ZO z I
133L.
RAAC kcclc Time
ErrmZzors
1200 SO
1100 A,40
&4UM TU 10 30 Z Errorin M540C
too 20
goo 10
Figure 5. Secoudar7 task performance (.eactiont..=e/ES--rors) as a function of level of :rizar-vtask.
1KANTOWITZ, HART, BORTOLUSSI, SHIVELY, & KANTOWITZ (I.-4)
n34
56
mea High workload
52
.~47
f 45
S43 -manLow wor
40
3434
Trials
Fill! Effect of COekPit Workload on a Auditory Task
SOURCE: GREEN & FLUX (1976)
23 5
UV-z
z z cor-4
f,:..
~z Zco co
[-4:
u 0 0
_ :z
> 0_0 _-
>D _ CC
f-40'-
C E-
00
E-4
- -S.-- - - -vW.V-kWb-W
1.5
1.0
0.5 B
0.
1.0
Low MEDI 1UM HIGH
Figure 3. Effect of load on ineati standardized scoresfor the timne estimation standard deviatzin technique.(Means wvith different letters are significantly dif-ferent, p < 0.05).
SOURCE: CASALI & WIERWILLE ('.982)
?'7
00m z tr-4
z>4 0 0
z z
H Y-4
occ 0
0 (P
z-4 EU
00 u
uz 0" 1-0
0 z
0.4 0-4-
CY4 0
?38
,r V w-v -L x -AL- L-w-w- w i! I
COMMUNICATION TASK PERFORMANCEAS A MEASURE OF COCKPIT WORKLOAD
40
-CHANGE IFF CODES
U/ & SQUAWK IDENTz 35-0
° 30n
-/ -CHANGE RADIO CHANNELS
z & REPORTING THREATS
- 25 //
0 20-z
amx
NO ADDED SIMPLE DIFFICULTTASK FLIGHT FLIGHT
CONTROL CONTROL
SOURCE: SHINGLEDECKER & CRABTREE (1982)
0
E-4-
.4.4 4Z H
E- 0 > zC0 0
QL4 0
a4 Z X
z >4
0 0U H)
-4 04oH E-4
E-Z 4
H o<
Hz ~ z.4 W4 a4 u 0
C/) HP4 W HI-
o 04
240
erio
?41
bA
L
0 . ' -
"C
4, L,
P4?
,IL .,,.- . -,.. ,- . . ,","...-, ,,.",". , ..'. . * . ',,- . " -,,. .., -" -..27 7-7 2 ' ,. ,, , o o"',. -,
A L-C4J, 1 6?A-~Z L~-"----~
/t'2
~ * ~ -
&- 0-4rv~4~& 1
-~ ,.4 /7 ~ /% .~.- -
4 j f;L.
4
- ~ ~ ~ 7 -
L\~m~A~ ~
~1
.4
-. *--
~ -r A ty5 /
.44
'4
4
A
I,
243
N~~~~~~~~~~N~S ~N9.JD N~'D"Z PJ~' 'WIi, .W . W ~~ r &4 ~1 ~WJ .
C ~\j~svrL A C~eilk ~
a'-
.1~~~O L'dJCCa
~-S~C. V3 t * P~
'N'
4?5
WP l 'V" V Xr''fY- XTT'L .-%rA.rWM V VX
c C C 31 lt P
C~
ex7.4l
~ S ~~y~di.\ rt- & -CGp .
So Ly~a
VV-l-nW U N" W-w-vvvv ww - u ,A s -% Twv*&b
C c2*. 0- t.4 I A u o
,.sit
'L v-4
247
%Maw
lot -oz. lati.) r e-SP
3-r=
4., L -Z
')4P
-P
It .'a
i .4' C2 t7 A. 6. )i0 -
4*. t --
Si
i Su I .
% (Us..-?j.
'"" " "1"'" ' ' . " ". - v'.,-.," '
t44
4~ ~vt as~c;'.ro4e
1 -7-11T 4L.P -\-,r
rqE4 0*ev44.% kov
?cci,
CD
04b
(4) t ) '"' 44.(.~ a
C
®0 t -e s
Al 441(c
M4 t;Q
&V'~D 'Z~'77.
A w-.o4,v Q
?r2 ., T ",I
0|
0!
m0
z?
!N 0.
U -0.
?5 *
.LS rit
S t c o v ~ r t4 -c nt
FA1Z
4v A,-'
21''
.A % Stcv*,w j?"lot lo-i
Addam9w. f~(77
-:3 ~ ~ ~ ~ c *Ftt4q I'fJ t1..-
Tul4og-
eI -c " 4) -g L ; S st1,0
V~~~~ 4%C%~*p
- ~. .~. - . ' s UrIV I N 7N .Nv X-x-N ~ --tU ll IFV' V' V? V ?v ~~
* C D
- -- - - - - -0
+ .0
(oil
vp r- Wb4 1t .MeS Sd'&-A
- Sivj(. ~rOrr
S4,C3
tn
-ti
oft a oi
?*e m
ts
r -
Iv - t
A44~4
?5p
i6m(
e$4LIYaC. %ssC.7
e T"s
leapwhi Na -%-
£ S~~y~loe w.r
IDA M.?Hu 4r
* ~ ~ Si3~.m9Cb
~ (e.. ~1Ts~r~4C7
4L Ltk4
'2 ~ 1;
4 S CeCL44~~v'
*%,- te3cv
116'r-
p. W
I26
flY.-A
LUJ
IL
. <
-7
z r-
00 0o z
0
263
LU l
> Z L.!
w _j_
LUL
Z
C)
?6
RxL lp 17 i r ~ -w - ~b' - Wu . f~-~ C tWX' v~ XW lW JXVm U1Wir U 2 W2XWlW1W7 V 'V FI -Yi
F-.
U)
oLUJ z0) 0
0 C/0
0 ~LU0j z r
0 >-< <QL LU> wL LLU _jL .L J
CLw zIIII ~ i >/- >U - L
~LJH
z
_jLLU LLJ
< a
z
165S
LUJ
LUL
LU 0
H ~?F6
/ R -~c
/-7.
CP C
I ID
~ ~O u ~ (\J 268
'0
: 0-
C:) Cy' C
C/I
CD UD
cca
00
ou-Jn
00
269
w 99316
cr 91 8
-1900 9 .
OUTER W4ARKE'E 0
W .IOOLE
ARE
II If~15 SEC. 15 SEC. 15 SEC. 15 SEC. 15 SEC.2ONE ZO-N ZONE ZONE ZONE
47
95
I, c
27n)
r-r
> z 0
z )U
HL LJ< U
rz < -j u -
LlLU L-I0 > L L J2 1r:< zU
X ,.
I ?7 -k
5~ Z-,C..v -RA-
ouC A~ r:AN
C e. f t c- -
,, -P
C> - S d 4' Va ,' ',,' ,,''.,',,'', ,'. :", .- ,, .'., " "& ,,.''."". " ' : " " " "" : -".'.'." .""."". "" " C " ,' " -''-''-,-,'-.''.-.
HOUG.OAD NERSURENEN.. (U) DOUGLRkS RItCERFT CO LOWS EACHCR N R 31FEB40 ET AL. JUN 9? AFUL-TR-?-3043-YOL-1
W~CLRSSZFXEO F3361S-W-C-36N FI 5/9 A"E4 h h7EE Nh,~hEEEhh
1110 flu M
t 1111126
'Mill %%~ %.--~ % %-% .*~~ %*
zCl) Z
U) Lij L C/)WL.j 0 -LJ LJ )
zCCm 0 Cffi
0l 0U-JWZOD
LUD<J w-Z 0 ~< -ij< Lii 0
0 ED LLw D
274
U) 30
U~uJ
0 z
SzuJ
cr 0
-7-o
0 CD C j 0 C
i33HHOO iN30CU3d NVN
275
00
-LJ
N
CD. 0
Ln0 0)
(03SVY) AON3I
?76J
VISUAL RARE EVENTEVOKED POTENTIALS
FREQUENT RARE
'II
BASELINE
200360MSEC
I ILOW
303400 MSEC
IK
MEDIUM
219415MSEC
I
r
HIGH\
263
277
LU uiJ wicr. ) cc
co UJC) -j ) 0U)< <
LUJ Lii< <<Co < ~
> D >< z LJ<~
H-) 0 > -- j 0 <
LU <o> F __>--- a- -i
IrmCf) OUDC < L< 5
LU L-JW \ W ) wm f)<
oCm Lii
0 ~ N
C)Lucc
79j
zz
0
LU 0-
0 rn [r<<a-
UF- H-C
cc~c Zl LU
< U) CW
T ~LU0I (-IU) LU -
F-LUi U rnL<LU nn < L
wL Z (f Z z>LUi
4 0 m 0Cf3zl <L LU
_jU a- >f >-LUa- uID0 -J -
DLUi LUJ 0rn Cr LUi 3 02C
279
-
zLUJ
o crm a
co zL
z i D mD < <LL
LLJ L>
H LU0 >5
Cl) Cc
cc:
280>
D (D
:D <U
C) 00 z C
.j 0 0
0
2391
U)
(n ~> z ZL LU)F- UJ-
F-z
< 0z-za 0 *00
2P 2
w
1<J zo) 0
m~.J0 Z 0
Q-LI 0L r
0 u Z 0~
> zLU Z
LR O - LLUU a
0<0-J
0
w)Xz
0 z28F3 C
-~L W~ LL <*v:*J- -.. W
Il
UJ LwmW,~> I2 <
(m) < Z~H -
cr~~L LU L
0i 0i 0 0
L.LA
A-7 -,
1 -
- - -
;z-
-42-
0A
~ -- ~c
/ - C2
CL/
oC) *-'
285
-70
0
- CC
j 'm
- U
coa.
0z-
cz
cn Q
I j j
- ~ ~ ~ ~ ~ ~ ~ : C' N - 0 0 C - 'o -, - , N ~I Al Y idNt N V - -
286
C-1 9
0~
/ 0
t-0
C) 0 CC
287c
C-
953
Li 93,6
S90 4 -
-1 900 /
OUTER IARKER*- - . .
-- i MIODLE
V. AMARKER
-I -R U N A r15 SEC. I5 SEC. 15 SEC. 15 SEC. 15 SEC.ZONE ZONE ZONE ZONE ZONE
105
uM
rO0 / w AIw. LR Cft(
II
IAA
MU
95 IA ./
, , , ,.LaL
288 li,isbrook, Jq70,. ,'€
) = :. .: = . .. 71
LU
I LU
H-
zL< LU0
<<L.. JD< W L
co-r
0)I 0 > U)> L
*4, -D -.-
AM t-T -' .-.
Oi2
AI9L0TJO OE N TIM
CLOSINGOURAT ION
DefinitI.of of Eye Blink Pararreters
SUBJECTIVE RAF-IN.GS
Tj~ 3&0v
cMr
rO CO OPIor fLYI &I
I A/
101
70
WT +
SoL
C oPtLor
iu r e 2c. sJ~k cM< sc2S M on Seam e n
>kc I 1: 1 9H
291
WWWWV'W0
zCO)
mlC)2i2 -O L UuJ CI
z> zm LU<W wl HO I0 0CLC
I-< 0 H >) H
o- <O- 0<o
C) < U- OZi< L -
L.Lj Lj Z 0 w292
C/)
w
CW)LLI
LLJ L0
000
103HU00 iN3083d NV31N'
0<
zLI
N
CL.
(03SIM) AON3LV-1
?n4~
-- w -v - -rw* - yr. V'.. r r.. WV r
VISUAL RARE EVENTEVOKED POTENTIALS
FREQUENT RARE
BASELINE
200;60 MSEC
LOW
303
400 MSEC
MEDIUM
219-, 415MSEC
I I/
HIGH
263485 MSEC
- 150 850 - 150 850
-'E 86 9
295
D -< z< C
0 < -
LU <0 - ' -u~
LL0 wiLJ L 0 Li Uoo
LU
296
Ni
0
Z -
cno z<F
00
< XO '0
< w c
wU z Cz>) Z
I- 1 -
Z)z <Z U0- a: ? - C/) < i
o/) CJ0 -JZ LW w z
a-rn LiiLL*@@@ .<
C, ?) ) j-
m 0C
P.
zLUj
0
0
oLL-
LL zI
LiiL
LUI
LUI
0w --)0>
LUJ
z
20
Z u
00
00
4-U
W7 -r Vv N. *
z 00 Z
z -j0
ww
QZ 0-
LL00 L
0LLJL
LUD CL)
LLI (n cl c
cc 0 C)
Q~ Z f 00
zL z0
U))'
0 C/a. UC,)L 0m
a. w :
x wL
LLj
00
LL m 4zL
Li..J
301
u'm
LuW 0 LLUO
*~~LLJ 0>~O%%mozZ0Z O
& zm w(-In QO0 ZZ z w 0
z U. LL0 x a
0 C/0
2 w w tl
Cfll
0 zr 50n0 0 WO<
a. wU. 0 LL C
cz, L X 0-J L CCI
(E 02zZ Lu . >L
Urn n< wOC) W < -
~0 00UL
a.L JIm L) o
_L Z- 0.1UJ w m0 )
303
.
mE
Cm)
a
zzw uJ
z LU
04
C0,3UL 0 W=
LI.. L
304
LLJ M
m0o J
0
w C/)
Wn w o (0
0 !.U
Z 0 LU uL 0C (w CL
M 0l) 0 0 l i
oL n -r z z
cpm 0w w 0w I~ w Cf L-
Oct0 wcnwwc
IL 2
305
0> -u-Wc
onu - w
m U)~
0
mlm
w
116
JW
~LL.
x nLIJJ
cj,
a- 0003 a 0
crZi
w~ cnnLLZM 1-1 z
m Z
R m cn Z .0LL
w
in0
307
00 w
(n (i)
/LLJ ,-WWU
=0j 01 14 w
cc lm
Iz z-J
Cm Z 0 LL 0Pwuo a:o C wic2.j M U) a m
w w L) O0
a. 0 0nEzc WC26OM
w C/) '
(U)M 0
LU LLJ C
0 z 0
308
L nl~bkkkll,
C/) 'IJ
Cl) f.
U~tdJ
w 0
z -LL
000 0
CZ 0 LLwj LL .
I-. I
O w-Z 0
0 C. 02L 0
I- 0
~Nm
> -<w Z
309
m Jw
U)zw wm
(n1
LUI wOumu 00
Cl) I' i() 0
z w M
LC cc <z
on LL
cc 0)Z> -LL zff E -
I- OLL.
0m
w tW0LL
U.J UJU so
31 ULL
0~ o z *a
W w
IJJ C) 0 WU 0
w C,, -j
z 0 ->
w ww
4n
1
020PLJ
x xC,
LLJ < a
~Cn -cn 0,-
-, CL m-LL
311
V1VTVW._V.-V_-.VJF-YO.,. 71 " -7
Subjective Group Meeting - 2/25/87
Synopsis: NASA TLX and SWAT were the measurement techniquesrecommended for use in the Part Task testing. There areenough references and data collections using these measuresto demonstrate their validity and reliability. The ModifiedCooper-Harper, a unidimensional scale, was discussed but notrecommended as it gives a "number" but does not allow you towork backward to how the number was developed and thus,offers no diagnosticity.
Meeting Summary
Danny Gopher said the goal of the meeting was to come upwith recommendations for workload assessment methodology incertification of aircraft. We must review the problem ofcertification as well as the process and identify key issuesin certification where our methodology can help.
aDel Fadden explained the Boeing certification process andshowed the Pilot Subjective Evaluation (PSE) form that isused by Boeing during the certification program. In this
* program the pilot is asked to compare the new aircraft witha known reference aircraft. If a pilot says a task is moredifficult, then he must discuss why it is more difficult.
Del pointed out that before certification, in simulation,most of the cockpit design work has been part task. A lotof design is done even before the simulator is available.An airplane evolves - other planes are used for reference.
Jean-Jacques Speyer said Airbus takes a unidimensional view.a.
v' They ask for lots of ratings, actually take ratingscontinuously on line. He proposed that all ratings togethergive a microscopic evaluation as to what is going on. TheAir Worthiness observer also gives a subjective rating. Ifthe observer sees that the pilot has not made a rating atsome particular time, he can ask the pilot to give ratingsat that point. (Instrument set up - observer gives crewmember a light and pilot is invited to give a rating at thattimee. Ratings are not pass-fail, ratings are g:ven cn anabsolute scale.
Group discuszsion followed on the two techniques. Insummazion, Boeings' method is a comparative measure, Air ' suses an absolute technique. Boeing takes pilot ratng3 postflight, Airbus takes ratings in-flight and at tne en,: :neflight pilots are given a long questionaire - i 0 ]'.,it, s
9~j Gopher said we must define the problem. Why are f -ratings important9 What does pilot feel about the t. K hemust perform? Is he comfortable with it? All -art'ucp.nt5
a, 313
agre " a u' p:lot subjective opinion in one form ora. ,.er ]jT erl:et it would be useful if we could
s'ar. ar:ize an<= use the same scale in all nations so we can
learn from each other
In the discusslon that followed on standardization thefollowing points were made:
1. We need experienced pilots to make judgements.2. We need to have some comparative point of reference3. Ratings should be made under some specified optimum
conditions.4. Raters must be trained to rate.5. Everyone must agree on the dimensions and terms of
workload so results will be communicable.
The point was made that if the same method was used indesign and certification, then there would be no surprises,although the method would be used in certification in a morelimited sense. Del Fadden pointed out that this is notalways possible, for example, for the design of much of the7J7 there is no reference airplane.
Danny Gophers opinion on selection of rating scales is touse NASA TLX and SWAT. He said they are the most viable andbest documented. He said the Modified Cooper-Harper is"dead", not being used and is based on old research. Hewould like to see both NASA ,TLX and SWAT used, at least tostart.
Modified Cooper Harper
Sandy Hart looked at the Modified Cooper-Harper when snebegan to develop a scale. In this scale the raters movethrough a series of decisions, these decisions determinewhether the rater ends up at the top, middle, or bottom ofthe scale. You can't take the final number and workbackwaros to determine how the number was developed. >AHAwanted a diagnostic scale that would allow them to workbackwarls .
NAS3A TLX
Subjects were asked to give their Jefiniti-n ofbefore tney started the tests to reduce intersubjeotvar~aclity ' J . Dur:ng the tests s ub ec-ts were a d w the.felt caused the primjrv __urse of -ornioad, then that ' tasi."was ,:-zen more wPL'gn Ih. ir-wl f... w~rrxz well for s m . t sk-but wel4htu don't have a- mu sh efect .n ccmp.;ex -a.was felt that wel-znt.z snoui- be usei for bot'- scr e -mnocompiex: .wev,: r
It was.r nointed )ut. that tht FAA 4s interersted ii h, '
workload functions listed in the FARs so perhap; a s.>'..
34
should be developed to evaluate these 6 functions. Thesefunctions are; flight path control, collision avoidance,navigation, communication, operation and monitoring ofair'raft engines and systems, and command decisions. It wassuggested that the EAA could be asked to weight thesefunctions so the manufacturers would have a standard to workagainst.
SWAT
Gary Reid said he and his colleagues did not set out todevelop a subjective scale. They were asked as workloadexperts how to measure workload.
They started out with the Simpson-Sheridan scale. Thedisadvantage of this scale is that it takes a lot oftraining to properly use its decision tree rating scale.Next they then took a three dimensional scale, regressionsapproach, looking for the smallest set of dimensions thatwould tell them the most. They were driven by practicalconsiderations such as cost. For example, is increasedsensitivity worth the cost? This scale fits their needs.They can gather ratings with some diagnosticity. In theirratings, they feel that if they hadn't had observers, theycouldn't tell why workload is higher in some instances.
SWAT covers 4 of the 6 functions listed in the FARs.Performance and physical demand are not covered. Gary saidthat if they need performance they measure it. Measuringmental workload is important. Physical is not as importantas it once was.
Discussion followed on the importance of getting down to a.one number" assessment, is the extra effort to get therereally necessary? It was felt by some "one number" providesstructure and information and is a valuable adjunct to whatis going on.
Danny Gopher said that the charge of the subjective groupwas to evaluate measures for use in the simulation study.It has been concluded that it is important to use subjectivemeasures. He reiterated that all manufacturers usesubjective measures in one form or another. This groupstrongly recommends the use of subjective measures. it isreasonable and advisable to find a systematic way to eiicitthese opinions. He feels that standardization is oneapproacn. How to get the information from the pilots is thequestion?
Both NASA TLX and SWAT give global assessmentz of tasks as awhole. Th- 3 are enough references and data collectionsusing them to prxvide valid and reliable results. TheAirbus on-line measurement model is different because onemust design situations so a single number will be meaningful
315
CZ - *Z r- d 7-
and one must have many such numbers. It is a costly anddemanding process and is not well documented. Gopher wouldlike Airbus to try NASA TLX and SWAT along with their methodto see if the methods agree.
Discussion followed on the importance of using both NASA TLXand SWAT. The following points were made;
1. The way in which you communicate with the pilot isdifferent.
-2. here are differences in the weighting systems of the twoscales
3. It is important to generate more than just a singlemeasure of workload.
The concept was discussed that assessment is a package, nota scale. Raters must be qualified and pilots must betrained. NASA TLX and SWAT are both used by in flight, asoften as possible. Ratings are taken as soon as possible.Both Sandy Hart and Gary Reicd stated that if data iscollected in flight, they are more comfortable with it.Results of tests, however, do not back up feelings that in-flight measurement is better. Gopher felt that if we canget data post-flight with no change in results, that wouldbe an important finding.
Much discusscn followed on what number is good enough.Should some subjective measures be taken in-flight ana somepost-flight? If the measures are taken in-flight, are theyintrusive? It was the feeling of the group that measuresdon't always have to be intrusive, for example, afterlanding you can gather ratings on outer, middle, innermarker and landing.
Danny Gopher said he would summarize the subjective groupmeeting results in the general meeting for 15 minutes, thenallow Sandy Hart and Gary Reid to give summaries since theydon't agree with Danny on many points. Sandy and Gary saictney gave their presentations yesterday. Sandy stated thatGophers opinisn does not accurately reflect that of thegroup,
316
V"
MINUTES
Performance Group Meeting - 2/25/87
Summary -
The performance group discussed the cost of measuring workload forbehavioral, physiological, and subjective measures. Much support wasgiven for including a Sternberg Task in the simulation test battery.
Several valid and reliable measures and implementation requirementswere agreed upon. The measures suggested were: Sternberg task,critical tracking, choice reaction time, mental arithmetic, andembedded communication.
The Cost of Measuring Workload -
Performance measures (procedural errors): Measure impairments toperformance.
Physiological measures: Measure psychosomatic effects of stress andoccupational diseases.
Subjective measures: Measure conscious experience, estimate theability to cope with goals and achieve criteria, and are sensitive toone general "work of intentions". This influences performance on thevery general level. The costs or possible effects of subjectivemeasures are that misjudgements may affect selection of goals andcriteria, may affect motivation, may affect risk taking behavior.
Types of Certification -
The group established that certification varies depending on whatorganization is doing the certification.
British: Look at the relationship between subjective, physiological,and behavioral. A study by Jean Jacque Speyer was mentioned testingthe possibility of using control reversals during approach as ameasure.
Boeing: Weather is a random event. A simulator can impose whateverweather desired. Certification uses a performance margin. Failures,weather, etcetra are added. Frequency of occurrance can bemanipulated. Performance margin can be large or small. Usestraditional timeline analysis.
317
4|
Implementation Requirements for User Acceptance -
The group developed the following requirements to insure useracceptance:P!1 - The measure must be non intrusive.
4
2 - The measure must conform to all safety standards.
3 - The tasks must be within the realm of "normal" methods.
4 - The measure must not lower crew self image.
5 - The measure must be non career threatening.
Valid and Reliable Measures -
The group suggested and agreed upon the value of the followingmeasures:
1 - Sternberg (auditory/visual) can be highly intrusive. Must makeefforts to move toward "real" or "normal" tasks.
2 - Critical tracking (psychomotor) can be too complicated. Thismeasure was considered of boarderline acceptability but the groupvoted to keep it as it can be useful.
3 - Choice reaction time (visual).
4- Mental arithmetic (auditory/visual) allows for muchflexability. Can be embedded and made "realistic" to piloting tasks.
5 - Embedded communications (auditory) "normal" secondary tasks.
Problems and Considerations -
The group raised the following problems and considerations:
1 - There may be a problem with comparing a new craft's performancewith a "reliable and safe" old craft's performance. The comparisonwill not always be reliable. New crafts are often too different fromold crafts. This can yield misleading results.
2 - Must consider that the design stage implements engineering te.5tpilots, the testing stage implements line pilots, and thecertification stage implements engineering test pilots.
3 - Theory must guide implementation.
4 - Must consider the training on, and difficulty of, tasks (datavs. resource limit).
5 - Must consider f'atigue from introducing the tasks in terms ofadding energy exT iture to the crew especially at work underload.
. . .. . . .
313 m
Sternberg Task as a Measure -
The group cited several workload measurement studies looking athandling, displays, and crew coordination aspects of flight. All
studies used a Sternberg task.
1 - 1982, this study evaluated 2 HUD display formats. The measureused was a visual Sternberg task. Pilots flew ILS. While theexperiment was in progress, the experimentors recorded several verbalresponses such as whistling and short comments about the task.
2 - This study varied frequency of input during a terrain followingflight director profile. An adaptive visual and auditory Sternbergtask was used. A top level of error was preset. The study found thatthe visual Sternberg task discriminated between different levels ofworkload while the auditory did not.
3 - This study varied visual disorientation with a Malcom horizon.The measure used was a Sternberg letter presentation task. Resultswere mixed.
4 - Dunn, 1985, Helicopter Proceedings Conference. This studytested kenisthetic displays. Pilots "flew by the feel" of the
instruments.
5 - This study tested a cross coupled instability tracking system.The measure used was a Sternberg variable pitch task where theinstability level was varied.
After reviewing these studies, the group agreed the Sternberg taskshould be among the measures tested during simulation.
Questions Raised bIy the Group -
1 - How to address user popularity? Aspects of a potential measuremay appeal to large air line manufacturers, but not to small generalaviation manufacturers or vice versa.
2 - How to assess dimensionality? What is being measured?
3 - How to address the demonstration of worst case? What are theprobabilities?
4 - How to assess practicality? We must consider cost, degree ofsimulation, and flight testing.
5 - How to address methods? We must determine where, when and howthey are relavent. We must decide whether or not to test the limits.
6 - How to tailor the measure to each situation in which it isused?
7 - How to do a comparison of techniques?
3 Q
3214.
.. , . ,b,.- -.- - '> .-- 3.> % , , - -', . ... -b- . -- % .,. '., .,.,. bi.% ,,-J.-I-/- -,. - -,-, .
* b_C ALA44or'
4~c C. 5Y14~ ry%, (\to IC
04.rO YCiL
r-~ -uw~ ~ w ~ r -- - 2 wj'U "UT -r Nw. 1w IV, W.- W - r~ W W N In -- V'l -l" W
I e
C~e , LZEU C 006
3 ?3
11L ~~ ~ ~ ~ W-L Wau 61 1VT-vvv
:%
USAF/FAA REVIEW OF WORKLOAD MEASUREMENT METHODS
SUMMARY REPORT:
Physiological Measures Group
'a
Prepared by:
John A. Stern, Ph.D.Washington University Behavior Research Laboratory
at Malcolm Bliss Mental Health Center1420 Grattan Street
St. Louis, Missouri 63104
325
- uww~~~wu~~ww ~q-Vw-WW v-WIWVVV-WWW-IWITM v1!' y W'3 1 -m .Y P 'F -. .--
2
Recommendation for immediate implementation
1.0 Constraints
1.1 Measure must be usable in aircraft cockpit and simulator.
1.2 Measurement procedure must not impose secondary task performancerequirements on subject.
2.0 Recommended measures
2.1 Hleart rate and heart rate variability (HR).
2.2 EOG: eye blink and eye movements.
2.3 Perhaps aural canal temperature.
3.0 Rationale for choice of measures
3.1 IHR and IR variability
Reports by Mulder and Moray dealing with HR variability as measure ofmental effort and comments by Alan Roscoe concerning HR.
3.1.1 HR variability as defined by Mulder
- Abstracts interbeat interval (IBI) for fixed number of IBI's(IBI accurate to + 5 msec). Moray uses 256 to compute.
- Conduct Lagrange interpolation for every 500 msec.
- Autocorrelate these equidistant values with maximum lag of 10% ofcomputed values.
- Fourier transform autocorrelation functions.
- Smooth raw spectral densities with Hamming window.
- Derive power of five spectial points (.06, .08, .10, .12, .14 lz).
- Derive natural log of power of the five points.
- Activity in .06-.14 Hz is sensitive to "mental effout" manipuilation.Greater energy reflects relaxation, lower energy mental effort.
326 "
-* -
3
3.1.2 Comments about procedure from principal advocate(s)
- Morey reported that a variant of this procedure is currentlyin use in his laboratory and works to index mental work loadin a variety of situations. HIe also reported that althoughhe used the Mulder algorithm, he could not replicate Mulderresults in earlier investigations. The cause of this in-ability to replicate is a mystery. Other researchers haveexperienced and reported the negative results.
- Alan Roscoe reviewed the utility of HR as a measure of"arousal," "attention," or "alertness." This measure shouldbe useful in long duration flights. In the present context(30 minute flights), it may not hold much promise, althoughone would expect that the two failure conditions may well beassociated with immediate increases in HR. What will bedifficult to determine is whether such Increases ate asso-ciated with the psychological threat produced by the equip-ment failure or an increase in motor activity associated withdealing with the problem.
In any case, HR variability cannot be measured without ac-quiring HR information, therefore, HR remains as a measure inour proposed "battery."
3.1.3 Potential problem with respect to part-task simulation run asdescribed.
- Some of the segments are only 1 or 1.5 min in duration.This may be too short a time period to obtain reliablespectral information.
3.1.4 Solutions
- Don't analyze data for segments less than two minutes (or250 seconds) with Mulder procedure.
- Combine data from equal work load segments?
- Sample data for longer periods, if at all possible.
- Await Moray's "on tine" filter technique to analyze data.
Try / technique for Vagal tone - Porges Black box.
3.1.5 The recording of heart rate or period poses no majorproblem. Such measures have been successfully recordedboth in the simulator and under flight conditions.
Z! _. _N' -497
3.2 EOG - Eye blink and eye movements
- Recommendations based on tesults of AAMR[./HEG (Wdiison), I2AFSAM/VNE
(Miller), and WUBRL/MBMH(C (Stern). Measie has been ; d inlaboratory and simulator, and is currently being used tot in-flight
monitoring (AAMRL).
3.2.1 Blink measures utilized
- Blink rate obtained results: differences in rate betweenpilot in command of aircraft and second in command.
- Increase in blink rate as a function of time-on-task have
been demonstrated in a variety of conditions.
- Inhibition of blinking is related to visual task
difficulty.
- Blink amplitude: not very discriminating for effort athand. USAFSAM data on smaller amplitude blinks being
associated with poorer performance (probably function ofpartial lid closure as sleep deprived subjects performlong duration (4 1/2 hours) in simulator (Morris data).
- Blink closing and/or closure duration (50% window):Closing duration is defined as time from blink initiationto peak closure. Closure duration is defined as timebetween the blink entering a window defined by half
amplitude of blink and exiting that window. Closureduration has been shown to be sensitive to visual task
demands (work load), with shorter closure duration blinksassociated with more demanding tasks (demonstrated inlaboratory and simulation settings).
3.2.2 Comments about procedures from principal advocates (Wilson,
Stern)
- Data can be collected in simulator and in flight. Appearsto be sensitive to the type of work load manipulationsproposed in McDonnell-Douglas/Boeing joint effort.
3.2.3 Potential problems
Dutation over which data is sampled foi variuuv '.,,ik loadlevels is at the lower end of acceptability. 01e Minuteof data is too shott to produce reliable rate data; mliybe -icceptable for other blink measures.
I
W."M- W. 1-77 --
rW N- -Y N
Since eye po : it in in!cr t i,)i 'u i 11 be advocate(i as a
promising mea-u. tot uture ifr pl mntat ion, I (not the
working group) -.0id l ie to iecomrend the rucoi ding of
horizontal eye li:emen t i Ii.ing E(7 procedures. if,
as suggested by May,, alterations in dwell time on
particular instLum('nts may index visual information
abstraction ineffici i 'ie),- then fixatin pause durationsas evaluated "i th El(; ,.av p:ovide- a reasonable (and iuex-pensive) way of obta i niji i, h (or -nimilat ) information.
3.3 Intra-aural temperature
- Recommendation based on c ,.-.5(u. tei Hanco, k. He has been usingthis measure at NASA-Ames in their si-ullators.
- Possible measures advocated ,te: a-oir,,e temperature and tepera-ture change measured from orie ear; and of differential temperaturebetween the ears.
3.3.1 Comments from pt inc ipal ido We
- The measure ret Io t r l-iil (perhaps hvpothalamic ) ftem-
perature. To the extent that meta bolic "need" producesenhanced blood flow in the brain, this technique allowsfor the evaluation of such changes in metabolic act ivitv.
- It is in format i on t hat i - i ex pen: i ve ly acqn i r eu (, and
Hancock will make his device available to the project.
3.3.2 Comments from panel participants
- We did lot have available elevant esource material toevaluate the claims, by ttiktvock. lie will puovide io\)glaswith reprints relevant to tiee use of this ptocedure in the
evaluation of work load.
- There were comrment, imut t(cbinological problems such aspositioning of sercu :,o that it picked up tymponic
" membrane tempe, a ltiir t Ireh(l than skin tenlperat te.
- Technical probl, , iT i, c.t,,l '.'ith the headset worn by thep 'lot.
3.3.3 Potential pLlt i ....
•n - Some ate I i i o
,- rh paI , . ' t ' l , rut tiis
flIi ghIt ; i ui I '
.; . . ... .. .. . . . ... ... -.:. ..-.. .;. .. . .* .......... . -:_
4.0 Recommendations for the NEAR juture: Measures recommended lot
consideration
4.1 Event related brain potentials (ERFs)
4.2 Eye: point-of-regard information
4.3 Voice analysis
Rationale for choice of measures
4.1.1 Event related potentials (Principal proponents - Kramer,
Stern)
- The University of Illinois Cognitive Psychophysiologylaboratory, in conjunction with aviation psychology(Wickens), has extensive experience with the use of this
technology to evaluate aspects of work load as well as
cognition. The University of Illinois investigators havefocused on the use of ERPs to both primary (embedded) andsecondary task stimuli, while the Washington Univeisity
effort has begun to investigate the use of "irrelevantprobe stimuli." All three appear to be promising, withrespect to simulator applications and the embedded and
probe stimuli procedures should be usable in flight
environments.
- The use of embedded signals (embedded signals aie part of
the primary task) for the triggering of ERPs wasrecommended as an alternative to using secondary tasks.Other suggestions involved the use of "irrelevant"
probe stimuli to elicit ERPs and the use of subject pro-
duced "responses," such as saccadic eye movements, to
trigger the averaging process.
- A fall-out from recording ERPs is the EEG. Spectral
analysis of this data was suggested as another window to
capture levels of arousal or alertness (the other windowbeing HR). Since alertness should not be a problem in the
proposed simulation, this idea was not explored furthet.
- The technology for i cro, ding EEG (including ERPs ) insimulato, ; and in light is available and bein iset(ERPs -- Jniv t!- i ty of Illinois, Kramer; AAMRI,/ 1 (;, Wi 10onand EEG in f I ight recot ding, B. Sterman, cent! i tuge
USAFSAM I!,,w is ). ERP hive been demonstrated to he ;ens itive to task dveiPand (. voi k load effects.
4.1.2 P1)ble2,.
Signal/ni, I tI., ' :K 1 ot in tete ;t i f IfIintlvd eg radd ("I I t Ia In I t )F I i t I nd !simlllt in oi n(1d I 11s.q
%g~ _' .% A,1d
7
4.1.3 Solutions
- Solutions, such as special electrodes, amplifying thesignal at the source; and development of filtering proce-dures for signal "sensitizing," are in reasonably advancedstages of development.
4.2 Eye point-of-regard - Principal proponent - Moray.
- There is considerable literature demonstrating the utility of thistechnique in flight simulation (this material is available in thebibliography provided). Moray suggested, as an example, applyingthis technique to work load assessment, that dwell time oninstruments is generally in the nature of 500 msec. If the eyereturns to the same instrument frequently (i.e., average dwelltime is decreased), it might suggest that the person's ability toretain information abstracted in short term memory is impaired.This might be considered an indicant of high work load.
With respect to using the technique in flight, Moray expressedoptimism about the possibility of developing the necessary in-strumentation to record eye position information under such con-ditions. However, such instrumentation is not currentlyavailable.
4.3 Voice analysis (Principal proponent - Walrath)
- Little comment, other than that it might be a useful procedurefor the future. There was some discussion about its utility toevaluate "stress," but few comments relevant to how it might beused to evaluate aspects of work load. Several articlesdescribing laboratory validation were entered in the referencedata base. Current work, sponsored by NASA/Langley will bemonitored. Since pilots engage in voice communication, usingvoice output would be the least intrusive measure of all thosesampled by us.
5.0 What physiological measures contribute to the assessment of work load.
.5.1 They provide some measures that cannot otherwise be obtained(unless one can impose secondary tasks), measures that .ill bemost useful in the evaluation of conditions involving underloadof the operator. The states of concern mentioned include"arousal," "alertness," "attention," "daydreaming," "drowsiness,"and "microsleep," to mention but a few.
5.2 They provide information about moment-to-moment changes in theoperator, rather than average values (averaged over time). Thus,point! (or narrow windows) of "momentary" overload can beidentitied.
331
5.3 They are unobtrusive and objective.
5.4 They should be used to complement other measures. As amplydemonstrated during the general presentations and discussions,subjective, performance, and physiological measures are far fromperfectly correlated with each other. They, therefore, providecomplementary (not competitive) bits of information to thoseconcerned with the evaluation of mental work load.
5.5 They have the potential for allowing us to discriminate between"controlled" and "automatic" information processing by theoperator.
6.0 Sensitivity
The issue of sensitivity of physiological measures to graded changes in(or levels of) workload was not discussed. Our discussion focusedprincipally on looking for differences between resting and "load"conditions. The literature, at best, has attempted to discriminatebetween three levels of work load (low, medium, and high).
A number of reasons (rationalizations) can be used to account for therelative lack of effort in this important area. The first is ourdifficulty in establishing a metric, other than an ordinal one, fordefining work load levels. The second is the lack of a substantial database relating physiological measures to work load that unequivocallydemonstrates the utility of such a measure for work load assessment. Thefield is still youngl
A third is the hope that physiological measures will provide the metricfor defining work load levels with greater precision than currentlypossible. We should point out that our definition of "work load" may beradically different from that of the Human Factors Engineer. It is oursuspicion that their preferred definition deals principally with the loadimposed on an operator by a given configuration of hardware. Ourpreferred definition is in terms of the impact of the imposed load on anoperator. Thus, the point in the information processing chain that issampled by those who attempt to define work load on the basis of what isimposed on the operator is different from those who focus on the impactof the imposed load on aspects of operator petfotmance. In out caso, th-performance measure is the output from one or mole physiological systems.
7.0 Warning comments
7.1 At the current stage of deveolpmont, those enli!tnd to c]] ctphysiological data must be trained: to discriminate signal (Tomnoise; in the proper application of electiodes; and inappropriate signal condiiioning procedures (amplifihatinn,
filtering, e f .
L.. . . ... ... . ... - .. -. -.'. - -. .',. ;'.. -.V .,. % : "V ": "" " ' " - " W. -:
5. 9
7.2 Environments in which bio-electric data is to be collected haveto be "sanitized," i.e., sources of electrical artifact have to beshielded, equipment properly grounded, etc.. This may require theservices of a biomedical engineer.
7.3 Data reduction is, at best, a semi-automatic process.
7.4 Data collection and reduction is a relatively costly procedure,when compared to paper and pencil tests or recording the outputsfrom "manipulanda" or in-flight equipment.
333
l 4".. . . . . - . - - . - - . - -
I-
I
8 X P 1 I I NE T A L D E SI G I
SESSION ONE: SWAT in-flight UASA/TLZ poet-tligkt
ORDERNUMBER: 1 2 3 4
SFO - SCK (LO) SMF - SFO (HI) SMF - SFO (LO) SFO - SCK (HI)SMy - SFO (HI) SFO - SCK (LO) SFO - SCK (HI) SMF - SFO (LO)
SESSION TWO:
SMr - SFO (HI) SFO - SCK (LO) SFO - SCK (HI) SMF - SFO (LO)
SFO - SCK (LO) SMF - SFO (HI) SMF - SFO (LO) SFO - SC9 (HI)
So 1,9,17 2,10,18 3,11 4,12
eoeeeeeeeeeeeeoeeeeeueeeu**eoe~eoeeeoeeeeeoeeeaeoeeee ee......eo
SZZSION ONE: 1A3A/TLX in-flight SWAT post-fligbt
ORDERNUMBER: 1 2 3
SFO - SCI (LO) SMF - SFO (HI) SM? - SFO (LO) SFO - SCK (HI)SMF - SFO (Hi) SFO - SCK (LO) SFO - SCK (HI) SMF - SFO (LO)
J ESSSON TWO:
SM' - SFO (HI) SFO - SCI (LO) SFO - SCI (HI) SMF - SFO (LO)SFO - SCK (LO) SMF - SFO (HI) SMF - SFO (LO) SFO - SCK (HI)
So 5,13 6,14 7,15,19 8,16,20
~335
s ~ ~~eggee*g~g ..... geee.g..-.....*-.. .... .g.......g-..-. g -... .'I.Q.,
MEA URE EE VIIDV-4
9g 1 3 0 2 1N 1 T I DO 0 3
FLIGST P2A3 OP1N MN.SURKXM1T WINDOW CLOSE ML3SRE XIT VIDOW
0 Takeoff: PR .GT. 1 .50 FLLPS 5
* Climb: FLAPS UP 1:00 (One minute latbr)
0 Cruise 1: 10,000 ft 2:00 (Two mnutea later)
0 Cruise 2: 3 Minutes after 10,000 ft 2:30 (Two and a halfm inutes later)
* Descent: Throttles to ldle 5,500 ft
0 Approach: Looalizer actviation Outer Marker
' Touchdown: Middle marker 1:30 (One and a a.,manutes later)
33
I
..
J
'
ip
336
'S~E F: - _=A~ ~7G E E:=~ 'c~~*HEARS F/E, "NO. ,S ENGINE IS SURGING.
Ckt ;_5*NOTES #3 EPR, NI AND N: FLUCTUATING.~I*NOTES #Z EGT RISING.
__________ S)_CAPT TELLS F/O, "BRING 07 TO IDLE.O.- ) \ *CAPT TRIMS RUDDER AND STABILIZER. )CTES FIC,
q BRINGING #; THROTTLE TO IDLE.Afqn *HEARS F/E, "#Z IS STILL SURGING. APPEAR TO
. GETTING COMPRESSOR STALLS."c. CALLS, "ROGER, LET'S SHUT IT OCWN. NGI NEHUT WN CHECKLIST."
L
,0,1(i> )3(4APT Ab4QS POWER, TRIMS RUDDER AND STABILIZ
00:09:=O ENG:NE FAILURE/SHUTDOWN CHECKLIST.
1.4 ,C(%XA*HEARS F/E, "ESSENTIAL POWER OPERATING GENEFATCR....- 4 .-l'HERS F/E, "THRUST LEVER -- CLOSED."__...~ -....~-.... -a % *cAPT tbQZ.jNO. 3 THRUST LEVER. CAv ... . . -
'4 . ,a--(-\HERS F/E, "START LEVER -- CUTOFF."S, _ *CpAPT PLACES START LEVER TO CUTCF-.
-l ' . SAYS, "CARGO OUTFLCW VALVE -- CLOSED.'-< . '-.sc Li/E Sw'S, "GENERATCR SPEA'ER LISHT -- ON."
4 f q, (Ip9*F/5 SAYS$ "ELECTRIC L LOADS -- MCNITCR."
"" , ', , 3AY3, "ENG:lE SL-- Ai . : Z:; ---
4.L.)±*/E SAYS, "WING AND ENGINE ANTI-ICE SWITC:-E" --
SE.-RS F/O RESPOND, "CLOSE:."61'& E.PS 7/E, "ENGINE SHUTDOWN C:IEC, -IST CM-_'-E.
00: 10: 00 LEVEL OFF AT 11,000 FT AtD CROSS RESAS INERSE--:N.
S " 2.-, *' E-- T:.'JcE €,-* ':~ v'T T C"' 3
4. L (' q%,TEL_S F/E TO SET CRU:SE T-4P7ST 9:P ENG,'EINCP C..UISE.
L '( ldl*HEAAS F/E, "POWER SET.".7.CAPT TRIMS RUDDER AND STABIL!ZER FP CRLI'-.
a( %A(9d *HE:RS F/E, "ONE GENERATOR INOP CHECOLISTIS NOW COMPLETE."
( o ACKNOWLEDGES.
1511LE S AIRPLANE AT 11,000 FEET USING ADI,AL-:METER AND VERTICAL SPSE--d INDICATOR.
t.. P *LACZ3 FD . . ._- ALTITUDE HOLD SWI-:4 ON.n I.}- < *iC-ELERATES TO 288 KNOTS. (2 ENGINE CRUISE)
4.a • R"4.-,.LS, -LANDING LIGHTS OFFr .IT. 141"A I-ETS AIRSPEE: CURSOR TO -59 KNOTS.
'2 - 93EOs3RVES APPROACHING CESIRED AIRSPEED.11' S 9ETS THRUST TO MAINTAIN DESIPED AIPSPEED.
OBSERVES DISTANCES TO MODESTO ON DMEINDICATOR.
1-, 6DGaERVES 099 DES COURSE INTERCEPT TO MODES-,%/CRTAC.
1 S: 1 w2.3120I-ITIATE3 RIGHT TURN ON HSI AS CCUPSE DEV!ATtCNIND CATOR APPROACHES CT.NTER.
337
C-
~Y~TWWVWYWWVWTvWWWW iv w~w w Wr wl WRV fWMo v WV.~~J~ -777 - ' .- ' .-
?IC zo r V 8A C T 01. LXGRT PATE _ I I COMTRCLS
..
2. COLL.. hVO XD.112 2. D13PLAY3
3. NAYialTION - 3. VROC D~R34j4. CoNNUMICATION7 4 41 ~ 4. XXXzzXXX~XXXXX uXXIS.Of 4 x(OIzOaIjj 5. mou::ot
-7. AcTo _____
9(l) II COZENOL
9(2) MN DISPLAY 0
103) Is PIOCID. I
9(s) NN MOVIT.
9(SA)II COeNN. 1
338
4e.
T A 3K L O A D I G
* PIYSICAL / MENTAL 0
LOW HIGHSFO-SCK SMF-SFO SFO-SCK SMF-SFO
T/O 75 / 124 71 / 117 75 / 124 71 / 117
CLIMB 39 I 7t 38 / 70 40 / 82 39 / 95AUTOPILOT INOP
CRUISE1 13 / 46 10 / 37 34 / 128 31 / 121NO# 3 ENGINE COMPRESSOR STALL
CRUISE2 1 / 6 1 / 5 11 / 50 11 / 49B SYSTEM HYDRAULICS FAILURE
DESCENT 47 I 137 45 / 141 46 / 140 53 / 164
APPROACH 63 / 185 66 / 199 61 / 177 64 / 190
LAND 28 / 62 28 / 62 26 / 59 26 / 59
TOTALS 266 / 635 259 / 637 293 / 760 295 / 795
901 896 1053 1090
339
SI1 P RIt Z NE T A L V A R I A B L K S
FACTORS: 2 X 2 X 4 X 7 DESIGN
FACTOR ONE: WORKLOAD (2 levels) LOW and HIGH
FACTOR TVO: ROUTE (2 levels) SFO - SCK & SMF - SFO
FACTOR TiRtE: FOUR POSSIBLE ORDERS A, B, C, & D
VACTOR FOUR: MEASUREMENT EPOCH (TASK LOADINGS) (7 levels)
1) Takeoff
2) Climb
3) Cruise I
4) Cruise 2
5) Desoent
6) Approach
7) Landing
3.
I..
340
DATA AN ALYSES
S E9 S I T I V I T T A 5 ALY 3 E3
COITENI VALIDITY
2 X 7 ANOVA Workload by Measurement Epoch (TASK LOADINGS)
o Various Dependent Variables, one at a time
o MANOVA approach presupposes a battery of tests
Then a series of planned comparisons:
Considerations: o Maintain comparison-wise Type I error rateo Select a test wnioh is ROBUST to departures
from equal variances and unequai sample sizes
434
,
.4
.4. .-.- ._,-. . . . .---. . ,"-"''.-"- .-.- , '":':r-4 "; .'-,', - -. -: :q ,'; :-, ,': -,-'-'
D AT A A NA LYSES1
CONTENT VALIDITY
0 SEISITIVITY TO DIFFIIKIT PEASES OF FLIGHT
Within a Flight Segment (i.e., SF0 - SCK) compare differentMeasurement Epochs (TAMK LOADINGS)
SF0 - SCK Takeoff (LOW)to
SF0 - SCK Cruise 1 (Low)
QUESTION: "Can the workload measure discriminate between different phasesOf tne same flight segMent?'
342
D A TA A NA LYSES9
CONTENT VALIDITIT
0 SENSITIVITY TO DIFFERENT FLIGHT SEGOIT3 (HIGH AND LOW TASK LOADINGS)
Between Flight Segments (i.e., LOW & HIGH) compare differentLevels of Workload
SF0 - SCK Descent (LOW)to
SF0 - SCK Descent (HIGH)
* 0 00 #0 0 0 #
QUESTION: "Can tne workload measure discriminate between the same phaseof the different flight segments?"
5 343
%. *
D AT A A 2A LYSES9
ALTERNATE FORKS TEST-RETEST RELIABILITY
0 SESSION ONE TO SESSION TWO
Collapsed across Flight Segments, compare different MeasurementEpochs (TASK LOADINGS)
SF0 - SCK Descent (LOW)to
SMF - SF0 Descent (LOW)
QUESTION: "Is the measure stable? Will the same relative difforencea inworkload be found with repeated testingY"
344
%I
D A TA A NA LYSE 9S
CONSTRUCT VALIDITY
0 CORRELATION COEFFICIENTS COMPARINu VARIOUS WORKLOAD MEASURES
Collapsed ac!ross Measurement Epochs (TASK LOADINGS), compare the sameFlight Segments
SF0 - SCK (HIGH)Physiological Measure #1to
SF0 - SCK (FIGH)SubJective Measure 11
B* # e " 0 0 0 .
QUESTION: "Are different workload measure: sensitive to thle samevarition in ASKDEMANDS?"
% %
> 0
0
.> E
0 la-
m~rw~
347.
-. ,.
0
CL
"~.0C,, - -
M) C)sam
Nam 0
C 0L
c) la) CDa-4 LM u
Cl. a CL
LL
0
34P :
0 L0
13.
34
-W.
cca
0
-J C i
a)a
cc
Cc,
0o €) ."
Q ao 0
(I))
cc-- Cm3_e 0 0c -0
00
z
N *~--~-50
CLI)
CL_ __ _
(fl~L- 1.
00
(Em)0y
a) J
ul54CDi)
LL0
-Jo _ _ _ _ _ _ _ _ _
(cU-
C!)LLe)0 4
LL_ _ _ _ U
C0
0~
.2 0omo"
0"
U%
rcn
CL)
ui U)
,_ ,_0 0 Cc
LL a . LM -Lm 0 cn--O 0 y is .-
U-)
0) U) (n .oCL o
-Ca
CL
352
-o 0 ¢
oOo
0
UL. 0in %4f 0 .
cr AMN0 a cncc 0 04 r.
oo - E c0 ~ *~5soeE C E
BOBI 0 Z 0 0
353
cz00
0 4)
U-' 00
C.) 0
0oc 0-c
OL ccocEL- >b
Lu. Eo-0o EG)Cc2CL 20O0 0OQ~m~c
U0 E
cc UU U
0 .. 00 0 0-
ca. 4 2
C~j cviL6 C6
IWO-
CL m C0 >
~~0C 00C
%fno Q + 0"* (-) a 0 0
cc W> 0 0- ~CZ
o 0 WW
CL LL -- cn QI~ ftw_ a U... - cz
=L T 0 0
0
LoL
oo 0
0 0
L..
(5 L6 q::r rOL6L
OppA -
0 C,,
Ula
0oC.)
0V 0 a
a
0j 4-
U) U,( C 6 d.T-Lu C C
0,.
i w - -' 'c~cc
-L- 0
0 0 0 0LL
(1 (10z- r 0 a)0 0000 E~1.
0 o0o 0
)- 0 CZ0 00
19t L6 r ccco a; C6 P-: 6S
3r7o9
cuc
0
04600
I--
00
UL 0
.00
00
.2 n
0 T0
CLo~r) *- i
L..
00
4) CD
-I n
cc 0
'U- CL
0 C -
r. aCL0
am0 0
-~~ oc * m
Q~l Cc0)C
C)~Cc
0 0 0
~~C c v 0 -
0@O
.00
LO)0
G)
0 0
LM0 co0
L- o6.o ~0 ~0
.n U.) mf (n 7-*a 0-0
r. C_
.0C *C 0- 0 C2
0 0
0~~Lm CcI..C0 LL 0 E
*c .* E c~6
ci 6co co S
cz 0
U).
0 CD
~~Cn
0 -cz
-oo E
C) 0Co r.
.)~ .- 2- 0 cla =
0 0.c0) r V
0CC r ooos 0LL a C.)
0 0s -
o 0 'V FLc
= 0
5.~ ~ 75* . . 5* * 5* -0 r,%
0 C0
iN=
kawo
0 C0
S"
0 mow
C0 0C0 Co
0 n, IIo ,-- 0 0
N'
.SI.
CCL0 E
cc) IC
00Q W x
C.) m-
cn 0 d/)
* 0 0
lo
00
6m .. 2
- E cc~
CL E x00
CLU
Cl) r.
cc
MoA. C)oQEM
-0
(D 0
LIM cU0 o-o(A U)C Cnu
00
a)-
> CL
" .:
000
U) C)
_ C, 0
0~
C_)- -'
• - *0- ")
E -W E .d
--' 0 -
E CE
=...
E0
c)
Cf) a.16 G
Ch
cc -r
ooE
CL C) cmB~
0 Q Qt
~~0 .46 0-
* 0
lr~r
0
F-.-
EE
E 0
II
0
C,. .I 0"o .n 0
C/ (1 .- -
i 0 --
_, (/) Q.6 "
j i* %% * ~
cuE
00I Ul-
0 -
cc 0
-_-4 - o 0-
c 0 (n-0Q E.
oz 0
0 0 L.4. 0UU.
0
• -- Cn .. ) 0:0300
cc 0.
3-6
• • I-
' I I ,-, "" . .' ." . " " "* . " * ." " ' ."" - . ."" "i . ", " i C" • "
ASSESSMENT OF CREW NAMWORKLOAD WORKSHOP SURVEY
MEASURE PREFERENCES
Because your views may be different from those already presented by thespeakers, or, from those presented from your working group, we are interestedin what you believe to be the best workload measures. If your views aredifferent from those already presented, please indicate which measures youbelieve are best, in order, and provide a rationale for your view.
,J
-1 Measure Type: Subjective Physiological Performance
Measure Rationale
1.
2.
SIMULATION RECOMMENDATIONS
How would you improve the desii of the part-task Simulation?
A
~369
SUGGESTED KILSURES
Questionnaire Summary
GROUP MEASURE COUNT
Physiological Voice Measures 3
Physiological Auditory Canal Temperature 2
Physiological ERP 2
Physiological EEG Spectra 1
Physiological Eye Movement 1
Subjective Modified Cooper-Harper 2
Subjective Airbus 2-8 Scale 1
Subjective Bedford Scale 1
Subjective Boeing Comparative 1
Subjective In Flight, High Level Subjective 1
Perform. Sec. Time Estimation 2
Perform. Sec. Embedded Communication 1
Perform. Sec. Unobtrusive Embedded Secondary Task 1
Perform. Prim. Primary Task (mission effectiveness) 1
Other Performance/Physiological 1
Other SSER 1
Other Subjective/Physiological 1
Other Timeline Analysis 1
371
Questionnaire Summary
SINULATION SUGGISTIONS
SUGGESTED ADDITIONS:o Increase the amount of mechanical failures (switch failure to
hold, CB fails).o Pre-correlate workload with timeline analysis results to confirm
validity.o Use glideslope as a primary task measure for AZ, ALT deviation,
power required (weighted differently across the scenario).o Include a very high workload condition to be sure the measures are
measuring anything.o Use filtering or other 0.1Hz analysis.o Use NDB approach or LDA approach to load the captain.o Address crew members as captain, first officer, and second officer.o Manipulate communications workload.o Perform an apriori test for level factor as a 212 (3 treatments
with 1 within subjects factor) MANOVA. If there is no effect, itcan be eliminated as a factor for later analysis.
o Use synthetic tasks early on, then embedded tasks later in fullsystem flight tests.
o Design a new simulator with built-in data systems and flexible,quick-change capabilities.
o Design criterion tasks with built-in measurement schemes.o Measure co-pilot workload.o Test measures in an actual 727 flight at some time.o Include more ECG data in 1 .0, 1 .7, and 1 .5 minute segments for FFT
analysis.o Make windows longer to allow heart rate variability to stabilize.
SUGGESTED ELIMINATIONS:o Eliminate throttle reversal (measures technique, not workload).o "Improve the scenario"o Eliminate 2 workload states.o Do not use 727 for testing. Old model, will probably never be
certified again. Can you be certain )f validity for a highlyautomatic plane?
o Eliminate compressor stall at 10,000 feet (unrealistic, should beat a higher altitude).
o Do not assume equa. workload at the 3 airports selected.
SUGGESTED CONSIDERATIONS:o Assymetric performance transfero Range effects I stress differential interact;ono New generation A/Co Full crew interaction
f Sensitivity of measureso Between subjects and between test run variatilityo Anticipaticn oDtweer high and low levelsD Wr.at part tasx simula&lon?
Me-nan-La. malfunctions load seccnd officer more than captair..C F~tfails "
oo
7 i
,.?.i". : / <., <,,','.
Questionnaire Summary
SUGGESTED REFENUECES
IN REFERENCE TO MEASURES:
o Speyer's handouts [Speyer]
o Otis Elevator (c. 1965-1970) [Parks]
IN REFERENCE TO SCENARIO:
o Poulton and Freeman (1973) [Hancock]
o WPAFB, Human Factors Branch, Crew Station Designs Division c. 1979
(call Richard Gesselbart (513) 255-4109) [MetcherJ
o O'Donnell and Eggemeier; Gopher and Donchin (current) [Derr-ick!j
17.
- 4" & - 4"'~ s.. *~ ... - -
'.4
*4
ILMED
.4 f\~P\f~gCat4 I?'S
D$VC0
0
a
*~~M '~q *~. -
