This article was downloaded by: [University of Tennessee At Martin]On: 04 October 2014, At: 07:07Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,UK

The International Journal ofAviation PsychologyPublication details, including instructions forauthors and subscription information:http://www.tandfonline.com/loi/hiap20

Effects of Pilot Experience onRecall of Information FromGraphical Weather DisplaysDavid O'Hare a & Amy Waite aa Psychology Department , University of Otago ,Dunedin , New ZealandPublished online: 17 Jan 2012.

To cite this article: David O'Hare & Amy Waite (2012) Effects of Pilot Experience onRecall of Information From Graphical Weather Displays, The International Journal ofAviation Psychology, 22:1, 1-17, DOI: 10.1080/10508414.2012.635122

To link to this article: http://dx.doi.org/10.1080/10508414.2012.635122

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all theinformation (the “Content”) contained in the publications on our platform.However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness,or suitability for any purpose of the Content. Any opinions and viewsexpressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of theContent should not be relied upon and should be independently verified withprimary sources of information. Taylor and Francis shall not be liable for anylosses, actions, claims, proceedings, demands, costs, expenses, damages,and other liabilities whatsoever or howsoever caused arising directly orindirectly in connection with, in relation to or arising out of the use of theContent.

This article may be used for research, teaching, and private study purposes.Any substantial or systematic reproduction, redistribution, reselling, loan,sub-licensing, systematic supply, or distribution in any form to anyone isexpressly forbidden. Terms & Conditions of access and use can be found athttp://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

THE INTERNATIONAL JOURNAL OF AVIATION PSYCHOLOGY, 22(1), 1–17Copyright © 2012 Taylor & Francis Group, LLCISSN: 1050-8414 print / 1532-7108 onlineDOI: 10.1080/10508414.2012.635122

Effects of Pilot Experience on Recallof Information From Graphical

Weather Displays

David O’Hare and Amy WaitePsychology Department, University of Otago, Dunedin, New Zealand

Good weather-related decision making is critical for the safe conduct of generalaviation flights. A number of tools have been developed in recent years to helppilots visualize the actual and forecast weather for their planned flight. In this studywe compare the effectiveness of a graphic weather visualization tool designed forpreflight decision making with conventional coded reports and plain English trans-lations. Qualified pilots from several aero clubs were asked to view 4 hypotheticalcross-country flights. Departure and destination weather reports (METARs) wereprovided for each flight in 1 of 4 formats—code, plain English, graphic, or bothgraphic and plain English. Following a short interference task, the pilots were askedto recall as much of the weather information as possible. Results showed that pilotscorrectly recalled most information from the dual graphic and plain English displaybut this was largely attributable to the additional time required to process the 2 dis-plays. Although the formats were rated as equally interpretable, experienced pilotswere able to recall much more from the traditional coded METARs than less expe-rienced pilots. Implications for the design and acceptance of novel display formatsof meteorological information for pilots are discussed.

Weather-related crashes continue to be one of the most common hazards in generalaviation (GA). This encompasses most activity by aircraft outside the military andscheduled airlines including agricultural aviation, recreational flying, air-taxi andcharter flights, and so on. The large majority of GA operations are carried outin unpressurized aircraft that spend most, or all, of their time operating below10,000 ft. At these altitudes weather conditions are at their most changeable and

Correspondence should be sent to David O’Hare, Psychology Department, University of Otago,P.O. Box 56, Dunedin, New Zealand. E-mail: ohare@psy.otago.ac.nz

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

2 O’HARE AND WAITE

difficult to forecast. In certain U.S. states and in many countries such as NewZealand, hilly or mountainous terrain combines with volatile weather to createchallenging flying conditions for GA.

Many reports and studies of GA weather-related crashes have been reported(National Transportation Safety Board, 2005). Numerous recommendations havebeen issued by the National Transportation Safety Board for improvements tothe measurement, presentation, and dissemination of weather information. Wherepilots used to rely on face-to-face or telephone briefings from trained personnelthere is now a great range of sources of information from TV channels to theInternet. In the United States, the majority of GA pilots utilize weather brief-ings from automated flight service stations (AFSS), particularly on days withpoor weather (Prinzo, 2006). Approximately 10% of pilots changed their flightplans after contacting the AFSS. However, in the cases where visual flight rules(VFR) flight was not recommended to pilots on VFR flight plans, 27% did notsubsequently change their flight plans (Prinzo, 2006). This is consistent with bothepidemiological (e.g., Hunter, 2001) and laboratory research (Pauley, O’Hare, &Wiggins, 2008) that suggests that pilots who continue into adverse conditions areoften aware of the risks but choose to tolerate them for some reason or other.

In addition to the traditional abbreviated coded format of official aviationweather reports and forecasts (World Meteorological Organization, 2008), pilotscan hear or read plain-language statements or view static and dynamic dis-plays of weather graphics. For example, in the United States the NationalOceanic and Atmospheric Administration Web site (http://adds.aviationweather.gov/) provides users with the option of obtaining a ‘translation’ of the codedAerodrome Meteorological Reports (METARs) and Terminal Area Forecasts(TAFs). Examples of a coded METAR and plain English translation are shown inFigure 1. The site also provides additional graphical representations of turbulenceand icing information.

A software package (FlySafeTM) has recently been developed in New Zealandto enable pilots to log on to their normal official source of aviation weatherinformation (www.metflight.co.nz) and receive a supplementary plain language“translation” of the coded reports (METARs) and forecasts (TAFs). In addi-tion, the software can generate a graphic display of the information as shownin Figure 2.

The aim of this study was to evaluate these additional methods of present-ing aviation meteorological information to pilots in comparison to the traditionalcoded format available from aviation weather Web sites. A critical factor to inves-tigate is pilot experience as pilots of different ages and flight experience levelsmight respond quite differently to the traditional (coded) and novel (graphic) for-mats. Previous research with GA pilots (Wiggins & O’Hare, 1995) has shownboth qualitative and quantitative differences in the information search strategiesof inexperienced and experienced pilots. Presented with a series of screens (e.g.,

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

EFFECTS OF PILOT EXPERIENCE ON RECALL 3

(a)

METAR for NZDN Dunedin –Tuesday 16th August

METAR 160200Z 02001KT 020VO80 40KM FEW100 SCT200 04/00 Q1023

(b)

METAR for NZAK-Auckland

FRIDAY, July 8 1000 LOCAL (all cloud above ground level).

WIND: 150 T at 19 knots

VISIBILITY: 15 kilometres

CLOUD: few 800 feet; scattered 2000 feet; broken 6000 feet

TEMPERATURE: 10 C

DEW POINT: 7 C

QNH: 1025 hectopascals

FIGURE 1 (a) The traditional coded METAR report. (b) A plain English translation.

aircraft performance, meteorological conditions) relevant to a VFR cross-countryflight, experienced pilots accessed fewer pieces of information in a more efficientand structured manner than the less experienced pilots. In this context we predictthat experienced pilots will exhibit superior performance regardless of weatherdisplay format. We also predict that differences between experienced and lessexperienced pilots will be less apparent with the novel and unfamiliar graphicformat and most apparent with the traditional coded METAR format.

From a theoretical perspective we wished to investigate the differences amongthe three formats (coded, plain language, and graphical) for pilots of differentexperience levels. From a practical perspective we also wished to include acondition that was close to operational reality as instantiated in software suchas FlySafeTM where pilots would be able to study both the plain language andgraphic formats.

Modifying the format of reported weather information can improve pilotperformance. For example, Ebbatson, Harris, and Jarvis (2007) found thatmost pilots were unable to correctly calculate the crosswind component whenwind information was provided in the standard (International Civil AviationOrganization) format (e.g., “surface wind 290 at 20”). This requires the pilot to

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

4 O’HARE AND WAITE

FIGURE 2 The graphic display of METAR information generated by the FlySafe software(color figure available online).

then mentally compute the actual crosswind component to the runway in use whileflying the final approach. Using an online questionnaire, the researchers foundthat only 1 out of 55 respondents could accurately compute the crosswind com-ponent from the standard format wind information. There was a strong tendencyto underestimate the actual crosswind component. A follow-up flight simulatorstudy compared the performance of pilots explicitly given the crosswind com-ponent (e.g., “crosswind 19 knots from your right”) with a group provided onlywith the standard format. Although there was no difference between the groupsin the proportion of pilots who chose to continue and land (despite the crosswindexceeding the crosswind limits of the aircraft), those who did not have to mentallycompute the crosswind component flew more smoothly and accurately than thosewho received the conventionally formatted wind information.

PICTORIAL DISPLAYS

Human factor and ergonomics researchers have extensively investigated thedesign of static visual displays (e.g., Sanders & McCormick, 1987). Graphicdesigners (e.g., Tufte, 1983) have provided extensive advice on the design of data

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

EFFECTS OF PILOT EXPERIENCE ON RECALL 5

graphics. These different perspectives generate similar principles (e.g., presentthe information the user needs without clutter and distortion). An almost univer-sally accepted assumption is that a graphical representation will be superior to asentential, or text representation (Scaife & Rogers, 1996).

Extensive research has been carried out on the role of various types of graphicrepresentations in enhancing reasoning and problem solving (Larkin & Simon,1987; Scaife & Rogers, 1996). Diagrammatic representations of physics prob-lems, for example, can aid understanding of the relevant concepts by groupingrelated elements (Larkin & Simon, 1987) or by helping to build an appropriatemental model (Glenberg & Langston, 1992).

A well-established finding in cognitive psychology is that pictures are betterremembered than words (Hockley, 2008; Paivio, Rogers, & Smythe, 1968). Thereare various possible explanations for this “picture superiority effect” from Paivio’s(1971) dual-coding theory to depth of processing theory (Craik & Lockhart,1972). Drawing on Paivio’s (1971) work, Mayer (1997) developed a theory ofmultimedia learning. Mayer’s first principle (which he called the “multimediaeffect”) is that “It is better to present an explanation in words and pictures thansolely in words” (Mayer, 1999, p. 559). Wright (1977) emphasized the advan-tages of using redundant or related pictorial material rather than text alone ininstructional material.

Britten (1989) investigated recall and recognition of television weather fore-casts. Britten predicted that information presented bimodally (i.e., both verballyand pictorially) would be better recalled and recognized than information pre-sented in just one mode. Following the picture superiority effect, it was predictedthat participants would exhibit better memory performance for the visually pre-sented weather forecasts than the auditory forecasts. A series of experiments wasconducted in which participants were exposed to a typical TV weather forecastand were then asked to either recall the material or complete a recognition test.Forecasts that utilized both visual and auditory modes were much better remem-bered than unimodal forecasts. Surprisingly, support for the picture superiorityeffect was much weaker. Advantages for the visually presented information onlyemerged in some of the experiments. The visually presented weather forecastswere generally not better remembered than the aurally presented information.

Based on the principles already outlined we hypothesize that pilots who receivetheir flight weather information in a graphic form will recall that information bet-ter than those pilots who receive the same information in plain English text or inan abbreviated coded format. Furthermore, we hypothesize that pilots who receivethe information in two modes (graphic and plain English text) will recall the infor-mation better than those who receive the information in a single mode. Finally,we expect that the effectiveness of dual mode and visual/graphic presentations ofmeteorological information will be greatest for pilots who have lower levels ofexperience with the traditional coded format.

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

6 O’HARE AND WAITE

METHOD

Participants

The participants were 2 female and 33 male pilots, aged between 17 and 63 years(M = 27.21, SD = 11.63). Pilots were recruited from three separate aero clublocations. The pilots had a median of 280 total flight hours of experience, and hadaccrued a median of 148 hr pilot-in-command. Pilots had accumulated a meanof 35 hr of flight (SD = 37.78) over the 90 days prior to the experiment. Thehighest level of pilot certification was an air transport pilot license (ATPL), heldby one pilot. Nineteen pilots held commercial pilot licenses (CPLs), and 10 hadprivate pilot licenses (PPLs). Four pilots did not have any certification at the timeof the study. On average pilots had held their licenses for approximately 2 years.Seventy percent of pilots held instructor ratings. All participants gave informedconsent prior to participation, and were tested individually.

Apparatus

An Acer TravelMate 2304L Ci laptop computer was used to run MicrosoftPowerPoint 2002. Weather reports and maps used in PowerPoint presentationswere adapted from FlySafeTM using Jasc Paint Shop Pro Version 7.04.

Materials

PowerPoint presentation: Overview. The experimental materials werepresented using four Microsoft PowerPoint presentations, consisting of a totalof 25 slides. These presentations were divided into four cross-country flights,each approximately 90 nm long in both the North and South islands of NewZealand. The order of the four flights was counterbalanced across each of thefour presentations.

Introductory slides. Each of the four flights was introduced by a slide,which provided the names of the departure and destination airfields, located theflight within New Zealand, and illustrated the topography between each airport.The purpose of this slide was to both familiarize participants with the location ofthe flight, and to prepare them for the presentation of the flight’s METARs.

A map of New Zealand, placed in the top left hand of the slide, was used todepict the regional location of the flight. This was achieved by framing the flight’slocation in a red box. To the right of this map was a larger map, which illustratedthe topography between the departure and destination airfields.

METARs: Content. Two METARs, one for the departure airfield and one forthe destination airfield, were presented for each of the four flights. This yielded

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

EFFECTS OF PILOT EXPERIENCE ON RECALL 7

a total of eight METARs for each of the four presentations. Each METAR con-tained the following information: time, wind direction and speed, visibility, cloudcover, temperature, dew point, and barometric pressure. The number of items ofinformation ranged from 30 to 38 (M = 34, SD = 3.37).

Display format. The METARs were displayed in one of four ways—coded, plain English, FlySafeTM graphic, or both plain English and FlySafeTM

graphic. Weather information within the coded and written English METARs waspresented in Arial size 16 font, in black lowercase letters.

Demographics questionnaire. Demographic information was assessedusing a short questionnaire. This questionnaire asked for information regardingthe pilot’s age and gender, as well as questions relating to his or her flight expe-rience. Pilots were required to indicate the total number of hours they had flownsince taking up flying, as well as the number of hours they had spent in cross-country flight. Pilots were also asked to indicate the total number of hours theyhad spent as pilot in command, and as pilot in command in the last 90 days. Thepilots were also asked to indicate the highest level of pilot certification held, thelength of time they had held this for, and whether they had ever held a flightinstructors rating. Pilots with the flight instructors rating were asked to indicatethe category of this rating (A, B, or C) and the length of time they has held thiscertification.

Load sheets. Participants were asked to complete a typical GA aircraft loadsheet after viewing the weather for each flight. This task was designed to preventparticipants from rehearsing information prior to recall. Participants were giventhe weight of the aircraft, passengers, and cargo in pounds. They were also giventhe number of gallons of fuel on board the aircraft. They were required to calculatethe aircraft’s takeoff weight from the information provided.

Interpretability scale. A 5-point Likert scale ranging from 1 (very easy) to5 (very difficult) was used to measure the interpretability of the four METARdisplays. Participants were asked to circle the number that best corresponded withtheir opinion.

Design

The experiment was a one-factor within-subjects design. The manipulated vari-able was the METAR display type, of which there were four different types, acoded version, plain English translation, graphical display, and dual-mode dis-play, which presented METARs in both plain English and as a graphical display.There were five measured or dependent variables for each of the four METAR

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

8 O’HARE AND WAITE

display types. These were viewing time, total number of weather items recalledcorrectly, total errors, total omissions, and rated interpretability of the display.

Procedure

Participants were tested individually. Prior to commencing the experiment, par-ticipants completed the demographics questionnaire. Each participant was thenseated in front of a laptop computer. The experimenter sat behind, and just to theright of the participants for the duration of the experiment. This was done so thatthe experimenter could monitor participants’ performance without serving as adistractor. Once comfortable, participants were instructed to begin the PowerPointpresentation by pressing the space bar. This began the first of five introductoryslides, which participants were instructed to read over at their leisure. Prompts atthe bottom of each slide were used to indicate when participants should move onto their next set of instructions.

Following these instructions, participants were given sample METARs illus-trating the coded, textual, and graphical display formats. Participants were told totake as long as they needed to familiarize themselves with each of the displays.Once the participants had finished viewing each of the sample METARs, theywere instructed to press the space bar for an introduction to their first flight. Thisintroduction provided participants with a topographical map showing the depar-ture and destination airfields. Participants were then informed that they were topress the space bar when they were ready to receive the first set of METARs. Afterviewing the METARs for as long as they wished, participants were then asked tocomplete a load sheet for that flight using pen and paper. Following this, partici-pants were asked to recall as much information as they could remember for eachof the METARs they had just seen. Participants were told that they could writeor illustrate this information on the page provided. Participants were told to labelany drawings or figures, and to take as long as they required to complete this task.Once participants had indicated that they could remember no more information,they were asked to rate the interpretability of the display.

This procedure was repeated three more times, for each of the three additionalflights. On completion of the final flight, participants were invited to comment oneach of four METAR displays, after which they were debriefed and thanked fortheir participation.

Scoring Criteria

The total number of weather items recalled correctly, total errors, total errorsgreater or less than 50% of reported values, and total omissions were counted foreach participant. Information recalled to within 10% of reported meteorologicalvalues was recorded as correct (e.g., 12 kt was coded as correct where the actual

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

EFFECTS OF PILOT EXPERIENCE ON RECALL 9

wind speed was reported as 13 knots). Conversely, responses greater than 10% ofreported meteorological values were recorded as errors. Of these errors, those thatwere greater or less than 50% of meteorological data values were also recordedas serious errors (e.g., a cloud height of 4,500 ft was recorded as incorrect wherethe actual response was 11,000 ft). Finally, omissions were defined as any datavalue omitted from recall. Participants’ scores for each dependent variable wereaggregated across each set of METARs to form a total score for each of the fourMETAR display types.

RESULTS

Recall of Weather Information

The number of items correctly recalled for each display type was submitted to a2 (certification) × 4 (display type) mixed analysis of variance (ANOVA). Therewas a main effect of display type, F(3, 90) = 3.763, p = .013. A pairwise analy-sis indicated that pilots recalled significantly more weather items when METARswere presented in the dual-mode display (M = 16.87, SD = 6.42) as compared towhen they were presented in the graphical display (M = 13.18, SD = 6.44; seeFigure 3).

There was also a main effect of certification, F(1, 30) = 7.991, p = .008.Profile plots indicate that highly certified participants recalled a greater amountof information (M = 17.17, SE = 1.16) than those with lower level certifications

0

2

4

6

8

10

12

14

16

18

Code plain English Graphical plain English+ Graphcial

METAR Display Type

Mean

Nu

mb

er

of

Item

s R

ecalled

Co

rrectl

y

FIGURE 3 Mean number of correctly recalled items of weather information as a function ofMETAR display type (code, plain English, graphical, and dual mode).

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

10 O’HARE AND WAITE

02468

101214161820

Code plain

English

Graphical plain

English +

Graphcial

METAR Display Type

Mean

Nu

mb

er

of

Item

s R

ecalled

Co

rrectl

y

High Certification

Low Certification

FIGURE 4 Mean number of correctly recalled items of weather information for the highlycertified and lower certified groups.

(M = 12.04, SE = 1.40) across each of the four display types. There was also asignificant interaction between display type and certification, F(3, 90) = 2.665,p = .053. Profile plots indicate that highly certified participants recalled a signifi-cantly greater number of weather items (M = 18.58, SE = 1.41) than less certifiedparticipants (M = 9.85, SE = 1.70) when METARs were presented in the codedformat (see Figure 4).

Effects of Recent Flight Experience

The number of items recalled correctly for each display was also investigated asa function of the total hours flown over the last 90 days. A 2 (flight recency) × 4(display type) mixed ANOVA was conducted. Again there was a main effect ofdisplay type, F(3, 78) = 3.020, p < .035. There was also a main effect of flighthours over the last 90 days, F(1, 26) = 5.235, p = .031. Participants who had accu-mulated more than 20 hr of flight in the 90 days prior to the experiment recalleda greater amount of weather information accurately (M = 17.05, SE = 1.37) thanparticipants with less than 20 hr of flight (M = 12.46, SE = 1.47). There was nosignificant interaction (see Figure 5).

Total Errors

The total number of errors made by participants for each of the four METAR dis-plays was submitted to a 2 (certification) × 4 (display type) ANOVA. There wereno main effects or interactions. Similarly, a 2 (flight recency) × 4 (display type)ANOVA also yielded no main effects or interactions. In other words, participants,

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

EFFECTS OF PILOT EXPERIENCE ON RECALL 11

0

5

10

15

20

25

Code plain

English

Graphical plain

English +

Graphcial

METAR Display Type

Mean

Nu

mb

er

of

Wea

ther

Item

s R

ecalled

Co

rrect

Lower than 20hours flight

Greater than 20hours flight

FIGURE 5 Mean number of correctly recalled items of weather information as a function ofrecent flight hours.

regardless of their certification level, or their recent flight experience, made asimilar number of errors for weather information presented in each of the fourdisplay types.

Omissions

A 2 (certification) × 4 (display type) ANOVA showed a main effect of displaytype, F(3, 90) = 9.390, p < .001. A trend analysis revealed that this effect wasquadratic. As can be been in Figure 6, participants omitted significantly lessweather information when METARs were presented in plain English (M = 13.34,SD = 6.15) and as graphical displays (M = 14.21, SD = 7.22). In contrast, thenumber of omissions was significantly higher when METARs were presented inthe coded format (M = 17.79, SD = 6.68) or dual-mode format (M = 17.25,SD = 8.55).

There was also a main effect of certification, F(1, 30) = 8.539, p < .007.On average less certified participants made a greater number of omissions(M = 19.14, SD = 1.56) than more certified participants (M = 13.23, SD = 1.29),across the four display types. Similarly, a 2 (flight recency) × 4 (display type)ANOVA again showed a main effect of display type, F(3, 78) = 7.416, p < .001,and also a main effect of the number of flight hours over the last 90 days, F(1,26) = 132.621, p = .048.

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

12 O’HARE AND WAITE

FIGURE 6 Mean number of omission errors as a function of METAR display type (code,plain English, graphical, and dual mode) (color figure available online).

Interpretability of METAR Displays

Both a 2 (certification) × 4 (display type) ANOVA, and a 2 (flight recency) × 4(display type) ANOVA showed no significant effects or interactions. Participants’ratings of each METAR display therefore did not differ as a function of cer-tification level or recent flight time. Participants rated the coded (M = 1.9,SE = .16) and plain English (M = 1.87, SE = .19) METARs as slightly eas-ier to interpret than the graphical (M = 2.1, SE = .19) and dual-mode METARs(M = 2.16, SE = . 22). In other words, participants rated each of the four METARdisplays as “very easy” to “relatively easy” to interpret.

Viewing Times

Viewing times were submitted to a 2 (certification) × 4 (display type) mixedANOVA. There was a main effect of display type, F(3, 87) = 6.83, p < .001.Pairwise comparisons indicated that viewing times were significantly longer forthe dual-mode display (M = 1.97 min, SD = 0.84), compared with viewingtimes for the coded (M = 0.60 min, SD = 0.16) and plain English displays(M = 0.64 min, SD = 0.16). There was no main effect of certification or aninteraction effect (see Figure 7).

The differences in correct recall noted earlier might be due to differences in theamount of time spent viewing each display type. To control for this, analyses of

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

EFFECTS OF PILOT EXPERIENCE ON RECALL 13

Mean Viewing Time (mins) as a function of METARDisplay Type

0

0.5

1

1.5

2

2.5

Code plainEnglish

Graphical plainEnglish +Graphical

METAR Display Type

Mean

Vie

win

g T

ime (

min

s)

FIGURE 7 Pilots’ mean viewing times as a function of METAR display type (code, plainEnglish, graphical, and dual mode).

covariance (ANCOVAs) were carried out with viewing times for each display typeas the covariate. A 2 (certification) × 4 (display type) ANCOVA, controlling forviewing time, showed no main effect of display type, F(3, 75) = 1.05, p = .375,and no interaction between display type and certification, F(3, 75) = 4.850, p = .1.The main effect of certification remained, however, F(1, 25) = 5.653, p = .025).A 2 (flight recency) × 4 (display type) ANCOVA again showed no main effectof display type, and no main effect of flight recency, F(1, 25) = .3.67, p = .069.There was also no interaction effect.

DISCUSSION

Pilots correctly recalled more information from METARs when the informationwas presented in dual mode using both a plain English statement and a corre-sponding graphic representation. However, this advantage was achieved at thecost of a tendency to omit more information and to take significantly longer toassimilate the information compared to the other single-mode formats. In fact,once viewing time was controlled for, the differences in correct recall between thedisplays disappeared.

Pilot expertise and recent experience were important determinants of perfor-mance. More expert pilots (as measured by their level of certification) recalledmore and omitted less information than less expert pilots. Similarly, those pilotswith higher levels of recent experience recalled more and omitted less than pilots

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

14 O’HARE AND WAITE

with lower levels of recent flight experience. Of particular interest was the factthat the difference between experienced and less experienced pilots was mostpronounced for the traditional coded METAR format.

Despite the various differences in recall and omissions as a function of dis-play type, pilots rated all the displays as equally interpretable. This dissociationbetween subjective opinion and objective performance is not unusual in humanfactors research. For example, early studies of altimeter design (Rolfe, 1969)found that pilots expressed a preference for one design (the separable analog-digital) but performed significantly better with the other design (the integratedanalog-digital).

The best level of recall of weather information was obtained when the informa-tion was presented twice in both plain English and as a graphic display. However,the superior performance seems to have been largely due to the increased timespent viewing the two displays of information rather than to any inherent advan-tage of multiple encoding modes. Surprisingly, the graphic display by itself didnot lead to improved performance. There are two possible explanations for this.One is that the particular graphic display generated by FlySafeTM might not havebeen the most effective design possible.

In a coded METAR the information is presented in a fixed order: wind directionand speed, visibility, cloud cover, temperature, dew point, and barometric pres-sure. The FlySafeTM graphic presented the same information in reverse order withtemperature and dew point on the left of the display and wind speed and directionon the right (see Figure 2). Memory for the information in the graphic displaymight have been impaired in this study as it was incompatible with pilots’ exist-ing knowledge structures. Similar effects have been found with drivers. Borowsky,Shinar, and Parmet (2008) found that the ability of experienced drivers to detecttraffic signs actually fell below that of inexperienced drivers when the locationof the signs was unexpected. O’Hare and Stenhouse (2009) subsequently showedthat an ergonomically redesigned graphic display in which the information ordermatched the coded METAR was more effective than the plain English statement.

A second explanation lies in pilots’ previous experience and familiarity withthe coded METAR format and inexperience with the graphic format. The resultsshowed that experienced pilots were able to recall far more with the highly famil-iar coded METARs than less experienced pilots and performed worst with thenovel and unfamiliar graphical display.

As pilots have access to an ever increasing range of sources of meteorolog-ical information, reliance on traditional coded METARs is likely to diminish.Many of the supplementary sources of information (TV, Internet, etc.) relyheavily on graphic formats. Attempts have been made to design flight plan-ning tools that incorporate graphic weather representations. In addition to theNew Zealand FlySafeTM software used in this study, other examples such as theAviation Weather Data Visualization Environment allow users to access abbre-viated plain-text versions of METARs and TAFs as well as topographical charts

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

EFFECTS OF PILOT EXPERIENCE ON RECALL 15

overlaid with symbolic representations of key information such as ceilings andvisibility, wind speed and direction, and so forth (Spirkovska & Lodha, 2002).

Other work has focused on in-flight display systems. Early work on a graphicweather system that would display near real-time radar precipitation imagerywas undertaken at Massachusetts Institute of Technology (Lind, Dershowitz, &Bussolari, 1994). More recently NASA developed the Graphical WeatherInformation System, which provides in-flight radar imagery and METARs inboth text and symbolic form (Latorella & Chamberlain, 2002) on a displaydesigned for GA use. In both cases users were generally positive and enthusi-astic about the technology and exhibited increased confidence in their in-flightdecision making. However, as this study demonstrates, although new display for-mats might offer advantages in the long run, providing information in ways thatare incompatible with existing displays could produce performance decrements inthe short term.

A well-designed graphical display of weather information can be effective(O’Hare & Stenhouse, 2009) in improving pilot performance. However, very care-ful consideration of human factors issues is required if the display is to reachoptimal effectiveness (O’Hare & Stenhouse, 2008). Surprisingly, in this studythe graphic display of weather information did not produce better performanceoverall except when it was paired with the same information presented in plainEnglish. Consistent with the multimedia effect (Wright, 1977) the combined dis-play led to the highest level of recall. However, this is most likely due to theadditional time required to process the two redundant displays. In operationalsettings where pilots are rushing, their preflight preparation time might not beallocated to processing the same information in two different formats.

As predicted, experienced and better qualified pilots performed better than lessexperienced or less well qualified pilots and the effects of previous experiencewere greatest for the conventional coded format and least for the novel graphicdisplay. The adoption of novel display formats requires the assent of more expe-rienced practitioners who find themselves gaining the least benefit from the newapproach. This could, in part, account for the inherent conservatism of the aviationindustry in regard to adopting new displays.

In conclusion, this study provides evidence that a novel approach to presentingmeteorological information using plain English and a graphic display can improvepilots’ memory for the information presented. As weather-related crashes con-tinue to account for a high proportion of GA fatalities, further development ofweather visualization tools seems likely. A research-based understanding of theirrole in improving pilot decision making deserves further attention.

REFERENCES

Borowsky, A., Shinar, D., & Parmet, Y. (2008). The relation between driving experience andrecognition of road signs relative to their locations. Human Factors, 50, 173–182.

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

16 O’HARE AND WAITE

Britten, A. J. (1989). Memory for television weather forecast information: The role of presentationmodality and content structure. Unpublished doctoral dissertation, University of Wales, Cardiff,UK.

Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research.Journal of Verbal Learning and Verbal Behavior, 11, 671–684.

Ebbatson, M., Harris, D., & Jarvis, S. (2007). Crosswind landings in general aviation: A modifiedmethod of reporting wind information to the pilot. International Journal of Aviation Psychology,17, 353–370.

Glenberg, A. M., & Langston, W. E. (1992). Comprehension of illustrated text: Pictures help to buildmental models. Journal of Memory and Language, 31, 129–151.

Hockley, W. E. (2008). The picture superiority effect in associative recognition. Memory andCognition, 36, 1351–1359.

Hunter, D. R. (2001). Retrospective and prospective validity of aircraft accident risk indicators. HumanFactors, 43, 509–518.

Larkin, J. H., & Simon, H. A. (1987). Why a diagram is (sometimes) worth ten thousand words.Cognitive Science, 11, 65–99.

Latorella, K. A., & Chamberlain, J. P. (2002). Graphical weather information system evaluation:Usability, perceived utility, and preferences from general aviation pilots (Paper No. 2002-01-1521).Langley, VA: NASA Langley Research Center.

Lind, A. T., Dershowitz, A., & Bussolari, S. R. (1994). The influence of data link-provided graphi-cal weather on pilot decision making (Rep. No. DOT/FAA/RD-94/9). Springfield, VA: NationalTechnical Information Service.

Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions? EducationalPsychologist, 32, 1–19.

Mayer, R. E. (1999). Instructional technology. In F. Durso (Ed.), Handbook of applied cognition(pp. 551–569). New York, NY: Wiley.

National Transportation Safety Board. (2005). Risk factors associated with weather-related generalaviation accidents (Safety Study No. NTSB/SS-05/01). Washington, DC: Author.

O’Hare, D., & Stenhouse, N. (2008). Redesigning a graphic weather display for pilots. Ergonomics inDesign, 16(4), 11–15.

O’Hare, D., & Stenhouse, N. (2009). Under the weather: An evaluation of different modes ofpresenting meteorological information for pilots. Applied Ergonomics, 40, 688–693.

Paivio, A. (1971). Imagery and verbal processes. London, UK: Holt, Rinehart & Winston.Paivio, A., Rogers, T. B., & Smythe, P. C. (1968). Why are pictures easier to recall than words?

Psychonomic Science, 11, 137–138.Pauley, K., O’Hare, D., & Wiggins, M. (2008). Risk tolerance and pilot involvement in hazardous

events and flight into adverse weather. Journal of Safety Research, 39, 403–411.Prinzo, O. V. (2006). An analysis of preflight weather briefings. In Proceedings of the Human Factors

and Ergonomics Society 50th Annual Meeting (pp. 45–49). Santa Monica, CA: Human Factors andErgonomics Society

Rolfe, J. (1969). Human factors and the display of height information. Applied Ergonomics, 1,16–24.

Sanders, M. S., & McCormick, E. J. (1987). Human factors in engineering and design (6th ed.). NewYork, NY: McGraw-Hill.

Scaife, M., & Rogers, Y. (1996). External cognition: How do graphical representations work?International Journal of Human–Computer Studies, 45, 185–213.

Spirkovska, L., & Lodha, S. K. (2002). AWE: Aviation weather data visualization environment.Computers and Graphics, 26, 169–191.

Tufte, E. (1983). The visual display of quantitative information. Cheshire, CT: Graphics Press.

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014

EFFECTS OF PILOT EXPERIENCE ON RECALL 17

Wiggins, M. W., & O’Hare, D. (1995). Expertise in aeronautical weather-related decision making:Across-sectional analysis of general aviation pilots. Journal of Experimental Psychology: Applied,1, 305–320.

World Meteorological Organization. (2008). Aerodrome reports and forecasts: A user’s handbook tothe codes. Geneva, Switzerland: Author.

Wright, P. (1977). Presenting technical information: A survey of research findings. InstructionalScience, 6, 93–134.

Manuscript first received: October 2009

Dow

nloa

ded

by [

Uni

vers

ity o

f T

enne

ssee

At M

artin

] at

07:

07 0

4 O

ctob

er 2

014