View
212
Download
0
Embed Size (px)
Citation preview
• The forecasters understood and adjusted for model uncertainty.
Models are not always accurate due to lack of observations and inadequacies of model physics.
• The forecasters estimated uncertainty at 2 stages:
BackgroundWeather forecasters examine current weather conditions upstream of the forecast area. They transform these conditions, according to the principles of atmospheric physics, into the future weather in the forecast area.
Satellite Image
MM5: Numerical Model
Previous research suggests the steps in the forecasting process are:
1. Initialize understanding of synoptic (large) scale weather
2. Build and consolidate a qualitative mental model (QMM) of the local weather
(Trafton, Kirschenbaum, Tsui, Miyamoto, Ballas & Raymond, 2000)
– From quantitative information– Cause & effect relationships of elements in atmosphere
3. Verify and adjust QMM (Pilske, Klinger, Hutton, Crandall, Knight, Klein 1997, Hoffman, 1991)
– Check against other information sources (e.g. numerical models)– Much of information gathering at this stage
4. Project forward to the time of forecast
RESULTS
There are special challenges involved in naval aviation forecasting.
PROCEDURE: Cognitive task analysis of navy forecasters• Collected think aloud verbal protocols. Subjects verbalized thoughts while completing the task.
(Ericsson & Simon, 1984)
• Recorded 4 forecasters’ verbalizations and computer screen images as they each produced a forecast, in February, 2002.
• Transcribed audio recordings. Broke down into statements and coded transcripts for information source (e.g. satellite, numerical models, radar imagery).
• We suggest that these replace building and consolidating a qualitative mental model.
Advantages: Reduces working memory load. It is a practical approach for individuals who are making rapid decisions amid constant interruptions.
Disadvantages: When there is an insufficient match between present situation and “if” condition, adjustment is difficult without the benefit of an integrated mental model.
Question: What value do the forecasters add to numerical model predictions?PROCEDURE: linear regression analysis
- Dependent variable = observed wind- Predictors = Navy forecasters predictions;
Numerical Model predictions
RESULTS:Wind Speed• Correlation between Navy Forecasters’ prediction and the observed wind speed was greater than the correlation between numerical models and observed wind speed.
Conclusions• Navy forecasters have a streamlined forecasting process.
- Use few information sources - Rely on rules of thumb
• Navy forecasters understand model dynamics.
- Evaluate model uncertainty- Use several standard evaluation techniques - Adjust the model prediction
• Navy forecasters add value to numerical model predictions.
Question: What is the impact of time pressure and information overload on weather forecasting?
RESULTS (cont.)
•The forecasters relied heavily on numerical modelsQuestion: What is the forecasters’ understanding of model dynamics?
ReferencesEriccson KA, and Simon HA. 1984. Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT
Press.
Hoffman RR. 1991. Human factors psychology in the support of forecasting: The design of advanced meteorological workstations. Weather and Forecasting 6: 98-110.
Pilske R, Klinger D, Hutton R, Crandell B, Knight B, Klein G. 1997. Understanding skilled weather forecasting: Implications for training and design of forecasting tools. Technical Report Al/HR-Cr-1997-003.
Trafton GJ, Kirschenbaum SS, Tsui TL, Miyamoto RT, Ballas JA, Raymond PD. 2000. Turning pictures into numbers: extracting and generating information from complex visualizations. International Journal of Human-Computer Studies 53: 827-850.
Wind Direction• The human forecaster outperformed the numerical model.
Naturalistic Decision-Making: A Cognitive Task Analysis of Naval Weather ForecastingSusan Joslyn, Karla Schweitzer & Earl Hunt
University of Washington
• The variance in actual wind speed accounted for by the human forecast subsumed that accounted for by the numerical model.
Multiple R= .68
.56
.68
.39Human Forecast
ActualNumerical Model
• Statements were coded qualitative or quantitative.
• The forecasters’ mental representation was predominantly qualitative, with the exception of pressure.
Fact statements by parameter: percent qualitative versus percent quantitative
Qualitative
Quantitative
Numerical Model alone=.15
Human Forecast=.46
Observed=1
Forecaster A Forecaster B Forecaster C Forecaster D
Synoptic Scale Analysis
40%
Synoptic Scale Analysis
8%
Synoptic Scale Analysis
50%
Synoptic Scale Analysis
16%
Estimate Specific
Parameters
52%
Estimate Specific
Parameters
65%
Estimate Specific
Parameters
40%
Estimate Specific
Parameters
6%
Write Forecast
9%
Write Forecast
3%
Write Forecast
11%
Write Forecast
78%
Consolidate Mental Model
0%
Check Mental Model
38%
Goal Structure for each forecaster:
Percent of source statements under each goal
• Most of the forecasters used rule of thumb (if-then rule) reasoning.
“If I see a system coming into the coast, I forecast strong southerly winds over Whidbey Island.”
“These systems aren’t normal… it's going to act a little bit different. ” (Forecaster A)
• Forecasts must be posted every six hours while completing other tasks.
• Forecasters are interrupted as many as 12 times while writing a single forecast.
• Forecasters learn about current weather conditions from:
- Surface Charts - Satellite Imagery
- Radar Imagery • Forecasters predict future weather using current observations and numerical models:
- Numerical models are computer programs that make weather predictions which are fairly accurate for large-scale phenomena over a short period of time.
Navy forecasting office
271
2623
108
7
63
Regional Forecasts
Weather Discussion
Surface Observations
Satellite
Radar
Numerical Models
Total number of statements by source
0%
10%
20%
30%
40%
50%
60%
70%
% Closer toActual
% Correct-Variable
Human Forecast versus Numerical Model: Forecast wind direction or variable winds
Direction: % Variable: % closer correct
Human Forecast
Numerical Model
Both/Tie
Numerical Model: MM5
Satellite
Evaluation involves comparing patterns (e.g. position of low) in numerical models and satellite image (actual weather).
Is the model doing a good job?
• All of the forecasters started with the synoptic scale analysis• Most of the forecasters omitted the step of consolidating and verifying the qualitative mental model
- Only Forecaster B (the most experienced forecaster) conformed to the predicted pattern
• Most of the forecasters gathered information in the middle stages of the process
- Surprisingly, Forecaster D gathered most of his information after he began writing his forecast- A strategy to avoid maintaining large quantities of information amid constant interruptions?
This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745
Estimating specific parameter values: 41% of model uncertainty statements involved checking the specific values.
Synoptic Scale Analysis: 59% of statements regarding model uncertainty were made as forecasters examined the evolution of large-scale weather patterns over time.
Example: Forecaster D compared the observed pressure to what the model had predicted.
Access current pressure 29.69
Calculate difference 29.69-29.64=.05 between current and
forecast (error)
Access predicted 29.57 pressure for forecast time
Adjust predicted 29.57-.05=29.52 pressure based on current error
Adjust predicted 29.52+.02=29.54 pressure based on model bias