Flight Data Handling with Augmented Reality
Doctoral Symposium
ICRAT ´18, Castelldefels, Barcelona (Catalonia)
June 25th – 29th 2018
> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29 DLR.de/fl • Chart 1
Hejar Gürlük DLR Institute of Flight Guidance Controller Assistance
www.DLR.de/fl • Chart 2 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Background
Scope of Thesis
Concept
Summary & Outlook
Study
Safety
Highest incidents and accident rates are related to “Runway” (62%) in contrast to “in flight” (3%) (ICAO Safety Report 2015)
Tower: Errors relate mainly to the field of visual perception major contributing factor for runway incursions (Hilburn, 2004)
0
10
20
30
40
50
Perception Memory DecisionMaking
ResponseExecution
Rule Breaking
Error types related to tower environment
60%
6%
3%
62%
Loss of Control in Flight
Runway
Accidents per flight phase
Accidents Fatalities
www.DLR.de/fl • Chart 3 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
www.DLR.de/fl • Chart 4
„Head Down vs. Head Up Problem“
Head Down (HD): • Information systems and devices; frequent
usage but scan times not as long as head up (Pinska, 2009)
Head Up (HU): • Outside view
• Crucial for maintaining adequate situational
awareness (Hilburn, 2004)
• Remains most important source of information (Ruffner, 2008)
Tower Controller Working Position
Head Up
Problem: frequent transitions from HU to HD: “…delayed detection and interpretation of visual information…major error source for runway incursions“ (Hilburn, 2004)
Head Down
> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
www.DLR.de/fl • Chart 5
Information Complexity
• Widely distributed head-down displays
• Controllers have to mentally merge the different information sources (HD / HU)
• Increasing information complexity (System Wide Information Management)
This can lead to higher workload in terms of
prolonged times for information acquisition and analysis
Tower Controller Working Position
> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Low Visibility Conditions
decreased capacity (throughput) and increased delay at airports operating under CAT II/IIIA-C
www.DLR.de/fl • Chart 6 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
www.DLR.de/fl • Chart 7 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Background
Scope of Thesis
Concept
Summary & Outlook
Study
Solution Approach and Aim of the Thesis
• Projection of relevant flight and topological data into the real outside
view by means of augmented reality
• prolongation of head-up times,
• improved situational awareness,
• reduced information search
• maintain capacity (even under low visibility conditions)
safety benefits
• Context-adaptive information presentation: user and task-dependent
presentation of only operationally relevant information
• What to present? When to present? How to present?
• reduction of controller workload (information acquisition and -analysis)
www.DLR.de/fl • Chart 8
Motto and Aim: display the right (amount of) information at the right time!
> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
www.DLR.de/fl • Chart 9
Augmented Reality
Issues: - overclutter - untailored information presentation - missing interaction or feedback - integration of AR into operations unclear - display ergonomics, perceptual issues, registration errors
> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Research gap: • operational concept for such an assistance
system (e.g. for different operational conditions) -> HOW TO WORK WITH THIS?
• systematic assessment of operational and human performance benefits
overclutter
Main Research Question of Phd Thesis
What are the human performance and operational
benefits of context-adaptive augmented reality
for air traffic control towers?
www.DLR.de/fl • Chart 10 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Scope of Phd Thesis (I) Development, Implementation and Validation of Operational Concept
1st Concept study „adaptive-augmented outside view“ (2016)
Workplace Analyses & Model Development (2015)
2nd Concept study „integrated information management“ (2017)
www.DLR.de/fl • Chart 11 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Scope of Phd Thesis (II)
3rd Study „low visibility“ and concept transfer on Hololens (2018)
„AR Attention Guidance“ (09/2018)
…work in progress…
www.DLR.de/fl • Chart 12 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Validation of Operational Concept (End 2018)
…work in progress…
www.DLR.de/fl • Chart 13 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Background
Method
Concept
Summary & Outlook
Study
Concept in a nutshell
Adaptive Information Management
www.DLR.de/fl • Chart 14
presentation of operat. relevant information
reduction of workload and displayclutter
enhanced
situational awareness
Increase capacity and safety under IMC?
> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Direct interaction
Context
Flight Plan Data
Surveillance Data
Environmental Data
Assistance System
Data
Integrated Information Management
indirect interaction
Ausblick
Background
Method
Concept
Outlook & Summary
Study
www.DLR.de/fl • Chart 15 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Research Aim
Experimental Setup
Results
Conclusion
2nd Concept Study „Flight Data Handling with
Augmented Reality“
LTU7HJ
www.DLR.de/fl • Chart 16 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
DLR 360° Apron and Tower Simulator
Research objectives
• Evaluation the current state of the
operational concept
• Identify most promising CWP configuration
• Test the effect of augmented reality based
flight data handling on human performance
www.DLR.de/fl • Chart 17 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Research hypotheses
H1 “An augmented outside view with integrated information management
(H1.1) leads to significantly longer head up times
(H1.2) yields improved situational awareness
(H1.3) yields lower workload
compared to a conventional tower controller working position”
www.DLR.de/fl • Chart 18 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
H2 “Direct interaction in the augmented outside view yields
(H2.1) higher situational awareness
(H2.2) lower workload
compared to indirect interaction”
Experimental Setup
[1] TFDPS (tower flight data processing system) [2] Weather Display [3] Air Situation Data Display [4] Ground Radar Display
www.DLR.de/fl • Chart 19 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Configuration C4 „context-adaptive augmented outside view with both interaction possibilities“
Configurations
Configuration Description Technical Setup
C1
“conventional CWP”
experimental tower controller working position (no augmented outside view)
C2
“indirect interaction”
C1 + augmented outside view + coupling with TFDPS
C3
“direct interaction”
C1 + augmented outside view + coupling with 3D “iDrive” Controller. Data entry via context-menu. No TFDPS available!
C4
“both interaction methods”
C3 + augmented outside view + TFDPS: data entry both via TFDPS and 3D controller possible
Configuration C4 „context-adaptive augmented outside view
with both interaction possibilities“
www.DLR.de/fl • Chart 20 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Eyetracking: Areas of Interests
www.DLR.de/fl • Chart 21 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Wearable eye tracking glasses SMI ™ Eye Tracking Analyzer (DLR)
Scenarios and Experimental Procedure
• Hamburg Local Control • IFR Traffic (just few VFR flights) • Arrivals only on RWY 23, Departures from RWY 33 • only VMC • Traffic volume (above medium, rather high) • R/T with 2 Pseudopilots • no telephone coordination with other ATC units
Part I: • 4 simulation runs á 45 minutes (ca. 48 IFR, 2 VFR
flights) • Eyetracking recording during simulation runs • Post-Run Questionnaires & Debriefings
Part II: • Workshop
Test persons (n=4): 3 DFS Aerodrome Controller, 1 Military Aerodrome Controller
www.DLR.de/fl • Chart 22 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Results: Eyetracking
34,5%
50,8%
58,2%
Standard Errors for Head-Up: [C1]: ±1.60 [C2]: ± 4.37 [C3]: ± 3.81 [C4]: ± 5.16 Standard Errors for Head-Down: [C1]: ±2.29 [C2]: ± 2.69 [C3]: ± 1.34 [C4]: ± 2.46
www.DLR.de/fl • Chart 23 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
C1 C2 C3 C4
Me
an D
we
ll Ti
me
s (i
n %
)
Controller Working Positions
Head-Up
Head-Down
89.7 68.1 43.0 57.7
57.0 42.3
10.3
31.9
Results: Situation Awareness (3D SART)
www.DLR.de/fl • Chart 24 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
34,5%
50,8%
58,2%
C1 C2 C3 C4
Mean SA 51,3 60,3 22,3 72,0
0,0
10,0
20,0
30,0
40,0
50,0
60,0
70,0
80,0
90,0
100,0
3D
SA
RT
Sco
re
Mean Situational Awareness
51.3
60.3
22.3
72.0
Results: NASA-TLX Workload Scores
> Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
34,5%
50,8%
58,2%
C1 C2 C3 C4
Overall Workload (all subjects) 8,0 7,6 10,6 6,6
0,0
2,0
4,0
6,0
8,0
10,0
12,0
14,0
16,0
18,0
20,0O
vera
ll W
ork
lad
Sco
re
NASA TLX (Weighted)
8.0
7.6
10.6
www.DLR.de/fl • Chart 25
6.6
www.DLR.de/fl • Chart 26 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Background
Method
Concept
Conclusion & Outlook
Study
Conclusion (I)
www.DLR.de/fl • Chart 27 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
• configuration C4 showed mean dwell times for head up increased more than 25% compared to a conventional tower CWP (hypothesis H1.1)
• configuration C4 promotes the situation awareness at most (hypothesis H1.2) • configuration C4 NASA TLX workload ratings were the lowest when compared
to the other CWP configurations (hypothesis H1.3) the results support the hypothesis 1 (H1.1, H1.2 and H1.3) almost entirely
In general, configuration C4 was rated as the most beneficial and best-suited CWP for an augmented reality based flight data handling
Conclusion (II)
www.DLR.de/fl • Chart 28 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
• Despite the high dwell times for head up configuration C3 (89.3%), direct interaction in the augmented outside view did neither automatically yield higher situational awareness (hypothesis H2.1) than C2 (indirect interaction) (actually the opposite was the case! C3 obtained the lowest S.A. ratings)
• C3 yielded highest workload scores (hypothesis H2.2)
• possibly due to …. absence of TFDPS (important for flight planning) …. fairly high traffic volume in the scenarios, …. insufficient training, unusual working method, …. limitations related to implementation and non- ergonomic handling Thus, hypothesis H2.1 and H2.2 could not be confirmed!
Is “Head Up Only” a desirable approach?
Summary
• Preliminary results although of descriptive nature are promising
• Combined direct/indirect Interaction with AR flight data is the most beneficial and best-suited configuration for an augmented reality based flight data handling
• Substantial increase of head up time when working with direct/indirect Interaction with AR flight data could be a strong indicator for improved situation awareness
But things are not that simple, with a look at configuration C3 There must be some sweet spot for head up
• migration tolerant system development: at early stages of system development,
integration of AR within the operational environment should be done carefully in small steps (thus existing ATC systems such as TFDPS should be incorporated within new introduced assistance systems. At least in the beginning…)
www.DLR.de/fl • Chart 29 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
Outlook
• Future work will focus on the enhancement of the interaction concept
• Evaluate the suitability of the operational concept under all-weather operations (3rd study)
• Final Validation Campaign currently under preparation sufficiently large sample size (n=15 ATCOs) will be determined in order to apply inferential statistics and test for significant effects
www.DLR.de/fl • Chart 30 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
… and to have fun and to keep me busy with lots of data collecting and analyzing…
Take Home Message
Results encourage a paradigm shift for flight data handling with augmented reality:
away from a display system to an operating system!
www.DLR.de/fl • Chart 31 > Flight Data Handling with Augmented Reality > Hejar Gürlük > ICRAT 2018 > 2018/06/29
www.DLR.de/fl • Folie 32 CONTACT Hejar Gürlük [email protected] +49 531 295 2591 German Aerospace Center (DLR) Institute of Flight Guidance Controller Assistance
Thank you very much for your attention! That‘s all!
Thank you!