Upload
dawson
View
49
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Cross-Platform Aviation Analytics Using Big-Data Integration Methods 2013 Integrated Communications N avigation and Surveillance (ICNS) C onference April 25 , 2013. Dr. Tulinda Larsen Vice President [email protected] Mobile. +1 (443) 510-3566 4833 Rugby Avenue, Suite 301 - PowerPoint PPT Presentation
Citation preview
Dr. Tulinda LarsenVice [email protected]. +1 (443) 510-3566
4833 Rugby Avenue, Suite 301Bethesda, Maryland 20814www.masflight.com
Cross-Platform Aviation Analytics Using Big-Data Integration Methods
2013 Integrated Communications Navigation and Surveillance (ICNS) Conference
April 25, 2013
The Analysis ChallengeT H E A N A L Y S I S C H A L L E N G E
Scale and complexity of aviation data limits research applications
Obtaining radar and airport data, schedules, weather maps and forecasts, fleet information
• Real-time transmission of very large data
• Proprietary and inconsistent formats
• No conditioning or validation
Problems Acquiring Data
Problems Analyzing InformationUsing data for strategic planning and recovery,cost improvement and new market opportunities
• Goes beyond desktop capability
• Time-consuming manual slicing of data
• Need weather and competitor information to answer key operational questions
Fuel and OilConservation
Gate and Terminal Use
Weather Plan & Ops Recovery
Pilot and Crew Staffing
OperationalOptimization
Big-data analytical methods can address these challenges
• Figurative “Cloud” The term comes from the common use of a cloud-shaped symbol as an abstraction for the Internet but application to virtual servers is as recent as 2006
• Cloud computing is the use of resources (hardware and software) that are delivered as a service over the Internet or other network
What is Cloud Computing?
C L O U D C O M P U T I N G
The cloud consists of terrestrial servers across the Internet that collectively store, manage and process data
ApplicationPlatform
Infrastructure
Monitoring
Content
Object Storage
Computation
StorageCollaboration
Communication
Financial Metrics
Content
Identity
Databases
Network
Cloud Computing Architecture
Public Cloud Providers
Community Cloud
Providers
Managed Services Private
Providers
Company Private Clouds
Shared between users
Private Infrastructure
Low Industry Expertise
High Industry Focus
• Cloud computing services can be delivered by an internal IT organization (company-owned private cloud) or
• By an external service provider (managed services private cloud or public cloud provider) or
What are Cloud Architectures?
C L O U D C O M P U T I N G
In aviation cloud resources can be customized and shared among consortiums of customers (community cloud) or
shared with customers in other industries (public cloud)
What is Big-Data Analytics?• The process of examining diverse, large-scale data sets to uncover
patterns, unknown correlations and other useful information
• Organizations have different levels of (1) database management expertise and (2) knowledge to process and analyze big data sets– “Big data” is a relative term based on the user
– Data tables in excess of ten terabytes (10TB) are difficult to work with using most relational database management systems, and particularly using desktop statistics and visualization packages, including Microsoft Excel and Access
• Unstructured data sources in the operational world simply do not fit into desktop or small-scale database structures– They can be hosted using cloud computing at lower cost, and mined
more efficiently, than with on-premises database architectures
B I G - D A T A A N A L Y T I C S
What are Big-Data Analytics Tools?• Big-data analytics employ software tools from advanced analytics
disciplines such as data mining and predictive analytics.
– Mining data, trends or analysis of these multi-terabyte data sets requires parallel software running on tens, hundreds, or even thousands of servers to keep pace with user demands and processing expectations.
• A new class of big-data methods have emerged to address user demands for horizontal scaling and availability of underlying data
– Hadoop and MapReduce, among others, offer fast processing speed.
– Great for large-scale static data sets, but not so great for real-time data
– Most organizations employ a hybrid method combining technologies
• A robust open source framework supports processing in clustered systems.
• Platform-as-a-service vendors (Microsoft, Amazon, Google) offer turn-key solutions for analysts to simply upload, link and compute basic data sets
– Great for simple historical analysis; bad for real-time or diverse data sets
B I G - D A T A A N A L Y T I C S
masFlight: A Global Aviation Data Warehouse and Big-Data Analytics Platform
M A S F L I G H T
Hybrid Architecture
• Physical architecture for secure data feeds
• Cloud-based instances for linking
• Managed cloud data tables
• Integrates with local BI and warehouses
Redundancy
• Multi-source data acquisition
• Real-time validation and processing
• Replication across cloud infrastructure
• Load balancing and parallel processing
Backup
• Cluster processing to reduce dependencies
• Monitored data integrity and performance
• Multiple geographic zones and clusters
• Imaging of tables for replication
Customization
• Customizable for specific user requirements
• Dashboards and web templates
• Integrated internal data in warehouse
• Connect to local BI systems
D A T A A N D A P P L I C A T I O N S
masFlight’s Data and Applications Platform
Data Input Feeds
Airport & Gate Status Multisource, real-time feeds
Current WeatherGlobal hourly conditions
Forecast WeatherStandard and severe forecasts
Flight SchedulesWhat’s planned to operate
Reference and Static DataGeospatial, airline, airport info
Government Economic Data Revenue and audited data
Secure U.S./Canada Radar
Authorized direct accessOther Airspace DataSatellite and transponder info
In-House ServersFor private gov’t feeds
Cloud WarehouseLinked Information
60TB structured data
Robots and Java Applications
Secure External Network
Automated collection
OUR CLOUD-BASED DATA WAREHOUSE OUR CUSTOMER APPLICATIONS
Web Application(masflight.com)
Cloud Managed Database Hosting
Dashboards & Web Services
HTML 5 / RubyAnalyst focusedCustomizableFast deploymentSaaS revenue model
REST web servicesFeed internal systemsCustom dashboardsFlexible interfaces
Virtual tablesUpdated in real timeBypass constraintsUltimate customization
masFlight Platform
M A S F L I G H T P L A T F O R M
Multisource, integrated airline operations data
Our platform shows where, when and why problems occur
• Examine diversions, cancellations, delays and determine root causes
• Deep-dive into airport gates, taxi times, and runway patterns
• Analyze air space usage and air traffic management
Planned Flight Schedules
Multisource Flight Status
Global WeatherData and Maps
AirlineOps Data
U.S. Radar Data
Airline FleetInformation
Airport Gate & Terminal Data
AirportRunway Data
Key Partners and Suppliers:
Origin weatherOrigin informationOperating airlineScheduled timesDeparture gate/timeTaxi-out/takeoff times
Arrival weatherDestination information
Landing/taxi timesArrival gate/time
Diversion dataAircraft information
Flight plan filedActual path flown
CongestionWeather diversions
En-route times and fixes
KIAD V268 SWANN
1502Z 1550Z 1620Z
E N D T O E N D C A P A B I L I T Y
Big-Data Analytics Facilitates End-to-End AnalysisA full picture of each flight is critical for analyzing operations
Query flights from planned schedule through post-operation recoveryUp to 500 data points per flight
Other sources only offer limited, disaggregated and unformatted regional data
A Global Solution
C O V E R A G E
North and South America EMEA and Asia
White lines are flights in the masFlight platform from February 8, 2013.Yellow pins are weather stations feeding hourly data to our platform.
Maps from Google Earth / masFlight
masFlight tracks flights, airports and weather around the world
• Global daily flight information capture ― 82,000 flights― 350 airlines ― 1700 airports
• Integrated weather data for 6,000 stations― Match weather to delays ― Validate block forecasts
at granular level― Add weather analytics to
IRROPS review and scenario planning
Example 1: Proposed FAA Tower ClosuresmasFlight used big-data to link airport operations across three large data sets:
– Current and historical airline schedules
– Raw Aircraft Situation Display to Industry (ASDI) radar data from the FAA
– Enhanced Traffic Management System Counts (ETMS), including Airport operations counts by type (commercial, freight, etc.), departure & arrival
T O W E R C L O S I N G S
Dots indicate closures; Red dots have scheduled service
Based on scheduled service March 1 – 7, 2013; scheduled service includes scheduled charter flights, cargo flights, and passenger flights
Findings: Proposed Tower Closings• From schedules database: 55 airports with
scheduled passenger airline service– 14 EAS Airports
• From ASDI & ETMS: 10,600 weekly flights on a flight plan (ex. VFR and local traffic)
– 6,500 Part 91/125 weekly flights
– 4,100 Part 135/121 weekly flights
Example 1: Big-Data Analytics Applied to ASDI and ETMS To Analyze Operations
T O W E R C L O S I N G S
Up to 5 5-10 10-15 15-20 20-25 25-30 30-35 35-40 40-45 45+
26
44
24 23
11 106
2 1 2
Distribution of Airports By Average Number of “Daily” Impacted Flights
Airports Affected by Tower Closures
Average Number of Daily Operations with a Flight Plan Filed
Cou
nt o
f Airp
orts
Source: ASDI radar data – Part 91/151 flying and Part 135/121 flying – March 1-7, 2013; masFlight analysisNote: Average “daily“ operations based on 5-day week
Example 2: Aviation Safety Causal Factor
For example, consider the following ASRS report (ACN 1031837):
“Departing IAH in a 737-800 at about 17,000 FT, 11 miles behind a 737-900 on the Junction departure over CUZZZ Intersection. Smooth air with wind on the nose bearing 275 degrees at 18 KTS.
We were suddenly in moderate chop which lasted 4 or 5 seconds then stopped and then resumed for another 4 or 5 seconds with a significant amount of right rolling… I selected a max rate climb mode in the
FMC in order to climb above the wake and flight path of the leading -900. We asked ATC for the type ahead of us and reported the wake encounter. The -900 was about 3,300 FT higher than we were.”
• Synopsis– B737-800 First Officer reported wake encounter from preceding B737-900
with resultant roll and moderate chop.
What causal factors can be identified from this narrative thatcould be applied to future predictive applications?
C A U S A L F A C T O R S
Data-mining algorithms can mine the text of safety reports to obtain specific data that can be used to analyze causal factors.
Example 2: Identifying Causal Factors
C A U S A L F A C T O R S
Indicators – Data Element Methods – Identifying Context and Causes
• Time of day
• Date range (month, day)
• Aircraft type
• Fix or coordinates
• Originating airport
• Destination airport
• Weather notes
We pinpoint the sequencing of flights on the IAH Junction Seven departure (at CUZZZ) during the specified wind conditions to find cases where a B737-900 at 20,000 feet precedes by 11 miles a B737-800 at 17,000 feet
• Search related data sets including ASDI(flight tracks, local traffic and congestion)
• Weather conditions for alternative causes (winds aloft, shear and convective activity)
• Airline specific information (repeated occurrence of event in aircraft type)
Big data gives us visibility into contextual factors even if specific data points are missing such as a specific date or route.
Big-data analytics gives us insight into unreported factors as well.
Example 3: Correlating Utilization and Delays
7 8 9 10 11 12 1360%
65%
70%
75%
80%
85%
90%
95%
100%
HOURS OF DAILY UTILIZATION
ON
TIM
E D
EPA
RTU
RE
PER
FOR
MA
NC
E
7.0 8.0 9.0 10.0 11.0 12.0 13.060.0%
70.0%
80.0%
90.0%
100.0%
NarrowbodiesBy Day of Week
7.0 8.0 9.0 10.0 11.0 12.0 13.0 14.060.0%
70.0%
80.0%
90.0%
100.0%
Widebodiesby Day of Week
Daily Utilization vs. On-time DeparturesJanuary 2013 System Operations
Correlation Coefficient -0.53
Includes AA, AC, AS, B6,
F9, FL, NK, UA, US, VX and WN
SOURCE: masFlight (masflight.com)
C O M P A R I N G O T P A N D U T I L I Z A T I O N
LGB
JFK
BOS
MCO
DCA
FLL
6.2
6.0
5.8
5.8
5.2
4.9
JetBlue FocusAverage Daily Deps
per Gate Used
U T I L I Z A T I O N B Y H U B
Example 4: Daily Utilization of Gates, by HubBig-data analysis of different carriers – daily departures per gate used
SOURCE: masFlight (masflight.com)
June 1 through August 31, 2012. Gates with minimum 1x daily use
ORDLAXSFOEWRDENIAHIAD
CLE
7.7 7.4
7.2 6.2 6.1
5.8 3.8
3.6
United Airlines HubsAverage Daily Deps
per Gate Used
SEASANPDXANCSFOGEGLAXSJC
7.8 6.4
5.5 5.4
5.3 4.4 4.3
4.0
Alaska Airlines HubsAverage Daily Deps
per Gate Used
ORD
DFW
LAX
LGA
MIA
JFK
7.2
6.9
6.8
6.4
5.0
2.7
American HubsAverage Daily Deps
per Gate Used
CLT
DCA
PHL
PHX
BOS
7.2
6.9
6.6
4.9
4.2
US Airways HubsAverage Daily Deps
per Gate Used
MCO
BWI
ATL
MKE
6.6
5.9
5.5
4.7
AirTran HubsAverage Daily Deps
per Gate Used
Conclusions for Big Data in Aviation• Big-data transforms operational and commercial problems that were
practically unsolvable using discrete data and on-premises hardware
• Big data offers new insight into existing data by centralizing data acquisition and consolidation in the cloud and mining data sets efficiently
• There is a rich portfolio of information that can feed aviation data analytics
– Flight position, schedules, airport/gate, weather and government data sets offer incredible insight into the underlying causes of aviation inefficiency.
– Excessive size of each set forces analysts to consider cloud based architectures to store, link and mine the underlying information
– When structured, validated and linked, these data sources become significantly more compelling for applied research than they are individually
• Today’s cloud based technologies offer a solution
C O N C L U S I O N S
Conclusions: Our Approach• masFlight’s data warehouse and analysis methods provide a
valuable example for others attempting to solve cloud based analytics of aviation data sets
• masFlight’s hybrid architecture, consolidating secure data feeds in on-premises server installations and feeding structured data into the cloud for distribution, addresses the unique format, security and scale requirements of the industry
• masFlight’s method is well suited for airline performance review, competitive benchmarking, airport operations and schedule design, and has demonstrated value in addressing real-world problems in airline and airport operations as well as government applications
C O N C L U S I O N S