Upload
sdavtyan
View
328
Download
6
Tags:
Embed Size (px)
DESCRIPTION
Insight Data Engineering Project
Citation preview
Seda Davtyan
Informed Traveler
Problem
Datasets• RITA (Research and Innovative Technology
Administration) On Time Performance dataset (October, 1987 till July, 2014)o Updated quarterly
• After unzipping all the files (1 file per year and month) the data is about 65 GB
UI Demo• Informed Traveler
Some InsightJet Blue 01/2012-
01/2013On Time: 67.39%
Carrier: 12.33%
Jet Blue 01/2013-01/2014
On Time: 61.36%
Carrier: 14.81%
Southwest 01/2012-01/2013
On Time: 80.76%
Carrier: 6.41%
Southwest 01/2013-01/2014
On Time: 72.17%
Carrier: 9.56%
American Airline
01/2012-01/2013
On Time: 75.48%
Carrier: 8.00%
American Airline
01/2013-01/2014
On Time: 68.70%
Carrier: 9.74%
Delta 01/2012-01/2013
On Time: 80.67%
Carrier: 5.61%
Delta 01/2013-01/2014
On Time: 81.81%
Carrier: 5.76%
Security + Weather Related Delays < 2%
ScreenshotsJet Blue 01/2013 – 01/2014
South West 01/2013 – 01/2014
AA 01/2013 – 01/2014 Delta 01/2013 – 01/2014
Data Pipeline
Flask API
RI TA
Data Collection
Tradeoffs
Calculate the number of delays of each type
Delay > 15
Carrier
Weather
NAS Security
Late Aircraft
Unclassified
New Field
Tradeoffs• Pig to clean up and transform the data
• UDF from piggybank to handle commas in a .csv file (CSVExcelStorage)
• HBaseStorage to transfer the data into HBase
• Construct composite row keys for fast HBase querying o Airline_Year-Month-Dayo Airline_Year-Montho Airline_Year
Seda Davtyan• PhD in CS&E, 2014
• Love Coffee and Camping• Greatly Enjoy Baking