Modeling Your Applications Based on Airport Operations

YYZ

Modeling Your Applications Based on Airport Operations

Yes, even the TSA Part

CDG

Introduction Bill Klosbill.klos@centricconsulting.com@williamklos

My day job…

But my hobby is…

❖ ATL

❖ AUS

❖ BOS

❖ CDG

❖ CLE

❖ CLT

❖ CMH

❖ CVG

❖ DAL

❖ DCA

❖ DEN

❖ DFW

❖ DTW

❖ DUB

❖ EWR

❖ FLL

❖ FPO

❖ HOU

❖ HPN

❖ IAD

❖ IAH

❖ JFK

❖ LAS

❖ LAX

❖ LGA

❖ LHR

❖ LUK

❖ MIA

❖ NAS

❖ NRT

❖ ORD

❖ ORL

❖ ONT

❖ PDX

❖ PHL

❖ PHX

❖ PSP

❖ PUR

❖ PVD

❖ SAN

❖ SFO

❖ SLC

❖ SMF

❖ SIN

❖ STL

❖ YHZ

❖ YYZ

Which gives me time to…

…and stand in line.

And of course complain.

Yet…

80%

On-Time

6.48% - Aviation System

6.87% - Late Aircraft

5.09% - Maintenance/Crew

0.34% - Weather

0.02% - Security

Waiting on me was never listed as an issue.

145,200 pieces per day

416 mishaps

0.287%

Mishandled Bag Rate

So while most of our experiences at airports contain stories of horror…

…they all work pretty much the same.

Airports must be doing something right.

Disperate, detached operations, all working together. Albeit loosely.

What lessons can I steal from airport operations and apply to system design?

7 Patterns

ORD

Airport Queue Patterns

Moving data through a system like people through an airport.

Typical Airport Day :)

❖ Get Your Boarding Pass

❖ Check Your Bag

❖ Get Routed to a Security Checkpoint

❖ Get ID Checked

❖ Get Scanned

❖ Get Food

❖ Jockey for Boarding Position

❖ Line Up for Take Off

❖ Line Up for Landing

❖ Jockey for Deplaning

❖ Retrieve Bag

Typical Airport Day :(

❖ Get Frisked/Searched

❖ Wait in Customs

❖ Retrieve Bag at Entry Point

❖ Get Scanned Again

❖ Re-Check Bag

❖ Repeat Boarding Process

Most of these are covered in the patterns we will discuss.

1 Queue/Queue Manager

aka, “The Tag Team”

Queue Worker

Capacity Queued Jobs

Typical Airport Scenario

Current Hell

Manager

Worker

Previous Hell

Next Hell

WorkerWorker

Typical Queue Scenario

Feeder

Queue Worker

Queue Manager

Queue Worker

Queue Worker

PreviousFeeder

NextFeeder

Capacity Queued Jobs

Queue Worker

2 Infinite Scalability

aka, “Open Up Another Damned Line!”

TSA

Normal Worker

TSA Manager

Pre-Check Worker

Frisk Worker

Gate

Physical Capacity

Resource Capacity

We can mimic the physical aspect of the airport model by adding more queues and corresponding workers…

Feeder

Queue Worker

Next Feeder

- Adding more queues with a corresponding worker can work.- Must be able to carve off work from the overloaded queue.- Work units need to be discrete.- QM needs to also balance the load.

Previous Feeder

Queue Worker

Queue Worker

Queue Manager

Personally though, I would rather do this…

Feeder

Queue Worker

Next Feeder

- Physical capacity is irrelevant (mostly).- Resource capacity is virtually infinite.- Manager determines how many workers to get to green. Resource

Capacity

Previous Feeder

Queue Manager

Simple Queue Manager

Scaling Formula

Increase scale

((Qd/Qw)/t) >= tmx

When the calculated time it takes for all the workers to complete all of the jobs is greater than or equal to the maximum allowable time for any job to complete this step - make more workers.

But only up to some maximum. Accidental $1500 cloud bills teach you a thing or two.

Decrease scale

((Qd/Qw)/t) < tmx

When the calculated time it takes for all the workers to complete all of the jobs is less than the maximum allowable time for any job to complete this step, you’re wasting resources. Time to trim the fat!

But we need to keep one lackey around to do the menial tasks.

3 Standard Prioritization #1

aka “The Haves”

1 1 11222334

You

Cool Kids

1 Queue Worker

Aside from Re-Insertions, which we will discuss later, I have not seen this in an airport environment.

Which is good, lest…

3a Standard Prioritization #2

aka “How the ‘Other Haves’ Live”

???

TSA

Normal Worker

TSA Manager

Pre-Check Worker

Frisk Worker

Ticketing

Gate

Physical Capacity

Resource Capacity

Courtesy Sorting

Random Inputs

Feeder

Queue Worker15

Previous Feeder

Next Feeder

10 3 1111214

7 6 2131621

14

4

2

5

1

11

21

3

8

15

17

16

910

12

18

1920

13

6

7

8 5 4917181920

Queue Manager

Queue Worker

Queue Worker

Queue Worker

Processes before anything else.

Processes only if nothing in queue above.

Processes only if nothing in queue above.

Processes only if nothing in queue above.

Depending on the timing of how jobs arrive in the queue,#15 may get processed, 1st, 2nd, etc, but no later than 15th.Where as #2 may not process before 8th.

4 Arbitrary Prioritization

aka, “Every Man for Himself”

Gate Area

AA Gate

10 Gate Worker12 151311

Gate Agent

141662

Security

Aircraft

21 13 184587191720

9

14

4

2

5

1

11

21

3

8

15

17

16

910

12

18

1920

13

6

71st Class

Diamond/Platinum

Windows

Aisles

- Due to the constrained nature of aircraft access, loading more passengers faster is not possible.- To a certain extent the passenger can control their access order (mob surfing).

Gate Area

SW Gate

2 Gate Worker5 7134

Gate Agent

61489

Security

Aircraft

10 15 1312111920161721

18

1

4

11

7

10

16

12

5

13

15

8 9

1821

17

14

1920

3

6

2

A

C

B

- Due to the constrained nature of aircraft access, loading more passengers faster is not possible.- To a certain extent the passenger can control their access order.

Random Inputs

Feeder

Queue Worker

15

Previous Feeder

Next Feeder

10 3 1111214

7 6 2131621

14

4

2

5

1

11

21

3

8

15

17

16

910

12

18

1920

13

6

7

8 5 4917181920

Queue Manager

Processes before anything else.

Processes only if nothing in queue above.

Processes only if nothing in queue above.

Processes only if nothing in queue above.

Depending on the timing of how jobs arrive in the queue,#15 may get processed, 1st, 2nd, etc, but no later than 15th.Where as #2 may not process before 8th.

5 Re-Insertion

aka, “Hey, can you let me cut? I had a thing.”

Ticketing

Ticket Manager

Ticket Worker

Baggage Worker

Security

Problem Worker

Kiosk Worker

The threshold of a green vs. yellow vs. red status is different for each queue type due to the average length of a transaction for that type.

Feeder

Queue Manager

Queue Worker

Problem Worker

Next Feeder

Notification Worker

In this scenario, when a Queue Worker encounters an error, it can:

- Attempt to fix the issue and re-queue;- Assign it to a problem queue to be handled via another process;- Notify the interested user.

The Problem Worker can fix the issue then complete processing or:

- Return it to the the main queue for normal processing;- Notify the interested user.

Previous Feeder

Feeder

Queue Worker1

Next Feeder

3

Queue Worker

Resolution Queue

Problem Queue

Instead of being moved to the back of the “Normal” line,problem jobs with resolutions get expedited treatment.

6 5 4789

Normal Queue

2143

Queue Manager

Queue Worker

6 Throughput Measurement

aka, “Metric Mule”

Measuring❖ Where to Measure

❖ At the Beginning of the Process

❖ At Important Junctions in the Process

❖ At the End of the Process

❖ What to Measure

❖ Time

❖ # of Transactions

❖ Size of Payload

❖ Frequency of Certain Transaction or Data Types

???

TSA

Normal Worker

TSA Manager

Pre-Check Worker

Frisk Worker

Ticketing

Gate

Physical Capacity

Resource Capacity

Courtesy Sorting

XX

XX

XX

- Mark time, distribute a card to passenger.- Retrieve card from passenger, mark card.- Profit!

X

Feeder

Queue Worker

Next Feeder

- Physical capacity is irrelevant (mostly).- There is no “unknown” or holding pen.- Resource capacity is virtually infinite.- Priority is set by business rules.- Manager determines how many workers to get to green.

Resource Capacity

Previous Feeder

Queue Worker

Queue Worker

QM applies formula to get to green.

Queue Manager

XXXXXXXXX

XXXXX

X- Mark time at queue entry.- Mark time at queue exit.- Adjust worker create/destroy thresholds based on latest data.

{

"qmsg":

{

"id":"log::97e3fa40-1aa6-49cb-8a50-d8b067b98898",

"company_id":"cmp::cc349da1-cf57-49b6-b1b3-4a8af9d52388",

"action":"log:api",

"objtype":"QAPILog",

"entered":"2014-09-12T06:18:19.169408997Z",

"processed":"0001-01-01T00:00:00Z"

},

"method":"PUT",

"remote_ip":"64.134.173.96",

"route":"/guest/graham.savage/location/74C9BFCF-4698-4D83-B867-77407C075C33/proximity/3",

"agent":"Grand%20Lucayan/1.0 CFNetwork/672.1.15 Darwin/14.0.0",

"status":404,

"size":96,

"ms":0.354969

}

Using the timing data…

((Qd/Qw)/t) < tmx

((Qd/Qw)/t) >= tmx

7 Begetting

aka, “You Had One Job!”

OPS

Passenger Worker

Baggage Worker

Security Worker

Flight Worker

Ticket Worker

The act of getting a boarding pass starts a process in multiple areas simultaneously.

Logging Worker

Error Worker

Metrics Worker

Notification Worker

Transaction Worker

Each transaction starts a chain reaction which could lead to sub jobs and a plethora of other messages.

So 1 transaction can spawn a dozen additional operations that will travel to their respective queues.

1. Queue/Queue Manager “The Tag Team”

2. Infinite Scalability “Open Up Another Damned Line!”

3. Standard Prioritization #1 “The Haves”

3a. Standard Prioritization #2 “How the ‘Other Haves’ Live”

4. Arbitrary Prioritization “Every Man for Himself”

5. Re-Insertion “Hey, can you let me cut? I had a thing.”

6. Throughput Measurement “Metric Mule”

7. Begetting “You Had One Job!”

DFW

Q&ABill KlosSenior ArchitectCentric Consultingbill.klos@centricconsulting.com@williamklos