Embed Size (px)
Citation preview
Easy and efficient
processing of massive data
sets
All non geeks, please get a coffee, this is going to be
painful!
SMS Campaigns logging system
PROCESSING LOGSSMS
SMS Campaigns logging system
PROCESSING LOGSSMS
Huge SMS Queueto much processing and logging time
Use Timers for Asynchronous Processing
RAW PROCESSING LOGSSMS
RAW PROCESSING LOGSSMS
Timer times out!
Use Timers for Asynchronous Processing
Best practices for building timers
• Never times out
• Doesn’t repeat work
• Ensures completeness
• Ensures data integrity
Best practices for building timers
RAW PROCESSING LOGSSMS
Best practices for building timers
RAW PROCESSING LOGSSMS
RAW
PROCESSING
Add Email campaigns logging
RAW PROCESSING LOGSSMS
RAW
PROCESSING
Database performance degradation and timeouts due to locks
Add Email campaigns logging
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
Sequence Timers
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
3
Add Website logging
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
3
Add Website logging
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
BUFFER MOVE
3
Buffer
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
BUFFER MOVE
3
Long peaks in SMS Campaigns caused starvation for Email and WebSite logs processing
Buffer
Round Robin
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
3
RAW
PROCESSING
BUFFER MOVE
1 2
3
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
BUFFER MOVE
3
EXTERNALSYSTEM
COLLECT RELEVANT DATA
31 2 4
4
Collect data to other systems
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
BUFFER MOVE
3
EXTERNALSYSTEM
COLLECT RELEVANT DATA
31 2 4
4
Too heavy to get the records to collect.Available time slot not enough.
Collect data to other systems
Queues
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
BUFFER MOVE
3
EXTERNALSYSTEM
COLLECT RELEVANT DATA
31 2 4
4
QUEUE
Queue entity to store the IDs to be processed.
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
BUFFER MOVE
3
EXTERNALSYSTEM
COLLECT RELEVANT DATA
31 2 4
4
QUEUE
Enqueue is done in the processing timers through a light check.
Queues
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
BUFFER MOVE
3
EXTERNALSYSTEM
COLLECT RELEVANT DATA
31 2 4
4
QUEUE
Dequeue next.
Queues
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
RAW
PROCESSING
BUFFER MOVE
3
EXTERNALSYSTEM
COLLECT RELEVANT DATA
31 2 4
4
QUEUE
Gets one record by ID instead of doing a heavy query to get a list of records
Queues
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
3
RAW
PROCESSING
BUFFER MOVE
1 2
3
EXTERNALSYSTEM
QUEUE
COLLECT RELEVANT DATA
No need to be included in the round robin with the processing timers
Queues
Queues – timer to process logs
Queues – Enqueue inside DoSomething
Queues – Timer to process queue
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
3
RAW
PROCESSING
BUFFER MOVE
1 2
3
EXTERNALSYSTEM
QUEUE
COLLECT RELEVANT DATA
Queues
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
3
RAW
PROCESSING
BUFFER MOVE
1 2
3
EXTERNALSYSTEM
QUEUE
COLLECT RELEVANT DATAProgressively slower!
Queues
Archive
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
3
RAW
PROCESSING
BUFFER MOVE
1 2
3
EXTERNALSYSTEM
QUEUE
COLLECT RELEVANT DATA
ARCHIVEMOVE
OTHER CATALOG
Multiple database catalogs
RAW PROCESSING LOGSSMS
RAW
PROCESSING
1
2
3
RAW
PROCESSING
BUFFER MOVE
1 2
3
EXTERNALSYSTEM
QUEUE
COLLECT RELEVANT DATA
ARCHIVEMOVE
A system that deals with massive amounts of data is
never finished…
