SCALABLE EVENT TRACKING

by Øistein Sørensen - Schibsted Payment

WHAT IS AN EVENT?

EVENTS

SPiD Core

Events

UDPLogger

file_logger

aws_sqs

Amazon SQS

EVENTS

Amazon SQS

EC2

Auto Scaling

DataPiper

Redshift

Mixpanel

Amazon SQS

UDP LOGGER

DATAPIPER

AMAZON EC2 DEPLOYMENT

EC2 DEPLOYMENT

Auto Scaling

EC2 instances

EC2 DEPLOYMENT

Auto Scaling Group

EC2Ubuntu 12.04

LTSm1.medium

$ bash < User Data

Launch Config

Auto Scaling

EC2 DEPLOYMENT

Public S3 Bucket

EC2Ubuntu 12.04

LTSm1.medium

$ bash < User Data

• S3cmd config

Puppet

S3cmd

EC2 DEPLOYMENT

EC2Ubuntu 12.04

LTSm1.medium

$ bash < User Data

Private S3 Bucket

• Puppet config• DataPiper• Upstart and logrotate

configs

Node.js

npm modules

EC2 DEPLOYMENT

EC2Ubuntu 12.04

LTSm1.medium

DataPipermixpanel

redshift

SQS

SOFTWARE DEPLOYMENT

SOFTWARE DEPLOYMENT

Private S3 Bucket

Upload

SOFTWARE DEPLOYMENT

Auto Scaling

2

SOFTWARE DEPLOYMENT

EC2Ubuntu 12.04

LTSm1.mediumDataPiper

mixpanel

redshift

Auto Scaling

SOFTWARE DEPLOYMENT

Auto Scaling

1

SOFTWARE DEPLOYMENT

EC2Ubuntu 12.04

LTSm1.medium

DataPipermixpanel

redshift

QUESTIONS?