The Big Bad Data

The Big Bad Data

or how to deal with data sources

Przemysław Pastuszka, Kraków, 17.10.2014

● Quick introduction to Ocado● First approach to Big Data and why we

ended up with Bad Data● Making things better - Unified data

architecture● Live demo (if there is time left)

What do I want to talk about?

Ocado intro

Ocado is the world's largest online-only grocery retailer, reaching over 70% of British households, shipping over 150,000 orders a week or 1.1M items a day.

Customer Fulfilment Center

Shop

How did we start?

Google Big Query

OC

AD

O S

ERVI

CES

Oracle

Green plum

JMS

Google Cloud

Storage

Compute cluster

User Cluster

Cluster Manager

Transformed ORC files

Raw data

Looks good. So what’s the problem?

● Various data formats○ json, csv, uncompressed, gzip, blob, nested…○ incremental data, “snapshot” data, deltas○ lots of code to handle all corner cases

● Corrupted data○ corrupted gzips, empty files, invalid content, unexpected

schema changes …○ failures of overnight jobs

● Data exports delayed○ DB downtime, network issues, human error, …○ data available in BD platform even later

Missing data

● Real-time analytics?○ data comes in batches every night○ so forget it

● ORC is not a dream solution○ not a very friendly format○ overnight transform is a next point of failure○ data duplication (raw + ORC)

● People think you “own” the data○ “please, tell me what this data means?”

That’s not all...

● Big Data team is frustrated○ we spend lots of time on monitoring and fixing bugs○ code becomes complex to handle corner cases○ confidence in platform stability rapidly goes down

● Analytics are frustrated○ long latency before data is available for querying○ data is unreliable

People get frustrated

Let’s go back to the board!

It can’t go on like this anymore

What we need?

● Unified input data format○ JSON to the rescue

● Data goes directly from applications to BD platform○ let’s make all external services write to distributed queue

● Data validation and monitoring○ all data incoming into system should be well-described○ validation should happen as early as possible○ alerts on corrupted / missing data must be raised early

● Data must be ready for querying early○ let’s push data to BigQuery first

New architecture overviewIN

PUT

STR

EAM

Event Registry

DataStorage

EventProcessorEvent

ProcessorEventProcessor

Compute Cloud

END

POIN

TS

Cluster Manager

Loading data

validatedeventsstream

INPUT STREAMKinesis

Event Registry

BQ

event type descriptor- schema - processing instructions

Google Cloud Storage

ad-hoc / scheduled

export

raweventsstream

BQ TableauConnector

BQ Excel Connector

BQ REST API

invalid events store location

( BQ )

invalidevents

ad hocevents replay

EventProcessor

GS REST APIgsutil

Batch processing

BQ

Google Cloud Storage

BQ TableauConnector

BQ Excel Connector

BQ REST API

BQ query

export

COMPUTE CLOUD

ComputeCluster A

ComputeCluster B

ComputeCluster C

GS REST APIgsutil

Real-time processing

INPUT STREAMKinesis

Event Registry

BQ

event type descriptor- schema

- processing instructions

Google Cloud Storage

validatedeventsstream

raweventsstream

BQ TableauConnector

BQ Excel Connector

BQ REST API

EVEN

T Q

UEU

E

BQ Sink

Cluster A

Cluster B

Event Processor

GS Sink

processed data readyfor consumption by

other

GS REST APIgsutil

Questions?