Infrastructure for Decision Makers

INFRASTRUCTURE FOR DECISION MAKERSQuestions for a better architecture

Eric Lubow @elubow #ddsea15

PERSONAL VANITY

๏ CTO of SimpleReach

๏ Co-Author of Practical Cassandra

๏ Skydiver, Mixed Martial Artist, Motorcyclist, Dog Dad (IG: @charliedognyc), NY Giants fan

SIMPLEREACH

๏ Identify the best content

๏ Use engagement metrics

๏ Stream processing ingest

๏ Many metrics, time sliced

๏ Multiple data stores

What do you mean infrastructure?

๏ Architects

๏ CTOs

๏ Lead Developers

๏ Developers

๏ Basically everyone

WHO IS MAKING THESE DECISIONS?

YOU WOULDN’T BUILD SOFTWARE WITHOUT PLANNING

FIRST, SO WHY WOULD YOU BUILD AN ARCHITECTURE

WITHOUT PLANNING?

๏ Architectures get built ad hoc

๏ Pieces tend to be built as needed and not always thought out

๏ Many lead developers don’t have a lot of architecture experience

๏ We don’t live in a perfect world and are usually time bound

๏ Product needs to be built and we’ll figure out the rest later (technical debt)

REALITY OF THE SITUATION

What are we actually going to talk about today?

๏ Hardware

๏ Cloud

๏ Databases

๏ Message Systems

๏ Scale/Scaling

๏ Costs

๏ Compliance

๏ Development ease

๏ Authentication

FRAMEWORK FOR BUILDING

๏ Developer / Operational Capabilities

๏ Available Support

๏ Monitoring / Instrumentation

๏ Testing / Staging / QA

๏ Repeatability of Systems

๏ Safety nets

๏ Pressure valves

๏ Administration ease

๏ Authorization

WHY SHOULDN’T I LEAVE RIGHT NOW

๏ Unsexy talks can have good information

๏ Understanding these concepts can save lots of technical debt

๏ There are lessons learned from not knowing which to ask questions

๏ I’m kind of entertaining

๏ In case I’m not entertaining, I’ll use some entertaining pictures

๏ I’m going to tell you a story

REASONS TO LISTEN

HOW DID SIMPLEREACH GET FROM …

TO …

Business/Application/Translation/Data Access

Router/Load Balancer/Config/Authentication

SERVICE SERVICE SERVICE SERVICE

Redshift

Platform

๏ Allows people to use a common language when discussing or solving problems

๏ Allows a common toolset for solving problems

๏ Simplifies difficult tasks

๏ Every language has frameworks: Ruby/Rails, Python/Django, Javascript/Ember.js

๏ Attempts to answer the questions:

๏ How should I do this?

๏ Is this a good idea?

๏ Is this the right tool?

WHY ARE FRAMEWORKS IMPORTANT

๏ Where is this going to live?

๏ How do I get data in?

๏ How am I going to store the data?

๏ How do I move data around?

BASIC QUESTIONS

๏ How should data look coming out?

๏ How do I get data out?

๏ How do I know if something is wrong?

๏ How do I maintain/scale/build?

๏ Is this going on the cloud? Amazon, Google, Azure, Rackspace?

๏ Do you need to be in a data center?

๏ Are APIs important?

๏ What kind of distribution of services / fault tolerance needs to be

available?

๏ What kind of SLAs do you need to meet (100% uptime)?

WHERE IS THIS GOING TO LIVE?

HOW DO I GET DATA IN?

๏ Build apps that follow the same paradigm

๏ POST data to an end point

๏ Consume off a queue

๏ Use message systems for queueing

๏ Message aggregation for efficiency

๏ Message sampling for throttles

๏ Try to avoid talking directly to a database from client facing applications

๏ Write your own client driver to talk to your architecture

HOW AM I GOING TO STORE THE DATA?

๏ What’s the latest cool technology?

CHOOSING A DATABASE IS EASY, #AMIRITE

๏ What is my data volume?

๏ What are my query patterns?

๏ Is my data (un)structured?

๏ Will data remain consistent?

๏ Am I read heavy or write heavy?

๏ Am I batch loading data?

๏ Is eventually consistent data ok?

๏ Can I have a DR plan?

๏ Legal/compliance requirements?

๏ Are there experts/enterprise support?

๏ What’s the community like?

๏ Easy to administer?

๏ Tooling, monitoring, language support?

๏ Cloud or iron?

๏ High volume ingestion or batch loading?

๏ Fault tolerance?

๏ Open source vs enterprise system?

๏ Employee learning curve vs. learning cost?

HOW DO I MOVE DATA AROUND?ROAD METAPHOR:

๏Messages = Cars

๏Message System = Highway / Roads

๏Database = Parking Lot

๏Cache = Cell Phone Lot

๏Commerce/Industry = Worker/Consumer/Analyzer

๏Enrichment = Gas Station

๏ Only recently starting to become part of important discussions

๏ Provide consistent interfaces between disparate systems

๏ Clients can have minimal architecture knowledge

๏ Everyone can speak the same language (JSON, please not XML)

๏ Allow for high availability

๏ Help minimize the cost of downtime

๏ Control data flow patterns

๏ Makes [horizontal] scaling easier

๏ Enrichment/in-stream modifications of data

๏ Instrument and monitor data states between systems

MESSAGE SYSTEMS ARE MY FAV

๏ Distributed and de-centralized topology

๏ At least once delivery guaranteed

๏ Multi-cast style message routing

๏ Simple to configure and deploy

๏ All for zero-downtime maintenance windows

๏ Ephemeral channels for testing data

๏ Channel sampling

nsq.io

HOW SHOULD DATA LOOK COMING OUT?

๏ Agree on a data format?

๏ XML, JSON, AVROJSON

๏ Again, please don’t use XML

๏ HATEOAS - heavy lift but decent client support

๏ What meta data should be sent with the response?

๏ How can unnecessary calls to an API be mitigated?

HOW DO I GET DATA OUT?

๏ Monolithic service architecture

๏ REST interface through a single URL to ask for data?

๏ Many micro-service end points?

๏ HTTP / RPC / THRIFT

๏ JSON API / HATEOS / CUSTOM

๏ How many libraries need to be written, tested and maintained?

And now back to our story…

SIMPLEREACH CONTEXT

๏ 100 million URLs

๏ 300 million Tweets

๏ 50k - 100k events per second (tens of billions of events per day)

๏ 200G new per hour

๏ 700T of total data (10T per month)

๏ 10T of hot data

๏ 2-3T of daily log data

๏ Excludes all monitoring data

Vertica + Cassandra

Vertica

STREAM-BASED DATA COLLECTION

Internet

rnal A

Vertica

Fire Hose

NEED FOR SPEED

๏ Concurrency

๏ Compiled code is much faster

๏ Statically typed languages make for less unexpected error situations

๏ Still speaks every other interchange language

๏ Cleaner code

MICROSERVICES: THE NEW HOTNESS!

๏ Fine grained, clearly scoped services

๏ Break 1 thing != break #allthethings

๏ Better fault isolation

๏ Easier to create throttles/release

valves

๏ Better able to monitor more

granularly

๏ Made everyone more devopsy

MICROSERVICES: THE NEW HOTNESS?

๏ Strict micro-service setups have

large database overheads

๏ Testing/deployments are more

complex

๏ More general overhead

๏ Slow down developer time

๏ Service discovery

Pros Cons

HYBRID MICRO-SERVICE / SHARED LIBRARY

Business/Application/Translation/Data Access

Router/Load Balancer/Config/Authentication

SERVICE SERVICE SERVICE SERVICE

Redshift

Platform

GENERIC SERVICE AND DATA FLOW

Redshift

Data Access Layer

Business Logic

NSQApplication Layer

Data Access Layer

Business Logic

Data Access Layer

Business Logic

NSQ Data Access Layer

Business Logic

logstash

SMART ROUTER

๏ Handles service state and service registry/discovery information

๏ Canonical reference for all things platform

๏ Prevents older versions of services from re-appearing

๏ Highly available proxy application

๏ Has burst-able capacity to mitigate DoS

๏ Auto-scaling tier

BUSINESS LOGIC LAYER

๏ Contains thicker macro services

๏ Aggregates common features and functionality

๏ Permissioning/throttling/access restrictions

๏ Centrally handling trigger events

๏ Exposing various API end points

๏ Orchestrating calls to the DAL

DATA ACCESS LAYER

๏ Responsible for CRUD

๏ Houses many of the data models

๏ Responsible for balancing throughput of data in/out of databases

๏ Minimize the number of DB connections by using pooling

HYBRID MICRO-SERVICE / SHARED LIBRARY

Redshift

Platform

WebApp 1 WebApp 2 Python App Go App

Ingestion Stream

Proxy/Router

Ingestion Stream

SMILEY HAPPY PEOPLE

HOW DO I KNOW IF SOMETHING IS WRONG

๏ Testing

๏ Monitoring

๏ Instrumentation

๏ No pull requests w/o instrumentation

๏ No pull requests w/o monitoring

๏ Build dashboards

DASHBOARD #ALLTHETHINGS

WHAT SHOULD I MONITOR/INSTRUMENT?

๏ Frequency

๏ Error rates

๏ Success rates

๏ Request Volume

๏ Message Counts

HOW DO I MAINTAIN/SCALE/BUILD?

๏ Already discussed monitoring/instrumentation

๏ Making sure you can maintain architecture is the same as ensuring you can

maintain code

๏ Have easy to use, flexible deployment systems

๏ Keep an audit trail

๏ Make processes repeatable and systematic

๏ Configuration management

๏ Automation (event based when possible)

๏ Easy enough to add and maintain but difficult to break

If you want to increase innovation, you need to lower the cost of failure.

Joi Ito, MIT Media Lab

WHAT JUST HAPPENED

๏ A little architecture knowledge is a good thing

๏ Don’t start out with complexity

๏ Build what you need with growth in mind

๏ Make sure you have the basics covered

๏ Might be something to the micro-service hype

๏ Monitor everything

๏ Allow customizations and innovations

QUESTIONS IN LIFE ARE GUARANTEED,

ANSWERS AREN’T.

Eric Lubow

@elubow

Data Day Seattle

#ddsea15

Infrastructure for Decision Makers

Technology

Presenting to Decision Makers

Using Advanced Analytics, Drug Makers Reach Decision-Makers More Effectively

Industry Perspectives: Technology Decision Makers

KEY MESSAGES FOR DECISION MAKERS

NEGOTIATION FOR DECISION MAKERS

Technology Decision-Makers

Business Statistics for Decision Makers

HTA AGENCIES AND DECISION MAKERS - INAHTA Decision Makers.pdf · HTA AGENCIES AND DECISION MAKERS . ... but the nature of decision makers and their interaction in the process needs

Influencing Elected Officials & Decision Makers

Assessment for Decision-Makers

Lobbying the Decision Makers

Decision Makers Needs Workshop Report

Mobile Learning for decision makers

Ch 3 Economic Decision Makers

BIOMETHANE GUIDE FOR DECISION MAKERS

Drupal for Decision Makers

How Can Developing Country Decision Makers Incorporate ... · How Can Developing Country Decision Makers Incorporate ... Country Decision Makers Incorporate Uncertainty about

SPONSORSHIP OPPORTUNITIES DECISION MAKERS

The magazine for decision makers

Financial Management for Decision Makers