6/18/14 Billing & Payments Engineering Meetup I

Billing & PaymentsEngineering Meetup

June 18th, 2014

• Mathieu Chauvin – NetflixValidating Payment Information

• Taylor Wicksell – NetflixFlexible Billing Integrations Using Events

• Jean-Denis Greze – DropboxMistakes (and Wins) Building Payments at Dropbox

• Anthony Zacharakis – LumosityBilling Migrations: Maintaining Your (Data) Sanity

• Alec Holmes – SquareScaling Merchant Payouts at Square

=== Break ===• Paul Huang – SurveyMonkey

Billing Globalization and Auto Renewal Optimization• Emmanuel Cron – Google Wallet

Moving from SE to Host Card Emulation• Feifeng Yang / Michael Chen – Electronic Arts

Ecommerce at EA• Krishnan Sridhar – LinkedIn

Real-Time Analytics and Smart Routing

Validating Payment Information

Mathieu Chauvin@matchauvin – linkedin/matchauvin

Signing-Up for Netflix

• Subscription Based• 30 Days Free Trial• Payment Information

Collection

• Pro Forma• Luhn Check (Mod10)• Card Type vs Card# RegEx• Authorization / Reversal

Payment Validation 101

But Wait! You Can Do More!

Authorization Amount

• $0.00, $1.00, or Full Price Authorization• Don’t Forget the Reversal!• Constraints by Country or by Card Type

Response Types

• Hard Declines• Soft Declines

MCC

• What is an MCC?• Figure out the Best MCC Fit

BINs

• What is a BIN?• Card Types: Credit, Debit, Prepaids• What About Non-Reloadable Prepaid Cards?

Forking Traffic

• Do You Use Several Payment Processors?• Why Don’t You Test Which One Performs

Better?

Use Cases

• Sign-Up• Updating Payment Information• Returning Customer

Country Specific Behavior

• Adapt To Your Market

The Feedback Loop

• A/B Tests• Analyze, Project & Evaluate Risks• Test Again

Conclusion

• Try Everything• Build Your Application Dynamically• The Feedback Loop!

Thank You!

Flexible Billing Integrations Using Events

Taylor Wicksell

The Billing Team

• Signups/Cancellations• Recurring Billing• Price and Tax Calculations• Discounts / Gifts• Financial Reporting

Current Architecture

• Netflix Data Center• Batch Processing and

Customer Requests• Oracle (Single-Master)• Data driven

Cloud Migration - Primary Goals

• Join Netflix in the Cloud• High Availability• Multi-Regional Active/Active• Scalability

New Design• Amazon EC2• Multi-Regional

• Cassandra• Event Driven

Order Flow

Cross-Cutting Concerns

• Publishing Data To Interested Parties– Hold Messaging– Financial Reporting

• Metrics/Analytics• Velocity Checks• Debugging

Wiretap to Common Event Pipeline

Common Event Pipeline

Current Implementation• SQS Entry Point• Custom routing service• Custom code for each

endpoint integration• Weak ordering of events

Future Implementation• Suro / Kafka Entry Point• Configurable Routing and

Transformation• Pluggable endpoint

integration• Option for strong ordering

of events

Sting – Broad Analytics

Druid – Event Aggregation Metrics

ElasticSearch – Event Level Debugging

Mistakes (and Wins) Building Payments At Dropbox

Jean-Denis Grèze & Dan WheelerJune 18th, 2014

Backend Tips• Not about increasing conversion• Not about pricing• Not about plan and feature

optimizations• Not about upselling• Not about consumer SaaS at scale• Not about self-serve in

SMB/SME/Enterprise

Pains of Scaling Payments• Thousands of customers to millions of

customers• SMB to Enterprise

– Custom flows!• International expansion

– Fraud– New payment methods (delayed

settlements)– Different price points

Out Of The Dark Ages• For a long time, only 0.5 engineers

worked on payments and billing• March 2013: consult w/ leading

payment engineers, PMs and executives on how to build an amazing payments team– 15+ in 1 year

Advice• Build a payments + billing backend

that is:– Flexible

• Migrations (sadface)• Requirements will change – often

– Auditable• Always know why and state changed

– Append Only• Never lose data

Team Ca$h

Team Ca$h• Engineers

– Some finance background– A lot of systems/analytics background

• PMs– Some finance/accounting background– Some product/marketing background

• Advisors– Payments experts– Tight feedback loop w/ finance

• Designers

Migrations

Migrations• You will have to migrate

– 3rd-party vaulting to self vaulting– New markets = new processors– If you are a growing company, your internals will

require migrations• Stakes are high

– Double-billing? Forgetting to charge some users? Inadvertently moving users from one pricing category to another?

• Old way:– Ad hoc– Tons of tests

I. Leverage Existing Code: Equivalence

• Write equivalence between old and new implementations (database, API, 3rd party providers, tests, etc.)

• Run everything through both systems at once, with equivalence being tested

• Every step of the way check that equivalence relations hold (e.g., old-style invoice has a new-style invoice equivalent)

• Turn off old system when everything works for X amount of time

Migration Pro-Tip• If you can migrate in both

directions at will on a per-endpoint basis, your life will be awesome and people will love you.

Moneytree

Logging• Dark Ages = tons of logging

• Very comprehensive, but ad-hoc = too much effort to re-create state

• Human error

II. Automated Logging• Automatically log

– Any DB write (graph, relational, etc.)– Any 3rd party API call (and some internal calls

like email)• Pre-log

– Any incoming 3rd party payload• Can recreate past actions if we introduced a

regression• LOG A REASON (and code version)

– 1 year is a long time

Automated Logging =

Time Machine

Not too much data• Dropbox = large scale for SaaS• Hundreds of millions of users

(provisioning is hard)• Tons of data, but still payments data

<< file storage data– Although we do have the benefits of

amazing infrastructure

III. States, Not Deltas• Generally # states << # deltas• States

– Pro 100 + Packrat– DfB + Harmony Enabled

• Deltas– Add 5 licenses, 6 licenses, etc.– Add 20 licenses and switch from monthly to

yearly• Use states and let the system figure out

how to get from start to end

IV. Possibility & Transitions Use Same Code Paths

• No difference between:– entity.is_valid_transition(end_state)– entity.perform_transition(end_state)

• Except that writes are turned off for the former.

• No change for “is something possible” logic to be different than “do the thing” logic.

States Are Nicetransition_space = MoneytreeTransitionsSpace.build_cross_product( entity=me, gateways=ALL_GATEWAYS, plans=[Pro100, Pro200, Pro500, DfB, DfBTrial], schedules=[Monthly, Yearly], currencies=ALL_CURRENCIES, features=ALL_FEATURES, tax_profile=[NoTax, SimpleTax, ComplexTax],)# …If transition_space.supports(FEATURE_PACKRAT): # …

Write Protection• Seems dumb, but need to be careful

not to accidentally change values in payments world. Have clearly-defined code paths that can touch state, talk to 3rd party components, etc.

V. One More Lesson• Payments + Billing != Finance• Business requirements don’t always

translate to what’s best/easiest in the world of accounting. You need to flexibly work in both worlds – can’t risk the user experience to make your finance dep’t happy (and vice-versa)

• Get a great PM

VI. Why Payments Are Cool?• Infrastructure?

– Payments service– Provisioning service

• Product?– Upsells? (+50% increase in revenue per user)– Gating features?

• Product Infrastructure!– Build a successful structure by emphasizing

hard problems in both worlds!

Now + The Future• Other Cool Projects

– ML for risk/anomaly detection (e.g., for payment methods that don’t settle immediately)

– Price AB testing (*)– Cross-platform upsell framework

• Questions?– dan@dropbox.com– jeandenis@dropbox.com

• Hiring– Get in touch!

Team Ca$h

Lumos Labs, Inc.

Data SanityA short treatise on

rebuilding a payment system

Lumos Labs, Inc.

the background

Lumos Labs, Inc.

Circa 2012Decided to rewrite our payment system

Lumos Labs, Inc.

ReasonsOld payment system has a lot of limitations

Lumos Labs, Inc.

Payment system limitations• Hard to add additional gateways• Models don’t reflect business

reality• Not built for reporting• Code is brittle

Lumos Labs, Inc.

3 months later…

Lumos Labs, Inc.

New payment system features• Trivial to add new payment

methods• Subscriptions are the core

model• Built with reporting in mind• Separate, well encapsulated

library

Lumos Labs, Inc.

Works like a charm!

Lumos Labs, Inc.

wait… just one problem

Lumos Labs, Inc.

Having two parallel payment systems is

not sustainable

Surprise!

As it turns out

Lumos Labs, Inc.

Just deprecate the old one

Lumos Labs, Inc.

Just deprecate the old one• Don't migrate anyone, let old

users churn out naturally (will take forever)

Lumos Labs, Inc.

Just deprecate the old one• Don't migrate anyone, let old

users churn out naturally (will take forever)

• Migrate everyone, but only most critical/current subscription info (loses history)

Lumos Labs, Inc.

Just deprecate the old one• Don't migrate anyone, let old

users churn out naturally (will take forever)

• Migrate everyone, but only most critical/current subscription info (loses history)

• Migrate everyone + full history (tricky, lots of edge cases)

Lumos Labs, Inc.

so, what’d we choose?

(drum roll)

Lumos Labs, Inc.

We decided to migrate everyone and their entire

billing history

Lumos Labs, Inc.

Why?One systemOne history

One source of truth

Lumos Labs, Inc.

Lumos Labs, Inc.

enter the sanity check

Lumos Labs, Inc.

enter the sanity checkJust make a sanity check before and after the migration to ask questions

Lumos Labs, Inc.

enter the sanity checkBoth models should answer certain questions the same way, e.g:• How much did the user pay for a

subscription?• How many total transactions did the user

make?• Was auto-renewal enabled on X date?

Lumos Labs, Inc.

class SanityCheck Methods = [:subscriber?, :transaction_count] # etc.

def initialize(record) @record = record @before_values = SanityCheck.values(record) end

def self.values(record) Methods.map { |m| [m, record.send(m)] }.to_h endend

enter the sanity check

Lumos Labs, Inc.

class SanityCheck ...

def check @after_values = SanityCheck.values(record) @diff = diff(@before_values, @after_values) end

def diff(a, b) a.delete_if { |k, v| b[k] == v } .merge!(b.dup.delete_if { |k, v| a.has_key?(k) }) endend

enter the sanity check

Lumos Labs, Inc.

enter the sanity checkUser.each do |user| sanity_check = SanityCheck.new(user) user.migrate! sanity_check.check if sanity_check.diff.any? # sanity check failed -- log an error, rollback, etc. raise ActiveRecord::Rollback else # woo hoo, success! user.update_attributes(:migrated => true) endend config/initializers/

user_auth.rb

Lumos Labs, Inc.

the result

Lumos Labs, Inc.

A resounding success!

the result

Lumos Labs, Inc.

(Mostly)

the result

Lumos Labs, Inc.

Did not work in all cases●Payment system behavior changed

over time

Lumos Labs, Inc.

Did not work in all cases●Payment system behavior changed

over time●Some concepts did not map

between systems

Lumos Labs, Inc.

Handling the edge cases●Replayed history as it would play out

in the new system●Still kept most critical information

the same (e.g. transaction timestamps)

Lumos Labs, Inc.

Skip ahead to todayMigrated 99.9994%* of all our users successfully to the new system

*The remaining .0006% live on for business, not technical reasons

Lumos Labs, Inc.

ThanksAnthony Zacharakis

anthony@lumoslabs.com@azacharax

Scaling Merchant Payouts at Square

Alec Holmes

‣ Infrastructure team that moves money

‣ Card processing (money in)‣ Settlements (money out)

Payments Team

‣ Monolithic Rails app‣ Database storage and IO limits‣ Controls around accounting data

Disaster Ahead

Monorail

Database

Payments Bills

Deposits

Where We Started

Ledger

Database

Payments Bills

DepositsMonorail

Database

API

Current System

Challenge:Migrating Live Data

‣ Stricter money-moving event model‣ Same event streams to all systems‣ Smart queueing of events

Challenge:Moving to APIs

‣ Replace direct data access with API calls

‣ Hadoop for analytics queries

That's it. All pennies accounted for!

Billing Globalization and Auto-Renewal Optimization

• Paul Huang• Engineering Manager• SurveyMonkey• June 18th, 2014

Our mission is to help people make

better decisions

We help people get answers to their questions

customers create surveys distribute to others recipients respond customers analyze and gain insight

Our customers have invented many ways to use our tool

98

90MUnique visitors every month

17K New sign-ups

daily “SurveyMonkey turns online

surveys into a hot business.”

“ Start-up companies using ‘freemium’ business models, including SurveyMonkey, are thriving as the cost of computer power and storage falls.”

One of the hottest startups to watch. 2.4M

Survey responses are generated daily

SurveyMonkey is the world’s largest survey company

Billing Globalization

In 2010...

Currencies:1 currency (USD).

Payment Methods: Credit Card, Invoice.

Pricing: one price per package.

Revenue: $ MM

Billing Globalization

In 2014...

Currencies: 39 international currencies.

Payment Methods: Credit Card, PayPal, Debit Card, Bank Transfer, iTunes, Invoice.

Pricing: each package with different price per currency, multiple prices allowed for price testing.

Revenue: $$$$ MM

Auto-Renewal Optimization

Retry Logic: retry failed payments at different intervals, ~60% retry success rate

Account Updater: get users’ updated credit card data (number / expiration date) before next renewal date, ~4% users at 96% success rate

Immediate Renewal: charge pending invoices when users update payment accounts, ~4% retry success rate improvement

TheRoad

Ahead

Upcoming Billing Projects in 2014

VAT 2015: preparations for VAT rule changes in Europe in 2015

Brazil Payment Processing: set up local entity, integrate with a new Payment Service Provider in Brazil

Continue Improving Retry Logic: increase retry frequencies

The End (BTW, we’re hiring…)

E-commerce at EA

Feifeng Yang / Michael ChenTech Lead Payment / Software Engineer @ Electronic Arts

E-commerce platform• Power hundreds of games,

millions of gamers• FIFA• Battlefield• SimCity, Sims• Origin• Play4free, etc

• Foundation services to EA game studios

Foundation services• EA Origin Store: http://store.origin.com• Payment Gateway• Ewallet System• Billing System• Subscription System• Risk and Fraud System• Shopping Cart System• Catalog System• Rating and Revenue Allocation System

Foundation services

Billing and Payment• Preorder and subscription• Package goods• EA Ewallet: million of transactions per day• Product types: base game, bundles, DLC, PDLC, etc• Payment methods

• Card processing• PayPal• Mobile billing• Other ~30 local payment methods.

Technologies• RESTful API framework: SimpleHttpServer (home grown, ported from open source project SimpleWeb)

• Application: OSGI, Use Equinox as OSGI container• Datastore: MySQL + Sharding, Oracle (to be retired)• Cache: Coherence cache (to be retired); Memcached• Security: OAuth 2.0, CAS 2.0 (Central Authentication SSO)

• JMS messaging: TIBCO, iHUB.

112

* Jun 1st to Oct 17th