Follow robert.rotarius@rakuten.com

AGENDAReally, really brief history of Rakuten Affiliate NetworkMoving faster with simpler processesMoving more efficiently with simpler technologySome things we learned along the way

REALLY, REALLY BRIEF HISTORY OF RAKUTENAFFILIATE NETWORK

A dinosaur in internet years.LinkShare was founded in 1996Rakuten acquired LinkShare in 2005Re-branded to Rakuten Affiliate Network in 2014

WHAT THAT GOT USTechnical DebtProcess Debt

BUT WHAT WE REALLY WANTED...Tech to feel empoweredMinimize distractionsEliminate blockers

THE CUSTOM REPORTING PROJECT

PROJECT GOALSSoftware Development

Introduce more structure to our JavascriptMore automated testsCleaner code, clearer patternsAPIs first

Operations

Move fast, efficient and iterate quicklyUse MongoDBShip Faster

THE SCHEMA { "_id" : ObjectId("5453ce5ee02222341b0000ea"), "user" : DBRef("User", "5453ce5de0945657b00000a"), "name" : "Sam Witwicky", "slug" : "sam-witwicky", "type" : "custom", "description" : "Cool human", "category" : "Humans", "updated" : ISODate("2015-02-24T03:00:57.802Z"), "created" : ISODate("2015-02-24T03:00:57.802Z"), "is_runnable" : true, "filter_groups" : [ { "_id" : ObjectId("5453ce5ee0909e9f1b0000eb"), "filters" : [ {

View on Github

THE MODEL namespace RM\Document;

use RM\Document\ReportColumn; use RM\Document\FilterGroup; use RM\Document\User;

use Doctrine\Common\Collections\ArrayCollection; use Doctrine\ODM\MongoDB\Mapping\Annotations as MongoDB;

/** * @MongoDB\Document(repositoryClass="RM\Document\ReportRepository") */ class Report { /** * @MongoDB\Id

View on Github

THE CONTROLLER namespace AppBundle\Controller;

use AppBundle\Document\Report; use AppBundle\Form\ReportType;

use Symfony\Bundle\FrameworkBundle\Controller\Controller; use Symfony\Component\HttpFoundation\Request; use Symfony\Component\HttpFoundation\Response;

use FOS\RestBundle\View\View, FOS\RestBundle\View\ViewHandler, FOS\RestBundle\View\RouteRedirectView;

use FOS\RestBundle\Controller\FOSRestController; use Symfony\Component\Routing\Route;

View on Github

THE HTML <div class="report"> <div class="report-id">5453ce5ee02222341b0000ea</div> <div class="user"> <div class="username">SomeUser</div> </div> <div class="name">Sam Witwicky</div> <div class="slug">sam-witwicky</div> <div class="type">custom</div> <p class="description">Cool human</p> <div class="category">Humans</div> <div class="updated">2015-02-24T03:00:57.802Z</div> <div class="created">2015-02-24T03:00:57.802Z</div> <div class="is-runnable">true</div> <div class="is-deleted">false</div> <div class="filter-groups"> <div class="filters">

THE JAVASCRIPT function Report() { var self = this;

this.name; this.description;

this.filterGroups = []; this.reportColumns = [];

this.addFilterGroup = function(filter){ self.filterGroups.push(filter); }

this.addReportColumn = function(reportColumn){ self.reportColumns.push(reportColumn); }

THE CSS .report { .name {} .description {}

.filter-groups { .filter { .operator {} .default-value {} } }

.report-columns{ .report-column { .format {} .sort-direction {} }

THE PLAYERS

SHIP IT!

Threat Level: LowJoeThe Product Manager

Automated testing helps him tooBeware of featuritis.Work with him to simplify the requirements

AUTOMATED TESTINGCreate a culture of testingImprove communication between stakeholders

BDD

FEATURE FILES

Scenario: Valid CRUD requests Given body of request: """ { "name": "My Special Report", "description": "The description of my special report" } """ When I make a POST request to "/reports/" Then the status code should be 200

When I make a GET request to "/reports/5453ce5ee02222341b0000ea/" Then the status code should be 200 Then the response should be: """

Threat Level: MediumDannyQA Guy

Long regressionsResistant to introducing code too close toreleasePush back on unrealistic test casesYou need to earn his trustGet him involved in the Feature File process

MOVE FASTER AND MORE EFFICIENTLYRelease more oftenEliminate distractionsDon't get woken up

Threat Level: HighSheilaThe System Administrator

Forced to limit her drinking for 1 week everymonth.Won't support your system withoutEVERYTHING being documented.She just wants to know how your systemworksYou are going to have to start sharingsupport duties

TAKING ON SUPPORTOne SOP to start: call the application ownerIncrementally build on existing proceduresTransition SOPs to Operations

USE MONGODBEasy to setupIntuitive query interfaceEasy to maintainJavascript friendly

No shardingLimited audience = Limited riskMongoDB was involved the entire wayConfidence was high

Threat Level: HighSamuelThe DBA

Fear introducing new thingsHates ORMsDoesn't like you designing your own schemaThinks NoSQL is a fadEnforcing data integrity at the app layerDe-normalization is okLow maintenanceAutomated backup and restore

WE MADE IT TO PRODUCTION. NOW WHAT?

THE PRIMARY NODE IS GETTING A BUNCH OF READS

READ YOUR OWN WRITE

WHAT IS IT?

IMPLEMENTATION WITH DOCTRINE ODM

namespace AppBundle\EventListener;

use Symfony\Component\DependencyInjection\Container; use Symfony\Component\HttpFoundation\Session\Session; use Symfony\Component\HttpKernel\Event\GetResponseEvent;

use Symfony\Component\EventDispatcher\EventSubscriberInterface;

class MongoReadPreferredSubscriber implements EventSubscriberInterface { /** * @var Session */ protected $session;

/**

PERIODIC NETWORK PARTITIONS

WHY WE MOVED AWAY FROM ARBITERS

5 Node Replica Set (normal)

A & B cannot see each other. A1 wins an election.

Writes go to A1

A & C cannot see each other. B1 wins an election

Writes go to B1

Connectivity is restored

USING A DATA NODE INSTEAD

5 Node Replica Set (normal)

A & B cannot see each other. A1 wins an election.

Writes go to A1

A & C cannot see each other. C1 wins an election

Connectivity is restored

APP HANGS WHEN THE PRIMARY CHANGES(No candidate servers found)

PAY ATTENTION TO YOUR TIMEOUT SETTINGS

connectTimeoutMSThe timeout for establishing a connection.

socketTimeoutMSThe timeout for read-write operations, isMaster and ping requests

A PROPOSED FIXPHP-1223

Use connectTimeoutMS instead of socketTimeoutMS for isMaster and pingrequests

CONSIDERATIONSTimeouts vary driver to driverKeep connectTimeoutMS as low as possible

SINCE THEN...9 number of certified MongoDB DevelopersDBAs are currently handling production support issues for all but one cluster(no issues so far)We've built a couple sharded clustersLeverage the MongoDB support team as early as possible

WHAT'S NEXTGLOBAL EXPANSION

Tag Aware ShardingReplace homegrown backup solution with MMS Backup