From dev to ops and beyond - getting it done

FROM DEV TO OPS ANDBEYOND

GETTING IT DONE

Volker Dusch / @_ _edorian

ABOUT MESoftware EngineerPHP since 11 yearsCICleanCodeDevOpsTDDShippingBullet points

INSTEAD OF ME

WORKING FOR

ResearchGate gives science back to the people who make it happen.

We help researchers build reputation and accelerate scientificprogress.

On their terms.

GET IN TOUCH

stackoverflow: Twitter: @_ _edorianG+: Volker DuschIRC: edorianMail: php@wallbash.com

AGENDAWe will look at what is necessarily to keep shipping a successful

product beyond its initial release and what role we developers shouldbe playing in this.

Development practicesPlanning and CommunicationDeployment and OperationsYMMV

BUT BEFORE WE GET GOINGTime for a little exercise

10 YEARS AGOHOW I GOT HOOKED ON CREATING STUFF ON THE

WEBCreating things is awesomeIt was super easy and fun

10 YEARS AGOCUSTOMERS

"We want a website!""Can you fix this little thing please?""What do you mean you're already done?"

10 YEARS AGOWHAT I DIDN'T KNOW BACK THEN

Things got bigger, a lot bigger"Web applications" vs. "Websites"Maintaining things was really hardIt's about more than just programming

INTERMISSION

INTERMISSION

MY FIRST WEB APPLICATION6 devs, growing to 20Big code baseQuite some data, for the timeThis talk is about everything we had to take care of, and more

SHIPPING BIGGER THINGS

LET'S SHIP IT!THINGS TO TAKE CARE OF

Fulfilling current requirementsFiguring out what to do nextDelivery and Operations

ASK QUESTIONS AT ANY TIME!

WE'RE DEVELOPERSLet's start with us!We get paid to do what we loveMost of us started because we where fascinated by programmingBut what is our job?

OUR JOB IS TO DELIVERGet things doneGive good estimatesAsk the right questionsKeep doing thatDon't slow downKeep our promises

THE COST OF HAVING US FOLKS AROUND

German numbers, YMMV €, Approximations

Salary: ~50k a year 50.000€ / Year

Adding non-wage labor cost 80.000€ / Year

Office space, water, hardware, coffee,plants, cleaning, travels, training

100.000€ / Year

Weekends, holidays, vacation, etc:We works 220 days a year.

455€ / day

"Classic" 8 hour work day 55 Bucks per Hour!

We are expected to contribute over 100.000 Bucks in business valueper year!

HOW?Evaluate everything you're doing by a simple question:

“Does the practice in question help us to continuouslyship what our business needs to succeed?”

CODERules for structuring our source code that have proven to help with

sustainability.

THE MOST BASIC THING:Separate the web-glue from the business logic.

Keep templates stupidHave services owning the logic for manipulation of business entitiesHide the data access behind a layer that you can maintainindividually

SOLIDSingle responsibilityOpen-closedLiskov substitutionInterface segregationDependency inversion

http://en.wikipedia.org/wiki/SOLID(object-orienteddesign)

COMPOSITION OVER INHERITANCEDon't inherit from things if you can just use them.

http://c2.com/cgi/wiki?CompositionInsteadOfInheritance

http://en.wikipedia.org/wiki/Compositionoverinheritance

DEPENDENCY INJECTION

This goes for you code base as well as for your whole Company.

Allows for quick and easy reuse of small componentsIf wiring is to hard people will copy/paste insteadCan be achieved in many ways. Pick one that works for you

http://pimple.sensiolabs.org/http://symfony.com/doc/current/components/dependency_injection/index.htmlhttps://github.com/researchgate/injektor

“In the end - everything is a wiring problem.”

REUSING CODEThe Dependency Manager for PHP

As easy as committing everything to your SCM. But with care freeautoloading.

TESTINGThe art of making sure that you notice failures before your customers

do.

Testing exist to give you confidence when moving forward.

THE FASTEST THING YOU CAN DO

Staging serverTesting your buildsAll without even touching Behat or PHPUnit

hits=̀curl -s staging.project.com | grep 'Login:' | wc -l̀;test $hits -eq 1 || echo "Frontpage error!"

data="login=test&passwort=secure&csrf="$csrfTokenurl="staging.project.com"hits=̀curl -X POST -d $data $url | grep 'Hello, testuser' | wc -l̀;test $hits -eq 1 || echo "Login error!"

OTHER DEPARTMENTS DON'T CARE ABOUT UNIT TESTINGNor should they!

Your fellow developers on the other hand ... :)

“The mechanic fixing your car doesn't ask you whatsong she should listen to while performing the task.”

CONVENTIONAL WISDOM

BUT TESTING IS HAAAARDWriting proper code is hardThe harder it is to use the code in question, the harder is writing testsfor itComplex tests means the code is to complex. Break it down.If anything: Mocking is hard(-ish).

Phake is your friend:http://phake.digitalsandwich.com/docs/html/FBMock: Mocking for grown ups:https://github.com/facebook/FBMockThe PHPUnit mocking API is still good enough.

TDDWriting tests can feel like extra work if you are rethinking an already

solved problem

TDD offers a way to first think about the problem, the interface andthe interactions and then filling in the details step by step until you are

done with the bigger picture.

QUICK WINS WITH BEHATWeb tests help you detect a big amount of wiring issues with littleeffortBefore growing to many of them consider maintenance"Full stack testing" will allow you to ship with great confidence

ENSURING MAINTAINABILITYGetting rid of all the things you might stumble over

CODE REVIEWThere are a lot of ways to go about this

Small teams can use commit based reviewWhen feature branching the merge back is a natural pointInternal coding DojoPair programmingSend emails when important changes occur

The main point of these ideas is to make sure everyone has a goodunderstanding of how common things are solved and to keep people

communicating.

CODING GUIDELINESA collection for formatting and structure rules so that everyone can

easily find their way around and produce uniform code.

Just create the rule set fitting your practicesAdapt when there is painDon't over analyze. Just do it

PHP CodeSniffer:

http://pear.php.net/package/PHP_CodeSnifferhttp://pear.php.net/manual/en/package.php.php-codesniffer.annotated-ruleset.phphttp://edorian.github.io/php-coding-standard-generator/#phpcs

“Coding rules:It doesn't matter what they are - as long as you have

them.”

COMPLEXITY GUIDELINESSimilar to a coding standard but focusing on hunting down potential

problems within your source code

Possible bugsSuboptimal codeOvercomplicated expressionsUnused parameters, methods, properties

PHP MessDetector:

http://phpmd.org/http://phpmd.org/rules/http://edorian.github.io/php-coding-standard-generator/#phpmd

CONTINUOUS DEVELOPMENT PACE"Done" means there is nothing left to clean upEvery once in a while you plan time to throw away things"Having a clean slate" is an attitude and a culture

“There are only two hard problems in ComputerScience: cache invalidation, naming things, and off-

by-one errors.”

CIHave a automated way of checking all the things you agreed on

Run web and unit testsEnsure coding guidelinesEnsure complexity guidelinesKeep the project deploying :)

Tools:

http://jenkins-ci.orghttp://jenkins-php.orghttp://www.sonarsource.org

WORKING IN SHORTITERATIONS

Every iteration is a chance for people to "sync" their Vision of theproduct with the current reality.

SHIPPING REDUCES COMPLEXITY"Did we already implement this a month ago?""That bug you just reported was fixed 2 weeks ago. Just notdeployed yet"Shipping let's you discover what doesn't work before you spend toomuch time on it.

PLANNING CAN BE A BIG MINDSET CHANGE“Nobody told me my job involved talking to... people”

ASSUME COMPETENCEWork with the basic mind set that other people are at least as good in

their job as you are in yours.

If they tell you to do "stupid" things find out what they want toachieveCompany "vision" is a big hereJust know what you want to get done and work together

HAVING EVERYONE INVOLVEDGetting everyone in the same boat and working towards a common

goal will speed you up like nothing else every will.

ProductDesignCopyrightingEngineering""Upper management""

If you can ensure that everyone was involved somewhere in the loopyou spend way less time on re-discussing and avoid confusion.

HONESTY SOLVES A LOT MORE ISSUES THAN ITCREATESTrue story.

TOOLINGTooling is needed when people can't get the information they need

A wall with Post'it notes and a stack of story cards might be all youneedWhen working remote spend time on making it look beautiful so thatpeople look at ithttps://trello.com/$millionOfOtherProducts

BUGSTechnical errorsCommunication errors (and forgotten features)Fix the technical issues quickly to reduce complexityDon't build "bug management".But at a certain size you need someone to dispatch bugs to safeeveryone time

STATSState are awesome!Knowing what your users actually do with your software is valuableSeeing them use the new thing you build is - like - the best ever(tm).StatsD: https://github.com/etsy/statsd/Graphite: http://graphite.wikidot.com/

OPS AND DEVOPSIf it's not in production it doesn't really count.

CLASSIC APPROACH

DEVOPSTalking to the people getting up at night so you don't have to.

Your SysAdmins care. A lot!It's your job to figure out a way to constantly deploy withoutbreaking thingsAutomation helps avoiding errors

BUILD AND DEPLOYMENT TOOLINGA collection of scripts that gets you from "source code" to "running in

production".

Create a buildCan be done in Jenkins. Still keep it in a script and let Jenkins runit.Run unit testsRun integration, functional testsReport on coding/complexity metrics

Once you are confident that things are going to go wellDeploy the build to a staging environmentRun web testsDeploy to productionRollback when really really needed

That's it. No magic - Just detail work.

HOW TO GET THAT SCRIPT?It doesn't matter what language you write that tooling in. There is no

generic answer.

Chances are there is going to be a lot of BASH in it.Whatever calls your BASH script isn't all that importantAntMore custom BASHAnt Build CommonsSF2 CLI commandsWhatever works. As long as you can maintain it and it's stable.Don't over-engineer. You'll iterate a lot over this.

LINKShttp://abc.tools.qafoo.cohttp://www.capistranorb.com/Example Ant: http://jenkins-php.orghttp://symfony.com/doc/current/components/console/introduction.html

PROVISIONINGGetting servers and development environments in a known state

Having the SAME versions on dev and prod is invaluable.The only documentation that gets always updated is the code.Being able to recreate servers safes a lot of hassle.Think about your firewalls :)

Having a virtual machine per project keeps you sane

http://www.vagrantup.com/https://puppetlabs.com/http://www.opscode.com/chef/

CONFIGURATION MANAGEMENTAny solution will require close collaboration with the folks running the

production boxes or lead to hassle.

Config files in puppet/chefConfigs for all systems in the SCM and only reading in environmentsSetting systems up in a way that they always look the same

DATA MIGRATIONSThis questions comes up a lot when talking about automation. There is

no easy answer.

Very specific to your project and environmentUsually not possible to have downtime while data is migratedDon't throw away DB columns when you change the code. Add newones and clean up later when you are sure.

THANK YOU!

QUESTIONS?Get in touch

Twitter: @_ _edorianG+: Volker DuschIRC: edorianMail: php@wallbash.com

Check those things out:

Facebook: https://www.facebook.com/Engineering?sk=notesEtsy: http://codeascraft.com/