The Commando Devops

THE COMMANDO DEVOPS

FRED DE VILLAMIL @FDEVILLAMIL

COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016

BACKGROUND

▸ 37 years old, Linux and FreebSD user, Ruby lover

▸ Managed infrastructures for various SAAS companies

▸ Specialised in taking over forsaken, outdated, undocumented platforms

▸ Loves working under pressure

▸ A sick sense of humour


PREREQUISITES

▸ Being in control of the budget, and having enough cash to do what needs to be done

▸ Total control over the team hirings

▸ Freedom on how to make things viable again

▸ Involved in the product roadmap


PRIORITIES… ACCORDING TO YOUR COLLEAGUES

▸ Sales / Client Success / Marketing: Stability

▸ Sales / Client Success / Marketing / Devs: Delivering new features

▸ Devs: On demand environments / Automation

▸ Management: Documentation / Conformity

RESTORE CONFIDENCE IN THE INFRASTRUCTURE

Priority is a singular word



COMMANDO DEVOPS MANAGEMENT

▸ Scrum is cool, but it lacks flexibility, Kanban lacks visibility

▸ 1 week sprints, 50% for planned projects, 50% for ongoing tasks

▸ 1 admin == 1 project, systematic code reviews for quality and knowledge sharing

▸ Team based sprint planning on Friday: we know what to do Monday morning

▸ Daily standup every morning, 10:00AM at the coffee machine


ONGOING MANAGEMENT

▸ Everybody solves the incident, not only the oncall guy

▸ Week based contact point

▸ Ongoing tasks are added to the current sprint or put on the backlog

▸ “No" or “later" are acceptable answers

▸ Always give an ETA!


DEFINE YOUR RELATIONSHIP WITH YOUR COLLEAGUES

▸ Your clients are people you’re facing directly. They’re the one you need to satisfy.

▸ Are you in contact with the company’s client?

▸ The relation is not immutable. You can switch from a service provider to an integrated component of the product team.


WHEN YOU HAVE NO TEAM

▸ Hire freelances while you’re looking for permanent hires

▸ Hire techno specific vendors

▸ Work with the developers who setup the existing platform


BUILDING THE TEAM

▸ Ensure we hire at least 2 people with the same skills

▸ We look for autonomous people

▸ Importance to know the technologies in place

▸ People who can learn quickly

▸ These people are senior profiles, therefore expensive!!!


COMMUNICATION

▸ Unified communication to avoid confusing messages or situation

▸ Avoid alarming messages at all cost

▸ Be totally transparent when reporting to the management

▸ Give ETAs as soon as you can. “I don’t know” is a totally acceptable answer.


WEEKLY REPORT

▸ Weekly report, 3-4 slides

▸ KPIs : budget, uptime, critical incidents, oncall hours

▸ Components at risk

▸ Current projects (with ETAs)


PLATFORM DISCOVERY

▸ Monitoring is your best friend, when it exists, often incomplete

▸ nmap + ssh + facter in a CSV: useful but lots of surprises

▸ tcpdump on network nodes (load balancers, firewalls…)

▸ Puppet / Ansible: often messy when they exist

▸ “Great old ones"

▸ Hand over with the former team (not sure they accept)


FROM THE BEGINNING

▸ Let the teams that manage part of the components keep their baby so you can focus on something else

▸ Build an inventory, rationalise the machine naming

▸ Restart ALL services one by one, many of them won’t come back

▸ Focus on things that work but you don’t know about instead of what you know about but doesn’t work


DOCUMENTATION

▸ Takes lot of time, but you desperately need it

▸ Track information flows between your components

▸ Add the operation manual in the monitoring alert messages


WORKING IN HOSTILE ENVIRONMENT

▸ Private clouds

▸ Networks you don’t manage

▸ AWS is a pain in the …

▸ Communication with your vendors: below a certain bill (amount depends on the vendor), the client is always wrong


DEVELOPPERS BUILT INFRASTRUCTURES

▸ One server == one vertical feature

▸ Hard to debug

▸ Impossible to scale

▸ Need to be split in smaller VMs, by service

▸ Adds lot of pressure on your network addressing

QUESTIONS ?

Career

The Commando Devops