COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
BACKGROUND
▸ 37 years old, Linux and FreebSD user, Ruby lover
▸ Managed infrastructures for various SAAS companies
▸ Specialised in taking over forsaken, outdated, undocumented platforms
▸ Loves working under pressure
▸ A sick sense of humour
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
PREREQUISITES
▸ Being in control of the budget, and having enough cash to do what needs to be done
▸ Total control over the team hirings
▸ Freedom on how to make things viable again
▸ Involved in the product roadmap
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
PRIORITIES… ACCORDING TO YOUR COLLEAGUES
▸ Sales / Client Success / Marketing: Stability
▸ Sales / Client Success / Marketing / Devs: Delivering new features
▸ Devs: On demand environments / Automation
▸ Management: Documentation / Conformity
RESTORE CONFIDENCE IN THE INFRASTRUCTURE
Priority is a singular word
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
COMMANDO DEVOPS MANAGEMENT
▸ Scrum is cool, but it lacks flexibility, Kanban lacks visibility
▸ 1 week sprints, 50% for planned projects, 50% for ongoing tasks
▸ 1 admin == 1 project, systematic code reviews for quality and knowledge sharing
▸ Team based sprint planning on Friday: we know what to do Monday morning
▸ Daily standup every morning, 10:00AM at the coffee machine
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
ONGOING MANAGEMENT
▸ Everybody solves the incident, not only the oncall guy
▸ Week based contact point
▸ Ongoing tasks are added to the current sprint or put on the backlog
▸ “No" or “later" are acceptable answers
▸ Always give an ETA!
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
DEFINE YOUR RELATIONSHIP WITH YOUR COLLEAGUES
▸ Your clients are people you’re facing directly. They’re the one you need to satisfy.
▸ Are you in contact with the company’s client?
▸ The relation is not immutable. You can switch from a service provider to an integrated component of the product team.
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
WHEN YOU HAVE NO TEAM
▸ Hire freelances while you’re looking for permanent hires
▸ Hire techno specific vendors
▸ Work with the developers who setup the existing platform
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
BUILDING THE TEAM
▸ Ensure we hire at least 2 people with the same skills
▸ We look for autonomous people
▸ Importance to know the technologies in place
▸ People who can learn quickly
▸ These people are senior profiles, therefore expensive!!!
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
COMMUNICATION
▸ Unified communication to avoid confusing messages or situation
▸ Avoid alarming messages at all cost
▸ Be totally transparent when reporting to the management
▸ Give ETAs as soon as you can. “I don’t know” is a totally acceptable answer.
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
WEEKLY REPORT
▸ Weekly report, 3-4 slides
▸ KPIs : budget, uptime, critical incidents, oncall hours
▸ Components at risk
▸ Current projects (with ETAs)
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
PLATFORM DISCOVERY
▸ Monitoring is your best friend, when it exists, often incomplete
▸ nmap + ssh + facter in a CSV: useful but lots of surprises
▸ tcpdump on network nodes (load balancers, firewalls…)
▸ Puppet / Ansible: often messy when they exist
▸ “Great old ones"
▸ Hand over with the former team (not sure they accept)
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
FROM THE BEGINNING
▸ Let the teams that manage part of the components keep their baby so you can focus on something else
▸ Build an inventory, rationalise the machine naming
▸ Restart ALL services one by one, many of them won’t come back
▸ Focus on things that work but you don’t know about instead of what you know about but doesn’t work
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
DOCUMENTATION
▸ Takes lot of time, but you desperately need it
▸ Track information flows between your components
▸ Add the operation manual in the monitoring alert messages
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
WORKING IN HOSTILE ENVIRONMENT
▸ Private clouds
▸ Networks you don’t manage
▸ AWS is a pain in the …
▸ Communication with your vendors: below a certain bill (amount depends on the vendor), the client is always wrong
COMMANDO DEVOPS = FRED DE VILLAMIL = SYSADMIN DAYS 2016
DEVELOPPERS BUILT INFRASTRUCTURES
▸ One server == one vertical feature
▸ Hard to debug
▸ Impossible to scale
▸ Need to be split in smaller VMs, by service
▸ Adds lot of pressure on your network addressing