View
1.755
Download
3
Category
Preview:
DESCRIPTION
Citation preview
Some Conference
OSCO
N 2011
Mistakes were madeSelena Deckelmann
@selenamarie
Some Conference
OSCO
N 2011
This goes out to all the sysadmins.
Some Conference
OSCO
N 2011
You can never think about failure too much.
Some Conference
OSCO
N 2011
Some Conference
OSCO
N 2011
Some goals around pessimism.
Some Conference
OSCO
N 2011
Plan for the worst.
Some Conference
OSCO
N 2011
Minimize risk.
Some Conference
OSCO
N 2011
Recover, gracefully.
Some Conference
OSCO
N 2011
Plan for the worst.Minimize risk.
Recover, gracefully.
Some Conference
OSCO
N 2011
Some Conference
OSCO
N 2011
Some Conference
OSCO
N 2011
Tales of failure to...
Document
Test
Verify
Imagine
Implement
Some Conference
OSCO
N 2011
Failure to document.
Some Conference
OSCO
N 2011
Moving Day
Thanks, David Prior!
Some Conference
OSCO
N 2011
Prevent documentation failures.
• Write documentation.
• Update documentation.
• Make documenting a step in your written process.
• Assign a fixed amount of time to that step.
Some Conference
OSCO
N 2011
Documentation tools
• Graphic designers. (Pretty wikis. Pretty docs. (Sphinx?) Diagrams.)
• Timelines.
• Bug tracking.
• Ordered todo lists.
Some Conference
OSCO
N 2011
Failure to test.
Some Conference
OSCO
N 2011
“My first day posing as a sysadmin (~1990, no previous training....) I deleted all zero length files on a Sun workstation.”
Some Conference
OSCO
N 2011
Prevent testing failures.
• Verify success criteria.
• Write tests.
• Test with a buddy.
• Have a plan.
Some Conference
OSCO
N 2011
Testing tools
• Your favorite test framework
• Repeatable shell scripts
• Staging environments
Some Conference
OSCO
N 2011
Failure to verify.
Some Conference
OSCO
N 2011
“What does ‘-d’ actually do?”
Some Conference
OSCO
N 2011
Prevent verification failures.
• Have a plan for things going wrong.
• Have a staging environment.
• Test your rollback plan, not just your implementation plan.
Some Conference
OSCO
N 2011
Verification tools
• Staging environments
• Your buddy
Some Conference
OSCO
N 2011
Failure to imagine.
Some Conference
OSCO
N 2011
For my group the bottom line was
"don't trust anyone".
Thanks, Maggie!
Some Conference
OSCO
N 2011
Recover from failures to imagine.
• Share your stories of failure.
• Talk with people who are different from you.
• Act out implementation scenarios.
Some Conference
OSCO
N 2011
Failure to implement.
Some Conference
OSCO
N 2011
Re-implement.
• Learn from mistakes.
Some Conference
OSCO
N 2011
Reflection.(or, the Post-Mortem)
Some Conference
OSCO
N 2011
Before
• Document the plan with numbered steps and a timeline.
• Test the plan and the rollback plan.
• Identify a “point of no return”.
Some Conference
OSCO
N 2011
During
• Screen sharing: UNIX screen, VNC, etc.
• Chatroom: AIM, Campfire (scrollback!)
• Voice: Campfire, Skype, VOIP, POTS call line
• Headsets!
• Designated time-keeper.
Some Conference
OSCO
N 2011
After
• Documentation updates
• Post-mortems to identify areas of success and areas for improvement.
• Limit improvements to 1-2 things.
Some Conference
OSCO
N 2011
Plan for the worst.Minimize risk.
Recover, gracefully.
Some Conference
OSCO
N 2011
Thanks!
Some Conference
OSCO
N 2011
Photo credits
• Flickr: sheepguardingllama
Recommended