Agile for Infrastructure Applying Agile Software Development Practices to other IT Domains Terry Hamilton IASSIST Computing Services Inc

Agile for Infrastructure

Applying Agile Software Development Practices to other IT Domains

Terry HamiltonIASSIST Computing Services Inc.

http://www.iassist.ca

October 22, 2007 Agile for Infrastructure 2

Abstract

Presentation Title:Agile for Infrastructure : Applying Agile Software Development Practices to other

IT Domains

Speaker:Terry Hamilton, President of IASSIST Computing Service Inc.

Abstract:Agile Practices have shown their value for countless Software Development

projects across many fields and specialties. However Software Development is not the only IT endeavor that can benefit from Agile. Agile for Infrastructure is a demonstration of how the principles of Agile can be applied by Systems teams to deliver Agile Infrastructure to large scale projects.


Not SOFTWARE DEVELOPMENT

• Agile as discussed in this presentation: – NO CODE > Pure Server and Middleware rollout– NO DEVELOPERS > SysAdmin types

• Agile is applied to Infrastructure in many places– Boeing, Citibank, IBM, more : internal and engagements

• RFP’s and Contracts starting to require Agile• Young Agile was SWAT Teams, SkunkWorks, etc.

– Agile has same purpose as these tried & true workarounds– Simply applying the Lessons consistently


With repeatable action and tangible metrics grip!


Agile for Infrastructure – Case Study

• AGILE (Scrum) for large cutting edge SOA build– IBM zSeries, zOS, 200-400+ Linux on zSeries, VMWARE – 100’s of Web Services, 10k’s of users– 40+ Middleware latest versions across zOS and Linux on Z

• WebSphere (application layer), Tivoli (sys mgmt), Rational (development)

• Application Environments for SOA worth tens M$– Will replace 30yrs of COBOL, 5yrs for application transition– Transactions worth $250M/hour (~1500tps @ peak)– Client & Public facing applications with regulatory implications

Obstacles to Agile Adoption

Obstacles are Opportunities


Biggest Obstacle - Chicken & Egg

• How to build an infrastructure to run applications that have not been designed yet?

• IT Team: “Tell us what you need!”• Dev Team: “Tell us what we’ll get!”

– Large hardware buy-in costs = unwillingness to fire the starting shot!– Months of repeated estimating and planning and debate

• This alone sold Agile!– We promised to deliver tangible, working product,

month by month, and still be flexible to changing requirements from the

application teams!


Obstacle – New Skills & Legacy Staff

• New Infrastructure skills = becoming good with new Programming Lang.– Skills with this middleware stack are rare– Original Project plan had 25+ infrastructure person years! – Reality = Start Now! Deliver DEV environment in 6 months or less

• Legacy shop – years of cookie cutter solutions (CICS, DB2, zOS)– Same processes, people and applications for years and years and years

• “Lets start with 10 people for 1 month and see where we get.”– Hand picked 8 go-getters from the staff plus two contractors– Only Scrum Master had any Agile experience or training

• Had to break from the existing culture– “Here’s a Post-it Note – go set your own deliverables!!!” – Biggest fight = dedicated team members – Had to detach from application dependencies and processes


Obstacles – Existing Processes

• Process is an overloaded word– Many times a Process is really

• Desk Procedures only known to few individuals or a single team• Habit instead of structure

– Officially following CMMI Level 3 processes– But existing CMM supported Development, not infrastructure

• Years since anything but upgrades have been formally required

• You cannot bypass or ignore existing processes.– But we can work in a way that doesn’t engage them– We isolated the team and the work

• removing dependencies implicitly removed ill-fitting processes– Don’t avoid processes just to be independent or different

• Re-use what works (or what is needed even if painful)


Obstacles – Management Doubt

• Agile is so new it is scary– “Self Managed teams” is a political phrase – use it wisely.– Scrum Master had to accept risk of Ownership

• Team had Ownership but additionally SM was the face for that risk– Agile was being evangelized by Middle layer (Architects)

• Sold it UP to management and DOWN to technical staff

• Evangelize ruthlessly and bring in Experts for the outside view– Scott Ambler could sell Agile Snow at the North Pole

• “Had to allow oversight” (Agile demands it! We wanted it!)– Management updated at weekly Mgmt meetings

• Invited but yet to actually attend a Scrum or Planning (but that’s another issue)• Existing processes eventually drove estimates that were scarier than Agile

– Risk 22 days on Agile OR risk delivering late or never.

Implementing Agile

Our customized techniques“Localized Process Improvements”


Daily Scrum Meeting

• Implemented AS IS!• Typical difficulties:

– Get folks to agree to another meeting each day– Become Tech chats instead of 1 minute Updates

• Typical advantages:– Identify dependencies– Face time for Team Members– Early warning indicators– Sense of dependency between team members– Sense of urgency to the deadline


Deliver Working Systems

• Every 22 days delivered working middleware / servers– Maybe not “complete” but installed, running and usable

• Included Install docs, Standards, Hardened• Following iterations added tasks

– Configured & Performance tuned, automation– Document procedures, integrate with XYZ, ABC and MNO products

• “DONE” clearly defined to the team– Reviewed and approved by the Infrastructure Architect / Product Owner– Available to the team so available to review as required


Release & Iteration Planning

• Agile for Infrastructure approach:– Middleware & servers prioritized with Stakeholders before iterations– Evolve the Operational Model as we learn

• Leveraged Iteration 0 – hardware & software pre-decided• Iteration 0 design a cut/paste from “Patterns for eBusiness”• Refined by learning what was wanted, needed and could do• Surprisingly little re-work required

– Products not up to the marketing glossy but worked well

• Stories = Middleware & Servers (the Release Plan) • Tasks = Steps to document, install, configure (the Iteration

Plan)


User Stories

• We had none!– Application couldn’t to tell us what their requirements were!– Devised our own: A piece of Middleware or a Server = a Story

• We took holistic approach– Build it to Install & Planning Guides

• Redbooks are great resource for IBM products• Whitepapers – but not just any whitepapers• Best Practices – as documented or consult experts

– Case-by-case asks for Application Architect input• This input changed constantly as app designs floated

• Decisions / Assumptions must favor the generic– Choices should prefer open-ended solution to allow changes later

started with none.


Continuous Testing

• This was Tough! • How to do this with Middleware and Servers?

– A completely different concept in QA• Decided on two checkpoints:

– IVT – Install Verification Tests• Use “sample” apps that come with the products

– Peer Reviews• Architect (Product Owner) walkthru of docs and install• Integration between products a defacto Peer Review

• Openly acknowledged that we couldn’t be perfect– Some changes likely required when the apps arrived


Burndown Charts

• Implemented (mostly) AS IS!• Tracked interruptions as a separate work task

– 100% Dedication is not the same as 100% focused• Focus and mind share are required, not just time spent• Staff don’t acknowledge time spent on “small favours”• Skilled people like interruptions – reflects on their abilities• Go-To Person syndrome – “Jim always has the answer.”• Interruption here is Business As Usual

– Hours spent on interruptions were tracked by individual• Just like a regular task but with estimated 0 hours work


Burndown Charts

Panic! 5 days in and already 40hrs behind!By tracking Interruptions we reveal several problems:

1. Not a dedicated team

2. Staff absorb extra work without revealing it

3. Skills resided in specific individuals

4. People didn’t feel empowered to say No.


Self Managed Teams

• Often a tough sell – Even after you put politics and entrenched practices aside…– The hardest part of getting to the team to form

• You can’t form a team only a group; they do the rest with encouragement!

• Three sided:– Management needs to know the Agile Experiment is in good hands

• A 30 day iteration wasted is still waste even if short– Team needs to learn to trust itself and its members

• One loose cannon will put everyone on the defensive– Trust in the PO and SM to assist (“lead”) and protect team

• Old habits – these positions of authority are deferred to in crisis


Information Radiator / Kanban

• Go Big or Go Home – use a Task Room, not Board– All 4 walls of our meeting room covered in Post-it Notes

• TODO Wall, WIP Wall, DONE Wall, • ACCEPTED Wall and ISSUES Wall (yes, that is 5 walls)

– Information Blanket not Radiator – Can’t be missed! • Post-it® Notes – The Best Tool Ever!

– Interactive! Team creates, moves through states, signed when done– Usual boring kickoff became impassioned, engaged planning session

• Management stood at the door amazed : the usually docile staff debating, cajoling, negotiating, juggling the to-do list

• Wiki as a team room & repository– Collaborative, version control, attachments, easy access– Easy to learn, easy to maintain, looks good too


Information Radiator - Task Room

Iteration Backlog(products & servers todo)

Tasks / Storiescommitted

“Above the Line” uncommitted

TODO Wall

WIP Wall

Done Wall

Accomplishments

Our Successes and Failures


Successes

• Other projects starting to evaluate Agile– Imitation of the project because it was a success

• 1st Agile project and 1st Iteration by 1st Scrum Team– only 5% off estimated hours (after “interruptions” accounted for)

• 200+ Linux on zSeries servers including Middleware– On time : Estimated in January, on time in October– On budget: actually reduced dedicated team size over time– On quality: 2 months of use and 2 only mis-configurations (bugs) so far– PROD ready at least 2 months ahead of applications being ready for it


Successes

• Demonstrate ability to identify and manage risks and issues– The Burndown is Information

• Project plans, Gantts, %Complete don’t reveal information, only document facts

– Obstacles tracked daily via Scrum– Risks, deferred workload, assumptions on Wiki as they were found

• Sr. Mgmt presented to Executives on our success– Made Agile more visible throughout organization and beyond

• Users are satisfied, Team feels successful, Mgmt relieved– Stakeholders can then start their work happy


Failures

• Agile – That’s when you skip the documentation, right?– Highly visible but not highly understood

• Not intended as a training exercise but could leverage better• Depth of understanding limited to “the guys with the post-it project plan”

– Suddenly seeing “Iteration” instead of “Phase” on Waterfall plans• Iteration is obviously the cool new thing• “The server guys did it so we can do it”

• Training and Education– One Team now has experience with Agile and Retrospectives were

positive but need to grow the # of teams & individuals with knowledge.– Have not yet taken opportunity to train more Product Owners or Scrum

Masters, limiting adoption chances.


Failures

• Generalists– Cross-over knowledge gained but not enough to support

• Middleware products require too much depth• Skills, techniques, dependencies are specific

– Department structures hard to break• Structure separated skills, most returned to their traditional

groups• Transition to Business As Usual

– Still negotiating to show that Agile works on BAU support• Works for “one time” build cycle but unproven on support

day-to-day


Lessons

• Sponsor Absolutely required.– 2nd Level management or higher– Person authorized to dictate allocation of staff across departments

• Evangelists– Bring someone from outside to sell Agile – Scott Ambler works.– Have someone inside to promote and start/run it

• Start with Agile/Scrum Skeleton and customize to fit– RESIST the urge to fall back to traditional approaches

• Yes, Excel is nice. No, it is not a time tracking tool!– Don’t mess with the basics when you are starting out

• Iterations of fixed length < 30 days, Daily Standup meeting, Dedicated resources


Lessons

• Burn Down chart is your absolute best friend– Simplest tool but hard for uninitiated to understand – take the time to teach

• Form the team carefully– 1st Agile team are “picked volunteers”, not pure volunteers, not assigned– Seed the team with potential: tech skills aren’t everything a team needs

• Load the dice– On Project 1, Iteration 1, be conservative with estimates, limit committed

deliverables, allow time for teaching, do training in advance (iteration 0)– Management willing to “risk” agile will usually understand the value of

motivating the team on the first iteration– Do not cheat but make Iteration 1 a success by preparing it right


Q&AQ&A

Documents

Agile for Infrastructure Applying Agile Software Development Practices to other IT Domains Terry Hamilton IASSIST Computing Services Inc