Upload
steve-nunziata
View
307
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Large companies seeking to adopt Agile are often challenged in their ability to scale. Functional silos have led to an overt reliance on formal 'paperwork' artifacts to demonstrate software development processes are 'in control', and can meet internal and external standards for auditability. How can Agile help break the 'artifact trap' paradigm, while providing better quality?
Citation preview
Chaos, Consistency, Creativity:
A Journey Through Agile Auditability
Steve Nunziata, PMP, PMI-ACP, CSM, SAFe SPCAgile Austin Monthly Meeting, October 14th, 2014
About Steve…
http://www.linkedin.com/pub/steve-nunziata/5/519/484/
PMP, ACP, CSM, SAFe SPC
EDS, Nike, Adidas, USAA
Agile Trainer & Coach
New Jersey / Oregon
Bassist Extraordinaire
Alamo Agilistas / PMI
Background: My Story
Zero to Sixty (Days): Chaos to Consistency
So… Why Are We Here? Opportunity:
Educate internal auditors to evolve away from formal artifacts and accept Agile tenets of visibility and transparency to demonstrate adherence to defined Quality standards.
We will collaborate on an approach to define an Agile Risk & Control framework that can start you on your journey.
How Would You Like: A 50% - or more – reduction in project ‘paperwork’ to demonstrate adherence to compliance processes?
Waterfall Agile
59
30
Project Compliance
Arti facts
A framework for consistent application of Agile practices and ceremonies across a large – and growing – organization?
SAMPLE: YMMVReally?
Remember…Use the Force
Remove, you must, Stories from the Backlog,
That, within an Iteration, completed, will not be…
AgendaCh
aos Failings of
Today’s Risk Management Processes
Cons
iste
ncy Why Audit
Execution Models Need to Evolve Cr
eativ
ity Creating an Agile Auditable Framework
Managing Risk – How Important is it?
The primary goal of a business is to… stay in business.
It is therefore necessary to continually evaluate, monitor, and address threats to retain market share. Otherwise, what would happen?
Managing Risk – The Risk Management Process
Risk Identificatio
n
Risk Assessment
Risk Response Risk Review
Managing Risk – ISO 9001 Summary
Part 4 – The Company must establish, document, and maintain a Quality Management System (QMS)
Part 5 – Management commitment in evidence for the QMS
Part 6 – Necessary resources must be determined & provisioned
Part 7 – Plan & Develop processes for product realization. The processes must produce documents that can be (1) reviewed for acceptance; and (2) used as proof of conformance
Part 8 – All reports of non-conformances, both of the product or the process, shall be reported upon, analyzed and lead to corrective action
Managing Risk – Risk & Control Compliance Framework
Risk Controls
Control Tests
Reporting & Review
Operational Risks
Incomplete Requirements Ineffective or Incomplete
Software Solution Poor User Experience Poor Project Execution
Plan
Formal Requirements Baseline Process
Project Execution Schedule Review
Code Peer Reviews
Evidence of Formal Signoffs
Published Meeting Minutes
Documented Decisions / Logs
Formal results of Audit published for review; opportunities for improvements noted
Auditors
Are Risk Management Processes Inherently anti-Agile?
Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif
SDLC & Process Audit Execution Models: Challenges
While Agile adoption and evolution has continued unabated over the past several years, traditional process audits have largely been unable to keep pace. Why might this be?
SDLC & Process Audit Execution Models
Req’s Analysis Design Build Test Deploy
Systems Development Life Cycle – Linear View
SDLC & Process Audit Execution Models
Source: http://julianeverett.wordpress.com/
Red Dotted Line: Waterfall
Blue Dotted Line: Agile
RISK
TIMEProject Risk Profile – Agile & Waterfall
SDLC & Process Audit Execution Models
Daily
24 H
ours
Iteration
2-4 Weeks
Release
~3 Months
Closure
~9-12 Months
SDLC Execution – Waterfall, Incremental, & Agile
SDLC & Process Audit Execution Models
Process Audit vs. SDLC Execution Gap Analysis
Closure
~9-12 Months
Release
~3 Months
Iteration
2-4 Weeks
Daily
24 H
ours
SDLC & Process Audit Execution Models
DailyIteration
2-4 Weeks
Release
~3 Months
Closure
SDLC and Process Audit Execution: Optimal Quality State
5 Steps to Establishing an Agile Auditable Framework
Risk Validation
Inventory Agile Practices
Create Acceptable Parameters
Determine Method of Control
Establish Operational Parameters
1
2
3
4
5
5 Steps to Evolving an Agile Auditable Framework
Risk Validation
Review and Validate the current Risk & Control Framework, ensuring traceability from Risks to Controls to Control Tests.
Operational Risk: Risk Control: Control Test:
Failure to Manage Project Risks
Risk Management Process
Evidence of a Periodic Risk Review (Risk Log)
Issue Management Process
Formal, Complete Issues Log
1
5 Steps to Evolving an Agile Auditable Framework
Inventory Agile Practices
Inventory the Agile Practices supported by the organization. Scrum practices and ceremonies provide a good start.
Match the Agile ceremonies to the list of Risks in the current Risk & Control Framework. Can a Ceremony or Practice provide an acceptable substitute? How / Why?
2
5 Steps to Evolving an Agile Auditable Framework
Inventory Agile Practices
Introduce the Agile Practice as a Control. Could it work? Could it be effective? What would be the value of the current control set – should anything remain, or can they be dismissed?
Operational Risk: Risk Control: Control Test:
Failure to Manage Project Risks
Risk Management Process
Evidence of a Periodic Risk Review
Agile Daily Standup
2
5 Steps to Evolving an Agile Auditable Framework
Create Acceptable Parameters
Research Industry standard ‘best practices’ for the ceremonies or practices you plan on using as a Control (mitigation strategy) for the Risk. A great example is Version One’s The Agile Checklist
Create a matrix defining minimally acceptable behaviors, along with anti-patterns, and radiate the desired outcomes in a common area
3
5 Steps to Evolving an Agile Auditable Framework
Create Acceptable Parameters
Agile Ceremony: Daily Standup
Best Practice Acceptable Partial Unacceptable
Occurs 5 Days per Week
Occurs 4 Days per Week
Occurs 3 Days per Week
Occurs <3 Days per Week
3 Core Questions Addressed
3 Core Questions Addressed
<3 Core Questions Addressed
<3 Core Questions Addressed
…Your Organization?
…Your Organization?
…Your Organization?
….Your Organization?
3
5 Steps to Evolving an Agile Auditable Framework
Determine Method of Control
Does the new Control Test require someone observe an Agile Ceremony, or is there a consistent formal artifact from an Agile practice that can be viewed?
4
5 Steps to Evolving an Agile Auditable Framework
Establish Operational Parameters
Review the total number of Control Tests. How many require observation from an Auditor?
Establish the Audit cycle & reporting time (Weekly? Sprint Level? Release Level? Other..?)
Train and deploy Audit resources
Execute an Audit cycle… and report to Risk Owners
Learn… and continue to evolve!
5
5 Steps to Evolving… Creativity
Host a Retrospective Ceremony with some of the Agile teams to uncover: What may be challenging teams in conforming to minimal standards? What opportunities can they recommend to evolve to controls? Are the audits providing value in holding roles accountable for their deliverables? Finally – when minimal standards are easily achieved – it’s time to take the next steps in maturity, and shift the pattern.
5 Steps to Evolving - Going Beyond... Challenge: can you evolve traditional, formal artifacts into a
more Agile framework? How can you continuously improve?
Picture Source: http://agile101.wordpress.com/2009/07/27/agile-risk-management-assessing-risks-step-2-of-4/
Positive Outcomes Better alignment of Controls and Tests to the project execution model
Real time, actionable feedback & reporting to teams and Risk owners
Scalable for future methodologies & practices
Continual quality assessments; a project can have multiple reviews
Sets a benchmark for Agile maturity across an Organization
‘Humanizes’ the Audit (not ‘check the box’) – gives teams a voice
Experience – 50% reduction in Controls… while doubling Quality
Leading – NOT lagging – metric; address problems before they manifest
Opportunity for two-way communication and learnings
Challenges Optimal model is labor intensive Inherent subjectivity in assessments (‘Auditor Bias’) Potential for teams to feel ‘over controlled’ Oversight and administration of the process Communication and support for changes Determining boundaries of adherence vs. non-adherence, and appropriate remedies Ever-evolving process; can feel like an ‘arms race’
Common Questions Does this model Scale? How much time per week would this require? Isn’t this just the Scrum Master’s… or (insert role here) – job? Could we use Pair Programming as a Control? What is the future of Agile Quality Assurance?
Objectives Met?
Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif
Remember: Auditors are the Board of Health!
Questions?
Thank You!
Information Sources Malik Imran Ullah & Waqar Ali Zaidi, “Quality Assurance Activities in Agile – Philosophy to Practice”. Sep. 2009.
Larry Whittington, “ISO9001:2008 Requirements Summary in Plain English”. http://www.whittingtonassociates.com/
Tor Stalhane, Geir Kjetil Hanssen, “The application of ISO 9001 to Agile Software Development”. 2008.
Buck Kulkami, “Agile Projects: An Emerging Challenge for IT Auditors”.
Information Sources R. Gopinath, “Guideline: How to Audit and Agile Project?”
George Schlitz, “Is your Agile Audit and Compliance Process really Agile?”
Christelle Scharff, “Guiding Global Software Development Projects using Scrum and Agile with Quality Assurance”