Embed Size (px)
DESCRIPTION
For the last four years myself and my colleagues at RainStor have been evolving a process for testing a structure data archiving system in an Agile development environment. In this talk I will discuss the evolution of a team from a rudimentary Agile implementation on an unreleased product, to our current process which uses the fundamental elements of Specification By Example to successfully deliver software functionality across 30 different platform/backend configurations to a series of high profile and demanding customers. Last year our company was used as a case study for successful implementation in Gojko Adzic's book on Specification By Example. My report will discuss the lessons learned during the early implementation and the challenges faced in moving away from a compressed waterfall approach. Through a process of incremental change we have identified and tackled the fundamental issues that undermined the development effort as a team. I’ll describe some of the mistakes made in attempting to implement a more formal process of requirements documentation into an Agile implementation and the benefits we uncovered on moving to a more flexible user story based approach. I’ll also discuss some of the issues around trying to implement user stories in a server system with no GUI and very technical and performance based requirements. Raising the importance of quality and the status of testing both within the development team and the organisation as a whole has allowed the challenges facing the team to be recognised and respected. The result has been a more collaborative approach taken between developers and testers both through “collaborative specification” of user stories and tackling the problems that impact the delivery of value to the customers. I also plan to discuss how we’ve expanded from documenting acceptance criteria for each user story such that we now document Criteria, Assumptions and Risks for each feature and, rather than a ‘Done/Not Done’ approach how we identify the confidence in each of these categories to measure the confidence we have in each new feature being implemented. Having the test team as an involved and influential team through the entire development process has also allowed us to implement a number of testability features to help to make the product more testable. I will discuss the benefits of having development understand and prioritise testability issues with some illustrative examples. I will discuss the challenges and benefits of developing our own metadata driven test harnesses as opposed to an off the shelf solution. I’ll detail how having control over these harnesses has allowed us to work towards a self documenting test system using realistic customer examples as “Automated Specifications” of the RainStor system allowing us to explain current behaviour to Product Management in terms of well understood customer scenarios.
Citation preview
An Evolution Into Specification An Evolution Into Specification By ExampleBy Example
Adam KnightAdam Knight
Twitter: adampknightTwitter: adampknight
Two ConversationsTwo Conversations
• Zachary KnightZachary Knight
““Can we see Can we see the picture the picture of the of the Monkey Monkey turning into turning into a Man?”a Man?”
• Gojko AdzicGojko Adzic
““Although their Although their process has process has almost all the almost all the elements of SBE, elements of SBE, they just think of they just think of it as their it as their homegrown way homegrown way of developing of developing software”software”
Evolution - ContextEvolution - Context
• Successful Strategy Successful Strategy Depends on ContextDepends on Context
• ProductProduct API/CLI interfacesAPI/CLI interfaces Scalable Parallel Scalable Parallel
ProcessingProcessing SQL ParserSQL Parser
• MarketMarket OEM PartnersOEM Partners Varied Varied
implementationsimplementations Traditional Delivery Traditional Delivery
ModelsModels
Evolutionary CyclesEvolutionary Cycles
• It takes time to adapt to changeIt takes time to adapt to change• Methodology is an environmental factorMethodology is an environmental factor• Faster feedback cycles encourage evolutionFaster feedback cycles encourage evolution
Requirements and SpecificationsRequirements and Specifications
• Included Functional DesignIncluded Functional Design• Abstracted out Customer DetailAbstracted out Customer Detail• Not maintainedNot maintained• Not completed in sprintNot completed in sprint
Requirements Documents Requirements Documents
User StoriesUser Stories
• Developer Developer ReservationsReservations
• Hard to break downHard to break down• Still Solution BasedStill Solution Based
• Value drivenValue driven• IndependentIndependent• RecursiveRecursive
Challenging RequirementsChallenging Requirements
• Rejecting SolutionsRejecting Solutions
• Requesting ExamplesRequesting Examples
• Raising Testing ProfileRaising Testing Profile
Collaborative SpecificationCollaborative Specification
• Defining SuccessDefining Success Acceptance Acceptance
CriteriaCriteria RisksRisks AssumptionsAssumptions
• Discussing ExamplesDiscussing Examples
• Breaking the Breaking the ModelModel
Test ApproachTest Approach
Early Test ManagementEarly Test Management
• Poorly selected TCM Poorly selected TCM tooltool
• Excel with Test Case Excel with Test Case EstimatesEstimates
Counting test cases is Counting test cases is inappropriateinappropriate
Exploratory TestingExploratory Testing
• Session Based ETSession Based ET Run times too longRun times too long Can’t parallelise Can’t parallelise
sessionssessions
• Thread Based ETThread Based ET Estimate chartersEstimate charters
• Reporting ConfidenceReporting Confidence
Test AutomationTest Automation
Automation PrinciplesAutomation Principles
• IncrementIncremental Growthal Growth
• DRYDRY
• User Input User Input DrivenDriven
• TestabilityTestability
Towards Living DocumentationTowards Living Documentation
• Self DocumentingSelf Documenting
• Reporting in Business Reporting in Business TermsTerms
• Specification for Specification for RefactoringRefactoring
Current PositionCurrent Position• Around100 End Customer installationsAround100 End Customer installations• Highly Rated Testing/Support TeamHighly Rated Testing/Support Team• ““The product was installed seamlessly and The product was installed seamlessly and
the customer was delighted.”the customer was delighted.”
RecapRecap
It takes time to adapt to It takes time to adapt to changechange
FROMFROM TOTO
Formal Formal RequirementsRequirements
User Stories, User Stories, Collaboratively Specified Collaboratively Specified using Examplesusing Examples
Test Case Test Case ManagementManagement
Exploratory ThreadsExploratory Threads
Bespoke Bespoke Automation ScriptsAutomation Scripts
Self Documenting File Self Documenting File Driven Test AutomationDriven Test Automation
““Improbably excellent teams Improbably excellent teams become so through a sustained become so through a sustained
series of entirely probable small series of entirely probable small improvements ”improvements ”
Jason GormanJason GormanBletchley Park : Bletchley Park :
www.bletchleypark.org/content/contact/donation/supwww.bletchleypark.org/content/contact/donation/support.rhtmport.rhtm
ThanksThanks
• Email: Email: [email protected]• Twitter: adampknightTwitter: adampknight• Blog: http://www.a-sisyphean-task.comBlog: http://www.a-sisyphean-task.com
• Email: [email protected]: [email protected]• WebSite: http://www.rainstor.comWebSite: http://www.rainstor.com
ReferencesReferences• Further ReadingFurther Reading
Specification By Example – Gojko Adzic ; ISBN Specification By Example – Gojko Adzic ; ISBN 9781617290084 9781617290084
Thread Based Test Management – James Bach Thread Based Test Management – James Bach http://www.satisfice.com/blog/archives/503http://www.satisfice.com/blog/archives/503
• ImagesImages All images in public domain or under Wikimedia or All images in public domain or under Wikimedia or
Creative Commons LicenseCreative Commons License 1&14 © BIODIDAC 2004 ; 2 Steveoc 86 ; 3 – Pangolin 1&14 © BIODIDAC 2004 ; 2 Steveoc 86 ; 3 – Pangolin
Hubert Ludwig ; 4 - Rafael Brix ; 5 - Hubert Ludwig ; 4 - Rafael Brix ; 5 - www.palaeocritti.com ; 6 Brachiosaurus - Богданов www.palaeocritti.com ; 6 Brachiosaurus - Богданов [email protected] ; 7 - Matt Martyniuk ; 8 – Public [email protected] ; 7 - Matt Martyniuk ; 8 – Public Domain ; 9 - Original uploader was Killdevil at Domain ; 9 - Original uploader was Killdevil at en.wikipedia ; 10 – Wikmedia commons no name ; 11&12 en.wikipedia ; 10 – Wikmedia commons no name ; 11&12 (unkown, extensively used in Public Domain); 13 – (unkown, extensively used in Public Domain); 13 – Yewenyi ; 15 - Richard Bartz, Munich Makro Freak & Yewenyi ; 15 - Richard Bartz, Munich Makro Freak & Beemaster Hubert Seibring ; 16 - Coniac PublishingBeemaster Hubert Seibring ; 16 - Coniac Publishing
