Griffin JonesConsultantAgile / Testing / Regulatory
March 2014 © 2014 Congruent Compliance LLC 2
Why Testing Is Not Dead … … In this Context
• Because there are “Survival Rules” associated with these systems• these systems need strong “harsh tests” (in a Karl Popper way)
• not just simplistic, mindless checking
March 2014 © 2014 Congruent Compliance LLC 3
THIS PRESENTATION …
Outline• What/Why/How Software is Regulated• 11 Surprising Aspects of Regulated SW• Same as Unregulated• Different from Unregulated
• Cognitive Dissonance• Questions
March 2014 © 2014 Congruent Compliance LLC 4
WHAT IS REGULATED
SOFTWARE?
Working Definition• Software subject to review by an internal or external regulatory body
WREST (Workshop on Regulated Software Testing)
• Share ideas and provide a forum for people who are interested in improving the testing of regulated systems
March 2014 © 2014 Congruent Compliance LLC 5
WHY I CARE
AND WHY YOU SHOULD
Authorization• Criminal, Civil, and Administrative Law
The Regulators are Police• Inspect, Search, Question, Confiscate,
Fine, Debarment• Deputies• Self-Policing
March 2014 © 2014 Congruent Compliance LLC 6
WELCOME TO WONDERLAND
“Skill, Pill, and Will” • Consequences• Scrutiny of Your Work• Moral Hazard• “Just World” Hypothesis• Good decisions can still have bad outcomes
March 2014 © 2014 Congruent Compliance LLC 7
BUT … SURPRISE!
cts of Software Testing• Some Aspects are Surprisingly Similar• Some Aspects are Surprisingly Different
Unregulated versus Regulated, Aspects of Software Testing
March 2014 © 2014 Congruent Compliance LLC 8
11 ASPECTS OF
SOFTWARE TESTING
SurprisinglySimilar
SurprisinglyDifferent
March 2014 © 2014 Congruent Compliance LLC 9
ASPECT 1 A …
Test Design and Execution• Schools of Testing: Quality and Standards
• Test Design Patterns [http://kaner.com/?p=100]
• Checking / Demonstration of Requirements
March 2014 © 2014 Congruent Compliance LLC 10
ASPECT 1 B …
Test Design and Execution• Failure Mode and Effects Analysis
• Problem Investigation
March 2014 © 2014 Congruent Compliance LLC 11
ASPECT 2 …
Tools and Automation• Commercial Tools• Open-Source• Customized Jigs• Testability Built into the Product• Lack of Tools
March 2014 © 2014 Congruent Compliance LLC 12
Accuracy and Rigor• “If it is not documented, it didn’t happen”• Attention to Detail is a Tell• Seriousness, Under Control, Honest, Professional• A Proactive, Self-Healing Culture
ASPECT 3 …
March 2014 © 2014 Congruent Compliance LLC 13
ASPECT 4 …
Traceability and Authorization• Under Control• Accountability
March 2014 © 2014 Congruent Compliance LLC 14
ASPECT 5 …
Evidence and Record Keeping• Reasonably recreate “the project” [Design History File]
• Retained for expected life of the device (after last sale), plus two years
March 2014 © 2014 Congruent Compliance LLC 15
ASPECT 6 …
The Halting Problem• How and on what basis does someone decide they have enough information to stop testing?
March 2014 © 2014 Congruent Compliance LLC 16
ASPECT 7 …
Process Standardization and Predictability• Process Police• Measurement Obsession• Mechanization• Reductionism versusHolistic System Thinking
March 2014 © 2014 Congruent Compliance LLC 17
ASPECT 8 …
Human Variability and Adaptability• Ordinary, flawed people• Their water also boils at 100 oC• Dependent on their skill, experience and judgment
March 2014 © 2014 Congruent Compliance LLC 18
ASPECT 9 …
Technology Choices• Old but well understood technology; or• Just invented technology
March 2014 © 2014 Congruent Compliance LLC 19
ASPECT 10 …
Morality and Ethics• Harms Innocent and Vulnerable• Destroys the Business• Becomes Public• Not obvious: Online Games• “Can you handle the Truth?”
March 2014 © 2014 Congruent Compliance LLC 20
ASPECT 11
Business Pressures• Competitive Markets• Reimbursement Codes• Regulated Marketing• Regulatory Uncertainty
March 2014 © 2014 Congruent Compliance LLC 21
THE BIG TAKE AWAY …USE COGNITIVE DISSONANCE
When working on Unregulated SW …• I constantly ask myself: “Would we be doing this for regulated SW?”• Reconsider the purpose and form of activities where you answer “NO!”
March 2014 © 2014 Congruent Compliance LLC 22
… THE BIG TAKE AWAY
USE COGNITIVE DISSONANCE
When working on Regulated SW …• I constantly ask myself: “What basic unregulated industry practices are we are not doing?”• Reconsider adopting those practices• How will you justify not doing them?
March 2014 © 2014 Congruent Compliance LLC 23
Regulated Software Testing• What/Why/How Software is Regulated• 11 Surprising Aspects of Regulated SW• Same and Different from Unregulated• Test Design and Execution• Tools and Automation• Accuracy and Rigor
SUMMARY … A
March 2014 © 2014 Congruent Compliance LLC 24
Regulated Software Testing• 11 Surprising Aspects of Regulated SW• Traceability and Authorization• Evidence and Record Keeping• The Halting Problem• Process Standardization and Predictability
SUMMARY … B
March 2014 © 2014 Congruent Compliance LLC 25
Regulated Software Testing• 11 Surprising Aspects of Regulated SW• Human Variability and Adaptability• Technology Choices• Morality and Ethics• Business Pressures
• Cognitive Dissonance
SUMMARY … C
March 2014 © 2014 Congruent Compliance LLC 27
IMAGE CREDITS
http://www.morguefile.com/archive/#/?q=target&sort=pop&photo_lib=morgueFilehttp://www.morguefile.com/archive/#/?q=old%20technology&sort=pop&photo_lib=morgueFilehttp://www.morguefile.com/archive/#/?q=key&sort=pop&photo_lib=morgueFilehttp://www.flickr.com/photos/bexross/2636921208/in/photostream/http://en.wikipedia.org/wiki/File:HAL9000.svghttp://upload.wikimedia.org/wikipedia/commons/a/af/All_Gizah_Pyramids.jpghttp://upload.wikimedia.org/wikipedia/commons/9/96/Waymarker_at_Southern_Upland_Way.JPGhttp://en.wikipedia.org/wiki/File:Painted_blaze.JPGhttp://www.morguefile.com/archive/#/?q=rubber%20duck&sort=pop&photo_lib=morgueFilehttp://www.flickr.com/photos/minnesotahistoricalsociety/5494632378/sizes/o/in/photostream/http://www.ebay.com/itm/Star-Trek-Original-Series-Science-Tricorder-Replica-/190807969198?_trksid=p2054897.l4276http://upload.wikimedia.org/wikipedia/en/7/72/Alicesadventuresinwonderland1898.jpghttp://upload.wikimedia.org/wikipedia/commons/7/78/Paris_2010_-_Le_Penseur.jpghttp://en.wikipedia.org/wiki/File:Keyboard_typing.pnghttp://upload.wikimedia.org/wikipedia/commons/b/bc/Library_of_Ashurbanipal_The_Flood_Tablet.jpghttp://en.wikipedia.org/wiki/File:Storage_containers_in_Svalbard_Global_Seed_Vault_01.jpghttp://upload.wikimedia.org/wikipedia/commons/1/19/Sevens_scrum.jpghttp://en.wikipedia.org/wiki/File:July_4_crowd_at_Vienna_Metro_station.jpghttp://blogs.msdn.com/b/geektester/archive/2010/12/30/life-saver-or-life-taker-therac-25-impact-of-poor-testing-testing-tragedies-1-learning-from-past.aspxhttp://en.wikipedia.org/wiki/File:Jack-in-the-box.jpghttp://www.wrestworkshop.com/
Thank you for attending this session.Please fill out the evaluation form.
Griffin Jones
March 2014 © 2014 Congruent Compliance LLC 28