Regulated Software Testing - Griffin Jones - TISQA 2014

RegulatedSoftware Testing

March 2014 © 2014 Congruent Compliance LLC 1

Griffin JonesConsultantAgile / Testing / Regulatory


Why Testing Is Not Dead … … In this Context

• Because there are “Survival Rules” associated with these systems• these systems need strong “harsh tests” (in a Karl Popper way)

• not just simplistic, mindless checking


THIS PRESENTATION …

Outline• What/Why/How Software is Regulated• 11 Surprising Aspects of Regulated SW• Same as Unregulated• Different from Unregulated

• Cognitive Dissonance• Questions


WHAT IS REGULATED

SOFTWARE?

Working Definition• Software subject to review by an internal or external regulatory body

WREST (Workshop on Regulated Software Testing)

• Share ideas and provide a forum for people who are interested in improving the testing of regulated systems


WHY I CARE

AND WHY YOU SHOULD

Authorization• Criminal, Civil, and Administrative Law

The Regulators are Police• Inspect, Search, Question, Confiscate,

Fine, Debarment• Deputies• Self-Policing


WELCOME TO WONDERLAND

“Skill, Pill, and Will” • Consequences• Scrutiny of Your Work• Moral Hazard• “Just World” Hypothesis• Good decisions can still have bad outcomes


BUT … SURPRISE!

cts of Software Testing• Some Aspects are Surprisingly Similar• Some Aspects are Surprisingly Different

Unregulated versus Regulated, Aspects of Software Testing


11 ASPECTS OF

SOFTWARE TESTING

SurprisinglySimilar

SurprisinglyDifferent


ASPECT 1 A …

Test Design and Execution• Schools of Testing: Quality and Standards

• Test Design Patterns [http://kaner.com/?p=100]

• Checking / Demonstration of Requirements


ASPECT 1 B …

Test Design and Execution• Failure Mode and Effects Analysis

• Problem Investigation


ASPECT 2 …

Tools and Automation• Commercial Tools• Open-Source• Customized Jigs• Testability Built into the Product• Lack of Tools


Accuracy and Rigor• “If it is not documented, it didn’t happen”• Attention to Detail is a Tell• Seriousness, Under Control, Honest, Professional• A Proactive, Self-Healing Culture

ASPECT 3 …


ASPECT 4 …

Traceability and Authorization• Under Control• Accountability


ASPECT 5 …

Evidence and Record Keeping• Reasonably recreate “the project” [Design History File]

• Retained for expected life of the device (after last sale), plus two years


ASPECT 6 …

The Halting Problem• How and on what basis does someone decide they have enough information to stop testing?


ASPECT 7 …

Process Standardization and Predictability• Process Police• Measurement Obsession• Mechanization• Reductionism versusHolistic System Thinking


ASPECT 8 …

Human Variability and Adaptability• Ordinary, flawed people• Their water also boils at 100 oC• Dependent on their skill, experience and judgment


ASPECT 9 …

Technology Choices• Old but well understood technology; or• Just invented technology


ASPECT 10 …

Morality and Ethics• Harms Innocent and Vulnerable• Destroys the Business• Becomes Public• Not obvious: Online Games• “Can you handle the Truth?”


ASPECT 11

Business Pressures• Competitive Markets• Reimbursement Codes• Regulated Marketing• Regulatory Uncertainty


THE BIG TAKE AWAY …USE COGNITIVE DISSONANCE

When working on Unregulated SW …• I constantly ask myself: “Would we be doing this for regulated SW?”• Reconsider the purpose and form of activities where you answer “NO!”


… THE BIG TAKE AWAY

USE COGNITIVE DISSONANCE

When working on Regulated SW …• I constantly ask myself: “What basic unregulated industry practices are we are not doing?”• Reconsider adopting those practices• How will you justify not doing them?


Regulated Software Testing• What/Why/How Software is Regulated• 11 Surprising Aspects of Regulated SW• Same and Different from Unregulated• Test Design and Execution• Tools and Automation• Accuracy and Rigor

SUMMARY … A


Regulated Software Testing• 11 Surprising Aspects of Regulated SW• Traceability and Authorization• Evidence and Record Keeping• The Halting Problem• Process Standardization and Predictability

SUMMARY … B


Regulated Software Testing• 11 Surprising Aspects of Regulated SW• Human Variability and Adaptability• Technology Choices• Morality and Ethics• Business Pressures

• Cognitive Dissonance

SUMMARY … C


QUESTIONS AND STORIES


IMAGE CREDITS

http://www.morguefile.com/archive/#/?q=target&sort=pop&photo_lib=morgueFilehttp://www.morguefile.com/archive/#/?q=old%20technology&sort=pop&photo_lib=morgueFilehttp://www.morguefile.com/archive/#/?q=key&sort=pop&photo_lib=morgueFilehttp://www.flickr.com/photos/bexross/2636921208/in/photostream/http://en.wikipedia.org/wiki/File:HAL9000.svghttp://upload.wikimedia.org/wikipedia/commons/a/af/All_Gizah_Pyramids.jpghttp://upload.wikimedia.org/wikipedia/commons/9/96/Waymarker_at_Southern_Upland_Way.JPGhttp://en.wikipedia.org/wiki/File:Painted_blaze.JPGhttp://www.morguefile.com/archive/#/?q=rubber%20duck&sort=pop&photo_lib=morgueFilehttp://www.flickr.com/photos/minnesotahistoricalsociety/5494632378/sizes/o/in/photostream/http://www.ebay.com/itm/Star-Trek-Original-Series-Science-Tricorder-Replica-/190807969198?_trksid=p2054897.l4276http://upload.wikimedia.org/wikipedia/en/7/72/Alicesadventuresinwonderland1898.jpghttp://upload.wikimedia.org/wikipedia/commons/7/78/Paris_2010_-_Le_Penseur.jpghttp://en.wikipedia.org/wiki/File:Keyboard_typing.pnghttp://upload.wikimedia.org/wikipedia/commons/b/bc/Library_of_Ashurbanipal_The_Flood_Tablet.jpghttp://en.wikipedia.org/wiki/File:Storage_containers_in_Svalbard_Global_Seed_Vault_01.jpghttp://upload.wikimedia.org/wikipedia/commons/1/19/Sevens_scrum.jpghttp://en.wikipedia.org/wiki/File:July_4_crowd_at_Vienna_Metro_station.jpghttp://blogs.msdn.com/b/geektester/archive/2010/12/30/life-saver-or-life-taker-therac-25-impact-of-poor-testing-testing-tragedies-1-learning-from-past.aspxhttp://en.wikipedia.org/wiki/File:Jack-in-the-box.jpghttp://www.wrestworkshop.com/

Thank you for attending this session.Please fill out the evaluation form.

Griffin Jones

[email protected]
