HCMDSS Workshop Certification & RequirementsWorking Group Report (WG 6)

June 2-3, 2005Philadelphia, PA

John Hatcliff, Kansas State University (hatcliff@cis.ksu.edu)

Participants

Peter Coronado -- Varian Medical Systems

Steve Dimmer – Calypso Medical Technologies

Jacob Flanz – Mass General Hospital

Elsa L. Gunter -- University of Illinois at Urbana – Champaign

John Hatcliff -- Kansas State University

Mark P. Jones -- Oregon Health & Science University (OGI)

Paul L. Jones -- FDA/CDRH/OSEL

Brad Martin -- National Security Agency

Rosemary C. Polomano -- University of Pennsylvania School of Nursing

Stacy Prowell -- University of Tennessee

John Rushby -- SRI Rick Schrenker --

Massachusetts General Hospital

Oleg Sokolsky -- University of Pennsylvania

State of the Art-- What We Do Well

Develop and approve stand-alone medical devices based on mature technology with moderate complexity

well-understood domain with a lot of experience we have a lot preceding examples with a lot of experience we know how to gather requirements, code, test, do hazard

analysis, etc. in a reasonably good way Advanced development techniques such as model-driven

development and software product-line architectures are working well in specific device domains (e.g., Guidant pace makers)

Established mechanisms for reporting, tracking, and publicizing device related “incidents” (analogous to reporting air plane crashes)

Light-weight formal methods (e.g., ESC Java) have been effective in security evaluations

in non-regulated or certified settings, tremendous progress has been made (Intel, Microsoft, etc.)

State of the Art—Challenges Within Bounds of Existing Technology

We know how to… test, write requirements, prototype, etc. but how we can do this more accurately, in a

more automated way, and achieve greater cross-leveraging of artifacts

Large-scale, complex devices still pose challenges e.g., proton therapy facility 100,000’s of validation procedures, test

cases …validation burden (time/costs) slows time to

market

State of the Art—Challenges Within Bounds of Existing Technology

System integration for devices based on existing technology We are doing pretty well with integrating

products developed by a single manufacturer, e.g., Varian (Linac, connected to imaging device, treatment machine)

But there are clinical cases where people are incorporating devices from other manufacturers (currently tech people must be trained to recognize potential incompatibilities)

even something as simple as incorporating tubing into pump (which were both approved separately) malfunctioned in clinical setting

State of the Art –The “Heroic” Clinical Engineer In general lack of a systems view or systems model that

would enable one to reason about clinical environments, processes, and ad hoc integration of devices

“Clinical Engineers” are faced with the task of deploying collections devices in ad-hoc ways in constantly changing devices

some devices interact directly some device interact through patients some devices with even looser coupling research on formalizing clinic environment, medical process

that accommodates variability across different environments

Devices being delivered with little or no, or inappropriate documentation (especially integration documentation)

Incorporate inter-patient variability of (e.g., very obese patients) in requirements for monitoring devices, etc.

Research Challenge --Requirements Research More automated and rigorous manner

for eliciting, capturing, and leveraging requirements

Support semantics querying or simulation of use cases directly from requirements

More effective automated derivation of test cases from requirements

Automated generation of visualizations and natural language description from requirements

Modeling/Formalization of Clinical Environments and Processes Trend: Proliferation of sophisticated devices in

complex environments where exceptions are the norm

Clinical engineers currently act as informal “system integrators” of devices

Must develop a more rigorous approach to capturing, visualizing, reasoning about clinical environment & processes

How to incorporate expected variability within clinical environments?

Can we mine these rigorous descriptions (e.g., of processes, workflows, etc) to guide device requirements development or detect problematic device interaction within a clinical environment?

Can we use these models to clarify goals and focus the tasks to achieve more rigorous clinical validation?

Re-orienting Current Procedures for Component-wise Certification Components as commodity

framework that creates a business environment that produces a variety of re-usable components are marketed

Certification effort should focus on validity of composing components certification artifacts should be reused (integrator should not have the

burden of re-certifying component internals) want to be able to synthesize certification of assembly from certification of

components Recognize the differences in integration scenarios

Commodity components purchased vendor to produce a plug-n-play system with a suite of components

Components purchased by hospitals and assembled by clinical engineers (should their be any oversight of such engineering?)

Requirements that capture all important aspects of interfaces functionality quality-of-service safety/faults

Elevating methods for, e.g., protocol validation to a system of systems level

Inspiration from but looking beyond RTCA Integrated Modular Avionics

Certifying In the Presence of Change/Variation/Evolution Frequent software patches Move from “software product lines” to

“certification product lines” FDA recognition of tools the perform

impact analysis and traceability to reduce the effort in retesting/re-certification

“Make” for certification More sophisticated tools for managing

certification artifacts and for automatically detecting what things that need to be rework in the presence of change

Problems with Existing Technology Reporting, tracking, and publicizing, and

interpreting device “malfunctions” (distinct from “incidents”) across manufacturers is problematic need mechanisms for reducing the burden

of re-certification impact/change analysis tools, having FDA

recognize artifacts produced by such tools some devices incorporate “flight recorder”

technology (e.g., Varian Linac) open up certification resources for looking

at bug-fixes, evolution of software

Dealing Effectively With Security Certification Issues Trend: desire to integrate a variety of device types

across a variety of network infrastructures Network types:

national, regional, local, home, body-area (wired, wireless), Trend: Blending of medical devices and medical

informatics systems Devices that stream data (pace makers) into medical

records that may be mined for broader clinical studies Trend: Competing concerns

security, availability, … competing quality of service when devices deployed on the

same framework (leading to the possibility of denial of life-giving services)

Need to investigate and vet security standards such as Common Criteria (most stringent levels require formal methods) and MILS architecture

Certification in Context ofData Communication Ontology of medical information Common language framework for

exchanging a variety of forms of data how does this relate to DICOM, HL7, etc.?

How do we certify that systems maintain the integrity of the data?

What’s keeping IEEE 1073 from being implemented?

Other Issues

Unique identifiers Physical Integration Standards Use of COTS software

Research Infrastructure Academics will likely not be able to effectively

address certification challenges with V & V technologies without a detailed understanding of certification process

Certification training for academics Workshop on certification/approval process working

with example artifacts Mock “red team” certification for products/artifacts

developed by research teams …likewise, the need for effective

communication of available V & V technologies to industry

Research Infrastructure “Open Experimental Platform” for High

Confidence Medical Devices Funding agency pays for…

…vendors to develop representative examples of certified systems

…vendors to provide description of development process

…vendors to provide “challenge problems” to researchers

…vendors to provide “face time” to meet with researchers and evaluate developed tools against base-line development processes

One example OEP proposed by Peter Coronado (Varian) et. al.

Research Goals (5 Years) Apply existing process modeling languages to model clinical

environments and processes Suite of sophisticated requirements tools…

requirements capture, simulation, querying Paths to incorporating more tools into the certification effort

(adding value) Certification of development tools (analysis, traceability tools) for

the purpose of reducing (re)-certification burden Instrument conventional formal methods tools (static analysis,

model-checking) to produce a variety of artifacts (test cases, natural language description of traceability steps)

Demonstration of pervasive use of model-based development techniques with automated reasoning for

component conformance to interfaces component capability based on checking interface capability end to end reasoning of system behavior …emphasize management, integration, and automatic generation

of certification artifacts – models form a substrate within which all certification-relevant artifacts are attached

Research Goals (10 Years) Re-orienting the certification process toward a

component-based certification Certified components as commodities Pervasive use of secure, QoS-aware, fault-tolerant,

certified middleware Integrated end-to-end model-based development

frameworks dealing with composition, evolution, change Effective demonstration (metrics, etc.) necessary

to change industry / regulatory practice Wide body of interoperability standards More sophisticated compositional analysis tools

Tools for compositional hazard analysis