SOA Quality Assurance:Distributed Environment Testing Strategies, Issues and Best Practices in Net-Centric Operations and Warfare

Jeff SimpsonFederal SOA Architect

An opening thought

“It is not the strongest of the species that survives, nor the most intelligent, but the one most responsive to change.”

Charles Darwin

SOA Quality Assurance Testing Problem

Net-Centric Environment Redefines QA

Net-Centric = Evolving Distributed Computing Systems

New software relies on old running systems

Testing environment cannot be controlledScale testing a special problem

Requires new relationships and responsibilities for Service Producers and Service Consumers

Requires a SOA Core Service BrokerService Level Agreements

Core Enterprise Services, Registry and Metadata Repository

Accreditation and Certification Services

What is SOA?

Oh, no … not another “What is SOA?” slide

What is SOA?

SOA is …

The latest acronym that represents the Savior Of Architects and “Marketeers” after they have beaten OO, Java, etc. into the ground.

What is SOA?

SOA is …(seriously)

The nexus between IT and Business – allow for a common dialog

An new approach to both IT and Business gets done

Reuse

Agility

Efficiency

Loosely-Coupled

Sharing

No. These are emergent attributes or properties, not the definition

What is SOA?

SOA is … (my definition)

SOA represents the current level of maturity of the practice of Distributed Computing Systems

It connotes a “sharing” paradigm and doctrine shift

It connotes Moving from the “Need to Know” doctrine (reactive) to the “Need to Share” doctrine (proactive)

“Need to Know” “Need to Share” + “Authority to Know”

Environment Redefines QA into Total AssuranceServices need to be compliant with “Safety Policies” with cover Functionality, Security, Operations and Standards

NCOW = Evolving Distributed Computing Systems

What is SOA not?

SOA is NOT ...WebServices, Enterprise Service Bus, etc.

SOA tools and software are NOT a panacea

SOA Infrastructure software is NOT a replacement for sound distributed systems engineering

SOA tools will NOT address required Social Engineering

SOA tools and Infrastructure software will NOT be universally and consistently understood

Get help from your vendors and understand that most SOA tools can be used cooperatively

Message-Oriented Middlewareasynchronous != loosely coupled

Understand the differences between Broker-based ESB and MOM-based ESB

New

A Methodology

Different Things to Different Peopleor … Back to the Future

“My guess is that object-oriented programming will be in the 1980’s what structured programming was in the 1970’s. Everyone will be in favor of it. Every manufacturer will promote his products as supporting it. Every manager will pay lip service to it. Every programmer will practice it (differently). And no one will know just what it is”.

Tim Rentch

Excerpt from Object-Oriented Analysis and Design by Grady Booch

Service-Oriented ArchitectureObject-Oriented Architecture2000’s 1990’s

SOA is not a Thing

It’s not even an Architecture … it’s just an approach toward an architecture (of which distributed systems architectures are the most applicable).

It’s a way of thinking and working such that the product of the work results in a systems that is:

Agile and Adaptable

Engenders reuse

Encourages and Enables Sharing of Existing Systems and Data

The SOA Products offerings only provide a toolbox of technical solutions, not a “silver bullet.”

SOA is not a Methodology

SOA nor SOA tools DO NOT do Distributed Engineering for you

ESB does not solve ALL design problems

MOM or EAI solutions are engineering solutions for specific Distributed Engineering problems

The only thing harder than designing and building a complex distributed system is TESTING and DEBUGGING them

SOA Governance

I’m not sure what this is really it. Nor do I know anyone else who has a really good answer.

“The Nexus between politics and operations”

Comprised of Policies, Procedures and MetricsService Lifecycle

Governance Tools:UDDI Registry

Metadata Repository

WebServices Management (SOA Software [BlueTitan], AmberPoint, etc.)

Traditional Enterprise Management Systems (Tivoli, HP OpenView, BPM Patrol, etc.)

Governance usually optimized for one of several outcomes:Reuse – The Cornerstone of Reuse is Communication

Agility

Positive Financial Outcome

Sharing

SOA Governance is just beginning to mature“Draconian”, “Autocratic”, “Oligarchy” or “Facist” Governance does not work well in large organizations

Understanding how to govern operations as large as NCOW is unclear

Usual System Testing

Development Environment

Continuous Integration

Unit and Integration Testing

QA Environment

Protected environment

Scale Testing

Hardware identical to Production Environment

Separate network

Production Environment

Same as QA Environment

Handles multiple Security Enclaves (NIPR, SIPR, JWICS)

Issues with Net-Centric SOA Testing

It impossible to perform traditional QA testing a Net-Centric Environment

Multiple Producer supported environments Development

Unit-Test

Scale-Testing

Production

Secure enclave testing required standalone serviceProducers must provide packaged service for SCIF based development

Like to use VMWare or other Hypervisor technology to reduce technology burden on consumer.

What happens if the service is a composite service?

Even with SCIF based development, final scale and functional testing wants to use the provider-hosted service

Issues with Net-Centric SOA Testing

There is no “Global” synchronized clock

All event correlation required use of cause-event based clocks – otherwise known as Lamport Clocks.

Unclear how to do this is a large-scale coordinated way

Auditing and Logging

Consistently available common logging and auditing services are required.

Should be provided by a shared service infrastructure (NCES?)

NCES does not list this as a common service nor a segment of Enterprise Service Mangement

Adding Friction

Producer Friction

Producer Accreditation and Certification

Producing “Composite” Services

Consumer Friction

Usage friction – adding hurdles to utilizing shared data and services

Vetting consumers in various degrees

Root-case analysis problems with provided composite services

Responsibility of Service Providers

Provide the service using a “Community Standard” interface

WebServices: XML, XSD, HTTP, SOAP, etc.

JMS, FTP, SMTP

NOT: MQ Series, .NET, Proprietary “extensions” to Standards

Provide an Accredited Service

Assert that the services lives up to “Total Assurance or Service Safety” policies and standards

Register the Service with the Shared Service Infrastructure provider

Handle the Unintended User?

The Unintended User?

There is much talk about the Unintended User.I’m not sure what people are talking about any more with that term.

I think people mean: The Unanticipated User

Some definitions may be in order:The Unanticipated User:

A user that you simply did not expect to show up and use your service.

This problem is solved by adequately scaling your service. Solutions include running service in a Cluster or “Virtualize” the service.

The Unknown User:

A user that shows up to use the service, regardless of whether you capacity planned for them, that are not authorized.

This problem is solved by dynamically changing security profiles and access control.

The Unintended User (a.k.a. the Demanding User)

A user that is probably known, and authorized, but does not want to use your service the way it was intended to be used.

Will need to “pay a premium” to the services producer

Responsibility of Service Consumers

Use the service the way the producers intended it to be used

Access service via brokered channel

Access via Shared Service Infrastructure (NCES) as opposed to direct access

No “unintended” Denial-of-Service attacks

No oversized payloads

No unreasonable SLA expectations

Open QA results

Report errors promptly

Utilize CES auditing, logging, security, etc. where possible

Role of the Broker-based Infrastructure

Provide the Authoritative Registry and Metadata RepositoryRegistry for Run-time

Repository for Design-time

Provide the Accreditation and Certification ServiceFunctional

Security/IA

Service Level Agreement Ranges

Provide SLA Adjudication ServicesNCOW Concept: SLA in the “Eye of the Consumer”

Provide Metrics publishing for Governance

Provide Scalable Service Network

Provide automatic “Testing Mode” switching

Conclusion

SOA is a dangerously overused acronym that required specific definition in the context of NCOW.

NCOW with SOA done correctly will display the emergent properties of Agility, Reuse, Efficiency, Loosely-Coupling, etc.

Functional Testing (QA) is brutally difficult to do in an open Distributed Computing Environment

The Social Engineering is a “long pole” in the tentGovernance from Broker, Provider and Consumer is not a solved problem

Most challenges in NCOW serivce creation and usage will depend greatly on how Social Engineering and Communication is accomplished

Thank You

jsimpson@bea.com