View
221
Download
3
Category
Preview:
Citation preview
1
Dictionary Driven Exchange Content Assembly
Blueprints
Concepts, Procedures and Techniques
(CAM – Content Assembly Mechanism Specification)
Author:
David RR Webber
Chair OASIS CAM TC
January, 2010
http://www.oasis-open.org/committees/cam
2 January, 2010 – CAM Draft Specification Development Related Materials
Agenda
• Today’s XSD Schema-based Exchanges
• Current accepted practice – pitfalls and challenges
• How to do this faster, simpler, more reliably?
• Accelerated process overview
• Blueprints and Dictionaries
• Constructing your exchange with Blueprint templates
• Leveraging re-use – standard domain dictionaries
• Creating your own domain dictionary from XSD or UML
• Generating Exchange Artifacts
• NDR evaluation, Exchange schema, mapping crosswalk, XML
instances, realistic data use, business rules documentation
• Summary
3 January, 2010 – CAM Draft Specification Development Related Materials
Today’s XSD Schema-based Exchanges
Current Practice – Pitfalls and Challenges
How to do this faster, simpler, more reliably?
Accelerated Process Overview
4 January, 2010 – CAM Draft Specification Development Related Materials
Current Practice – Conceptual
5 January, 2010 – CAM Draft Specification Development Related Materials
SampleXML
Instance
SampleXML
Instance
Sample
Style Sheet
Sample
Style Sheet
Flows/Use Case
Model
Flows/Use Case
Model
Exchange
Model
Exchange
Model
Schema
Building
Schema
Building
Import schema
Collection
Import schema
Collection
Data
Modeling
Data
Modeling
ModelingTool
ModelingTool
ExistingSchema
ExistingSchema
Exchange
Inception
Exchange
Inception
DataRef Model
DataRef Model
BusinessModel
BusinessModel
Standards
Mapping
Standards
Mapping
SearchTool
SearchTool
MappingTool
MappingTool
Current Practice - Mechanics
Crosswalk
Mapping
Crosswalk
Mapping
Pick list
spreadsheets
Pick list
spreadsheets
Packaging
& Posting
Packaging
& Posting
SW Dev
Tools
SW Dev
Tools
XMLArtifacts
XMLArtifacts
Test and
Inspect
Test and
Inspect
FormsForms
XMLEditor
XMLEditor
Support
Docs
Support
Docs
MetadataMetadata
Package
Sharing toPartners
Package
Sharing toPartners
UMLDomain
Model
UMLDomain
Model
Business
Rules
Business
Rules
Document
Schemas
Document
Schemas
ExtensionSchemas
ExtensionSchemas
Constraint
Schemas
Constraint
Schemas
6 January, 2010 – CAM Draft Specification Development Related Materials
Current Practice – Team Matrix
1 FTE
1 FTE
1 FTE
1 FTE
1 FTE
1 FTE
1 FTE
Resource
Prior Information
Exchange project work
One to two years actively
writing XSD schema
One year or more in
application area
Prior UML based modelling
SW tools training and XML
development
Data analysis and XML
content creation
Technical writer
- Proficient in modelling
methodology and exchange
development requirements
-- Proficient in complex XSD
syntax writing. Familiar with
developer support tooling and
constraints
-- Familiar with project
requirements and business
applications and also developing
XML based exchanges
-- Use of UML diagramming and
models. Information modelling
-- Knowledge of SW tools
available for target environment.
-- Creating test environments,
working with XML test cases,
test data generation
-- Writing documentation and
spreadsheets
Exchange practitioner / project
lead
W3C XSD schema expert
Domain business analyst
UML/ data modelling
practitioner
SW dev tooling user
XML testing and development
Documentation resources
ExperienceQualificationsSkillset
FTE = full time equivalent
7 January, 2010 – CAM Draft Specification Development Related Materials
Pitfalls and Challenges
• Significant amount of manual labor needed to develop all the
exchange documenting artifacts and XML related end products
• Multi-discipline team and supporting cast of exchange / XML
savvy developers needed
• Disconnect between the software delivery teams’ schedule and
process and the exchange development team and process;
production system not matching what the delivery doc says it does
• Alignment to existing domain Enterprise Data Model (EDM)
• Varying quality of hand checked results and no consistency of
technical approach to schema development techniques and re-
use of domain components
• Process not repeatable and predictable
• Scalability - differing production XML details across teams, often
incompatible across implementations and platforms
8 January, 2010 – CAM Draft Specification Development Related Materials
Delivery Level of Effort Estimates
Spiral analysis
Complex with steep
learning curve and
limited practitioners.
Manual preparation
and review
Manual preparation
and review
Manual hand editing
of XML from XSD
Test harnesses vary
Manual preparation
Constraints
Weeks
Weeks
Weeks
2 to 5
days
Weeks
Weeks
Weeks
Model information needs
XSD syntax writing
Excel spreadsheet
Excel spreadsheet
Sets of XML instances
Build test environments
Word documentation
Collect exchange needs
Perform XSD schema development
with EDM alignment
Documentation of each element
Document domain dictionary mapping
(pick list)
Create test cases and examples
Perform interoperability testing
Create exchange documentation
TimingsTasksComponent
Currently 800+ hour process for 300+ node exchange
9 January, 2010 – CAM Draft Specification Development Related Materials
Improving the Process
• Resolving the issues and challenges
• Ensuring consistent results that can be easily reviewed
• Leverage existing dictionary work and repositories of components that the enterprise already has
• Reduce the learning curve and need for specialized skills
• Business analysts not excluded from design, review and implementation by technical barriers
• Lock-step the development process to the exchange
• Customizable and configurable so can adapt to changing requirements
10 January, 2010 – CAM Draft Specification Development Related Materials
Faster, Simpler, Predictable
• Tooling automates much of the manual tasks; ensures predictable
quality of results
• Reduce need for specialized technical knowledge of XSD and XML
• Provide consistent approach that leverages best-practice techniques
with built-in smarts and knowledge
• Tooling checks for common pitfalls, applies NDR checks
• Allow business analyst to complete much of the design work and
crosscheck application details
• Leverage reuse of domain component dictionaries and blueprints
• Lockstep development to exchange artifacts and their delivery
• Accelerate development tasks (test cases, testing, schema writing)
• Produce result that are neutral to developer tooling platforms
• Process repeatable and replicatable when requirements / versions
change
11 January, 2010 – CAM Draft Specification Development Related Materials
Using Dictionaries & Blueprints
• Dictionaries provide reference sets of components to be used in exchanges; three possible sources:
• Dictionaries imported from existing industry schema
• Domain dictionary built from an Enterprise Data Model schema
• Reverse engineered out from existing exchange schema
• Blueprint
• Is the outline of the structure components to be used in an exchange schema
• Can import components from one or more domain dictionary collections
• Sketches out the desired information exchange with re-use of existing exchange component structures, plus any local additions / extensions / exclusions
• Expander tool reads the blueprint, references the dictionary, and constructs the complete exchange schema
12 January, 2010 – CAM Draft Specification Development Related Materials
Accelerated Process Overview
BusinessModelling Charter
Work-
GroupProject InceptionProject Inception
dictionariesdictionaries
Exchange
Content
Modelling
Exchange
Content
Modelling
Blueprints Dictionary
Dictionary
Expand and
Evaluate Results
Expand and
Evaluate Results
Web
basedDesktop
Excel
Search tools
Review / Tailor
Exchange
Structures
Review / Tailor
Exchange
Structures
Generate
Package
Artifacts and
Test them
Generate
Package
Artifacts and
Test them
Package
Exchange
Package
Exchange
Completed Exchange
StructuresNDR
Evalu-ation
Dictionary Generation
Completed Exchange artifacts
Test Cases
Validation
Engine
XML
examplesXSD
schemas
Subset Schema
Crosswalk
MappingSpreadsheet
Rules
Report
Rendering
Stylesheet
Import XSD
Naming /
NDR / spell check
Generate
Dictionary
Import EDM
Test examples and rules
Model Exchange
Expand, Review, Refine
CAM Enabled
13
Blueprints and Dictionaries
Leveraging re-use – dictionaries from industry standards
Creating your own domain dictionary from XSD or UML
Constructing your exchange and blueprints
14
Building Domain Dictionaries
EDM
EleEle
DefDef
DDLDDL
11
Export
Components in
XSD syntaxCollection of objects
from model
Option 1 – From Enterprise Data ModelImport XSD and refactor for use with OASIS CAM
22Option 2 – Derive from existing
exchange XSD schemaImport each XSD and merge into CAM dictionary
Exchange XSD
schema
Exchange XSD
schemaExchange
XSD schema
Exchange XSD
schemaExchange
XSD schema
Exchange XSD
schema
CAM template
CAM template
OASIS CAM
template
OASIS CAM
template
Model Components XSD schema
Model Components XSD schema
CAM template
CAM templateOASIS
CAM template
OASIS CAM
template
NDR
Evaluation,
Refactor,
Renaming
Tool
NDR
Evaluation,
Refactor,
Renaming
Tool
Apply Naming and
Design Rule (NDR)
checks and edits
44
NDR
Evaluation,
Refactor,
Renaming
Tool
NDR
Evaluation,
Refactor,
Renaming
Tool
44
Generate
Standard
Components Dictionary
XML
Generate
Standard
Components Dictionary
XML
Merge & Generate Dictionary
XML
Merge & Generate Dictionary
XML
55
55
XMLXML
Dictionary of exchange components
XMLXML
Import
Import
Import
Import
Dictionary of exchange components
33
33
Automated
Manual
LEGEND
Analyst Review
Analyst Review
ebXML CCTS compatible (ABIE,
BBIE, ASBIE)
ebXML CCTS compatible (ABIE, BBIE, ASBIE)
15 January, 2010 – CAM Draft Specification Development Related Materials
Blueprint Approach Overview
DefEle
Industry
dictionariesformatted as XML
Exchange
generator
tools (CAM)
Automated
Manual
LEGEND
Co
mp
on
en
ts D
efin
ition
(XM
L)
Local domaindictionaryformatted as XML
Build
55
22
ExchangeBlueprint Designer User Interface
Expand
StructureExchange Structure
Pick
ComponentsStructure Outline Blueprint
Targetapplications
EDM
EleEle
DefDef
DDLDDL
Exchange
Package
Exchange
Package
Exchange
Components
Exchange
Components11
33
66
77
Enterprise
Data ModelImport and refactor for use with CAM
DictionaryRepository
44
16 January, 2010 – CAM Draft Specification Development Related Materials
Blueprint Development Tools
Expander
Tool
Expander
Tool
Industry dictionaryIndustry dictionary
Domain dictionaryDomain dictionary
Component Definitions
44
Component Definitions
Web tool
Web tool
ExcelExcel
Search Tools
Search Tools
22
BlueprintDesigner
BlueprintDesigner
11
InsertDictionary
ParentComponents
InsertDictionary
ParentComponents
33
Completed ExchangeTemplate
Completed ExchangeTemplate
55
17 January, 2010 – CAM Draft Specification Development Related Materials
Blueprint
follows
model Console Log detail
Blueprint Expander Example
Exchangecomponents outline
EXCHANGE BLUEPRINTEXCHANGE BLUEPRINT
COMPLETED EXCHANGE TEMPLATECOMPLETED EXCHANGE TEMPLATE
11
33
StructureDetails Expanded
Expander Tool
Expander Tool
22
Dictionary Lookups
Dictionary Lookups
18 January, 2010 – CAM Draft Specification Development Related Materials
Exchange Template Editor
Exchange
Structure items
Rules for
each item
Domain
Components
Namespaces
Can add new
custom
domain Items
1122
33
44
19
Generating Exchange Artifacts
NDR evaluation, crosswalk mapping, Exchange
Schema, Subset schema, XML instances, business
rules documentation
20 January, 2010 – CAM Draft Specification Development Related Materials
Exchange Generation Steps
Toolkit for exchange artifacts generation
Toolkit for exchange artifacts generation
Suite of menu options and tools in desktop CAM toolkit editor
(each run custom xslt scripts on the exchange CAM template xml)
CAM Template
1. Run template Evaluation Report
2. Compare to industry dictionary
- create mapping spreadsheet
- create crosswalk xml
Generate business rules report
3. Generate exchange XSD schema
4. Generate XML test instance(s)
22
11
33
44
55
EXCHANGE TEMPLATEEXCHANGE TEMPLATE
Structure
Rules
Documentation
21 January, 2010 – CAM Draft Specification Development Related Materials
NDR Evaluation Report
• Provides scoring, alerts, warnings and potential
issues including:
• Naming and content model conventions
• Naming and restriction consistency checks
• Interoperability enablers/inhibitors checks
• Rules integrity and duplicates
• Statistics on exchange size
• Spell checking on component names
22 January, 2010 – CAM Draft Specification Development Related Materials
Evaluation Report (NDR) example
Part of an example report for LEXS getDataItemRequesttemplate displayed in toolkit HTML viewer
23 January, 2010 – CAM Draft Specification Development Related Materials
Compare to industry dictionary
• References industry dictionary of names and properties
• Matches on physical names
• Reports mapping details
• Compatible with Microsoft Excel
• Report can be used to do spell checking
• Generates crosswalk xml file
24 January, 2010 – CAM Draft Specification Development Related Materials
Example cross-reference spreadsheet
Formatted view in Microsoft Excel of import of cross-reference report details (from generated XML file)
25 January, 2010 – CAM Draft Specification Development Related Materials
Generate Documentation and Schema
• Documentation:
• Create HTML report of exchange schema details and associated content and business rules
• Report layout and content designed to be reviewed by business analysts
• Schema:
• Generate XSD schema for exchange
• Customizable exchange folder layout management by namespace for extension, subset and exchange schema components
• Writes XSD schema in syntax that is clear, simple and compatible with deployment tooling environments
26 January, 2010 – CAM Draft Specification Development Related Materials
Business Rules Documentation
Part of the example rules for LEXS getDataItemRequest template displayed in toolkit HTML viewer
27 January, 2010 – CAM Draft Specification Development Related Materials
Export Exchange to XSD Schema
Completed
Exchange
Structure
Set
Exchange
Options
Complete set
of exchange
schemas
generated
Export Template to
Exchange XSD:
28 January, 2010 – CAM Draft Specification Development Related Materials
Exchange Schema Generated
Each namespace file is import for
those specific type definitions
Reviewing XSD results in
a schema editor tool
Set of XSD files with filename
and namespace suffix
29 January, 2010 – CAM Draft Specification Development Related Materials
XML Testing Examples Generation
• Support for software development testing process
• Designed to allow creation of concrete realistic examples not just random value based
• Hinting system allows insertion of actual test system values into XML examples
• Can create both valid and invalid examples to support unit testing of application software
• Exclude capability allows generator to create examples that contain only a portion of the entire exchange
• Control over random seed value used allows re-generation of identical test cases
30 January, 2010 – CAM Draft Specification Development Related Materials
XML example generation wizard
View of CAM toolkit with LEXS getDataItemRequestand dialogue for XML test example generator tool
31 January, 2010 – CAM Draft Specification Development Related Materials
Running validation rules tests
• Built-in CAM validation engine allows testing of XML instances against actual exchange rules (CAMV).
• Critical to ensure that the exchange validates actual live production example scenarios correctly
• Allows deployed solution to match exchange schema details
• Errors can be reviewed interactively in exchange visual interface
• Post-processing of validation results allows unit regression tests to be created with reporting of errors, warnings and information level notes
32 January, 2010 – CAM Draft Specification Development Related Materials
Run Exchange Template
Pick XML test case to validate Run validationReview results in visual editor
33 January, 2010 – CAM Draft Specification Development Related Materials
Example Exchange Packaging Details
All metadata of owner and domain to be associated with the exchange.Metadata XML file
A machine-readable list of artifacts provided in this exchange package.Catalog XML file
Catalog
Record of cumulative changes from previous exchange versions. The initial exchange simple records its creation date.Change log
Components created because they were not in dictionary—may be part of mapping spreadsheet and include structure and definitions
of new components. Extended components
Mapping of domain components, tagged with constraints (i.e., cardinality, etc.) to dictionary components as a spreadsheet.Mapping to Dictionary
Business rules in one of the following formats: (1) plain or structured English, (2) written into master documentation, (3) generated
by a development tool.Business rules
Exchange model in standard open format (xmi, vsd, zargo) and standard open graphic (jpg, pdf, etc.) preferably a Unified Modeling
Language (UML) model.Exchange model
The Master Document is the main document for which all of the context and details around the exchange are explained. This
document includes, the overview, as well as details surrounding the exchange, business drivers and requirementsMaster Documentation
Documentation
Example stylesheet for display of instance(s).Stylesheet
Example instance(s) – may reference optional stylesheet.Sample XML Instance
Specification for extended components—separate local name-spaces of components not contained in dictionaryExtension Schema
Optional schema that includes additional constraints and code values for the main exchange schemaConstraint Schema
Base document schema that defines the full XML structure for the exchange and is generally named after the exchange itself.Exchange Schema
Itemized list of each dictionary component element and attribute included in the exchange. Crosswalk XML
Subset of the full exchange schema—a reduced set of components only used in this exchange, not every possible component.Subset Schema
Exchange Files
DescriptionPackage Artefact
34 January, 2010 – CAM Draft Specification Development Related Materials
Summary
Dictionary driven exchanges
Blueprint enabled reuse
Automated exchange package generation
Alignment to NDR Principles and Rules
Testing and validation support
35 January, 2010 – CAM Draft Specification Development Related Materials
Review• Top Down development
• Reference dictionary components
• Create exchange blueprint
• Run Expander tool
• Refine desired structure in visual editor
• NDR Principles and Rules
• Best practices for interoperability and schema techniques
• Dictionary driven reuse
• Enterprise Data Model and industry components
• Ensures consistency of definition and use
• Automated exchange package generation
• Schemas, XML, documentation, mapping crosswalk
• Test generated example XML with rules validation
37 January, 2010 – CAM Draft Specification Development Related Materials
Links and Resources
• DOWNLOADS -
• CAM Toolkit download
• https://sourceforge.net/projects/camprocessor
• SUPPORTING MATERIALS -
• NIEM Naming and Design Rules (NDR) 1.3
• http://www.niem.gov/pdf/NIEM-NDR-1-3.pdf
• RESOURCES –
• Additional support slides (following)
38 January, 2010 – CAM Draft Specification Development Related Materials
Blueprint Driven Approach
StaticConventional
Models, Artefacts, CodeWSDL,XSD,UML,XML
Analysis
Design
Coding
TestMaintenance
SW DEVELOPERSSW DEVELOPERS
Specification
Agile DynamicComponentsTEMPLATES
Adoption, integration
Installation
Implementation/Use
Needs
Requirements
Procurement
BUSINESS USERS
Data models
Excel spreadsheetBlueprint TemplatesXML visualization
XML artifactsXSD schema
DynamicDynamic
Software code
CompilersDeployment serversXSD schema
XML artifacts
39 January, 2010 – CAM Draft Specification Development Related Materials
•Template •Wizard• runtime configuration
•Library
•Object templates/Components
•design time
•Facets
•Questions / Data
•Technology Targeting• Syntax specific production rules
•Exchange Interface / Blueprints
• Visual editor + review / test /
completion steps
11
33
44
XML22 •Domain
Dictionary Details Stored
XMLXML
XMLXML
•Solution Specific Syntax
55
40 January, 2010 – CAM Draft Specification Development Related Materials
Domain Exchange Development Steps• Adopt formal Naming and Design Rules (NDR)
• UN/CEFACT – NDR
• OASIS UBL – Universal Business Language
• OASIS EML – Election Markup Language
• NIEM – National Information Exchange Model approach (http://www.niem.gov)
• OASIS EM - Emergency Management joint initiative with NIEM
• Develop data models of core components for use in exchanges
• Build Dictionary of Core Components
• Provide Principles and Rules guidance to schema team
• Use namespaces, Yes / No?
• Camel case naming convention?
• Schema constructs and restrictions on use?
• Information Exchange Package Documentation (IEPD)
• Describes formal exchange that conforms to NDR and principles and rules
• Provides schema, example XML, supporting artifacts
• Re-uses core components
• Defines domain specific components
41 January, 2010 – CAM Draft Specification Development Related Materials
Example Governance Structure
42 January, 2010 – CAM Draft Specification Development Related Materials
OASIS Content Assembly Mechanism (CAM) & Integration Technologies Guide
OASIS
CAM
Templates
Ontology
Classification
Discovery
Domain
Dictionaries
NDR
W3C
XSD
Schema
- WHAT?
� Provides lexicon
of information
content
� Describes
structure constructs
� Arranges groups
of information
� Simple content
typing
� Software tooling
interfaces
- HOW? WHY?
� Provides actual
use patterns
(templates)
� Supports context
handling and rules
� Rendering outputs
and documentation
for verification
� Enables
integration testing /
certification
� Versioning
- WHO?
�- Alignment of
meaning and terms
�- Consistent
domain definitions
�Modelling
methods and
practice
� Business
information
content building
blocks
- WHERE?
�Shared resources
of semantic
definitions
� Code lists
� Dynamic rendering
� Distributed
versioning control
� Role and access
security
management
- WHEN?
� Alerts
� Process control
� Workflow
� Automated
interfacing
� Business
Intelligence
Machine
Based
Reasoning
Shared
Semantics
Registry
- WHERE?
� Domain
classification
systems
� Ontology and
reasoning
definitions
� Associations
and linkages
� Search and
drilldown
� Modelling tools
Recommended