Extensible and DynamicTopic Types for DDSMARS – Jacksonville, FL – March 2010
Presenter: Rick Warren, RTI
Submitting POCs: Angelo Corsaro, PrismTech: [email protected] Gerardo Pardo, RTI: [email protected] Robert Poth, PrismTech: [email protected] Rick Warren, RTI: [email protected]
Supporting POCs: Virginie Watine, Thales: [email protected]
document number: 2010-03-02
Copyright © 2010 RTI - All rights Reserved 2
Structure of this Presentation
Overview of specification and process
Specification contents Very detailed discussion last time This presentation focuses on what’s changed
AB feedback
DDS Types: Assessment and Challenges
Good
Unmatched type safety => fewer defects
Unique type awareness => faster integration
Unique type awareness => higher performance
Bad
No standard API to define / access types at runtime
Hard to change types without rebuilding, redeploying
Hard to upgrade systems one subsystem at a time
Copyright © 2010 RTI - All rights Reserved 3
Vendor-Specific Solutions
OMG-Standard Solutions
Copyright © 2010 RTI - All rights Reserved 4
Where We Are
RFP issued: Jun. 2008
Initial submission: Nov 2008
Revised submission extended: Feb. 2009
Revised submission extended: Aug. 2009
Revised submission extended: Nov. 2009
Revised submission deadline: Feb. 2010 Submitters back on the same page Responded to comments from 2 (now 3) AB reviewers Discussed with Task Force in Long Beach meeting
Copyright © 2010 RTI - All rights Reserved 5
Design Principles
Don’t break anybody. Many people depend on DDS
Don’t modify existing APIs Don’t modify existing compliance points
Many people depend on RTPS/DDSI 2 Don’t modify existing protocols or formats Don’t modify existing compliance points
Keep it fast. New capabilities must permit efficient implementations. Don’t use them? Don’t pay for them.
Keep it orthogonal. Type compatibility doesn’t depend on: Types’ definition language(s) Data representation(s) Programming language(s) QoS configuration
Copyright © 2010 RTI - All rights Reserved 6
Overview
Type System
Type Representation
Language Binding
Data Representation
Copyright © 2010 RTI - All rights Reserved 7
Overview: Example
TypeRepresentation
LanguageBinding
DataRepresentation
IDL:Foo.idl
struct Foo { string name; long ssn;};
IDL to Language Mapping:Foo.hFoo.cFooTypeSupport.c
struct Foo { char *name; int ssn;};
Foo f = {"hello", 2};
IDL to CDR:
0000000668656C6C6F00000000000002
Copyright © 2010 RTI - All rights Reserved 8
Conformance Changes
Feedback on previous submission: conformance criteria, relationships to other specs unclear
New submission more precise, better organized: Conformance (part II, section 3):
2 conformance levels correspond to DDS, RTPS Programming language conformance:
Type system, type representations, language bindings, DDS Minimum Profile
Network interoperability conformance:Type system, data representations, RTPS
Relations to other specs (part III, section 8) Extensions to DDS:
“Programming language conformance” sections constitute new optional conformance level (“profile”)
No changes to existing DDS profiles
Extensions to IDL: Normative to impl.’s of this spec only;
no change to CORBA conformance
Copyright © 2010 RTI - All rights Reserved 9
Type System
Extensible, Compatible 6.5.1: (a) Type System as UML meta-model.
(b) Type substitutability rules. 6.5.2: Type extensibility, mutability rules
6.5.3: Type compatibility in presence of change 6.5.17: Allow type to evolve without breaking interoperability
6.5.5: Support keys. How does extensibility apply?
More Expressive 6.5.4: Map data type 6.5.6, 6.5.19: Sparse data: object contains only subset of fields defined
by its type
Copyright © 2010 RTI - All rights Reserved 10
Type System Overview
class Type System
NamedElement
Type
+ nested: boolean {readOnly}
PrimitiveType
«enumeration»TypeKind
CollectionNamespace
Aggregation
Namespace
Enumeration
NamedElementNamespace
Module
Namespace
BitSetAlias
ConstructedType
«enumeration»ExtensibilityKind
FINAL_EXTENSIBILITY {readOnly} EXTENSIBLE_EXTENSIBILITY {readOnly} MUTABLE_EXTENSIBILITY {readOnly}
+/container
0..*{frozen}
+/containedType
*{addOnly}
+extensibility_kind
1{frozen}
+element_type
1{frozen}
+kind
1{frozen}
+base_type
1
Copyright © 2010 RTI - All rights Reserved 11
Type System Overview
Entirely familiar from IDL: Primitives Strings, narrow and wide Arrays and sequences Aliases (typedefs) Unions Modules
As in IDL, but extended: Structures—including single inheritance
(for substitutability: 6.5.1) Enumerations—specify bit width and constant values
New relative to IDL: Maps—like std::map or java.util.Map;
required by RFP (6.5.4), fundamental OO collection Annotations—for extensibility (6.5.2, 6.5.3)
module Foo { struct Bar { long baz; }; typedef Bar Quux;}
Copyright © 2010 RTI - All rights Reserved 12
Type System Changes
Clearer, more-flexible type extensibility kinds1. Previous kind (Boolean): “extensible”=true:
Add/remove/reorder type members Renamed to “mutable”
2. Previous kind (Boolean): “extensible”=false: Add to end, but don’t add elsewhere, remove, or reorder Renamed to “extensible” Now allow removing from end as well as adding to end
3. New kind “final”: Don’t allow any modifications
Explicit annotations identify all levels in IDL Un-annotated types presumed “extensible” (2) Can be overridden by IDL compiler option
Several concrete examples added
Copyright © 2010 RTI - All rights Reserved 13
Type Extensibility Example
@Extensibility(
EXTENSIBLE)
struct Coord2D {
long x;
long y;
};
@Extensibility(
EXTENSIBLE)
struct Coord3D {
long x;
long y;
long z;
};Assignable
Copyright © 2010 RTI - All rights Reserved 14
Type Extensibility Example
@Extensibility(
FINAL)
struct Coord2D {
long x;
long y;
};
@Extensibility(
FINAL)
struct Coord3D {
long x;
long y;
long z;
};Not Assignable
Copyright © 2010 RTI - All rights Reserved 15
Type Representations
IDL XML, XSD TypeObject
Static Definition 6.5.11: Programming-language-
independent type serialization format(s) 6.5.9: Represent types in IDL and XML 6.5.7: Support IDL subset used by DDS 1.2
Dynamic Definition 6.5.12: Type discovery
Copyright © 2010 RTI - All rights Reserved 16
IDL Type Representation Changes
Added optional pragmas to delimit IDL segments using new extended syntax#pragma dds_xtopics begin [<version_number>]
// IDL definitions
#pragma dds_xtopics end Document conformance requirements for human readers Help non-conforming compilers recognize problems
sooner, give more meaningful errors No particular errors or warnings mandated
Copyright © 2010 RTI - All rights Reserved 17
IDL Type Representation Changes
Use struct syntax for inheritance, not valuetypestruct MyStruct : MyBaseStruct { … }; Type system doesn’t include all valuetype features, so
syntax was confusing Unclear whether valuetype could extend struct
Copyright © 2010 RTI - All rights Reserved 18
IDL Type Representation Changes
Cleaner user-defined annotation syntax Previously used methods, like Java annotations:
@Annotation local interface MyAnnotation {
long my_annotation_member();
}; Now uses attributes:
@Annotation local interface MyAnnotation {
attribute long my_annotation_member;
}; Still uses local interface (alternative: struct)
Emphasizes that no runtime state exists Allows future support for multiple inheritance, if desired
Copyright © 2010 RTI - All rights Reserved 19
XML Type Representations (No Change)
XML Type Representation Terse XML syntax designed ground-up for this Type
System Easy to read, plays nicely with auto-completing editors
XSD Type Representation Based on existing IDL-to-WSDL mapping specification Extended for new Type System concepts from this spec
Copyright © 2010 RTI - All rights Reserved 20
TypeObject Changes
Type Representation suitable for DDS discovery
Renamed from “TypeCode” Name similarity to CORBA TypeCode deemed A Bad
Thing: this spec doesn’t depend on that one .Net also has “TypeCode,” something else entirely…
Improved string sharing to make representations much more compact
Copyright © 2010 RTI - All rights Reserved 21
Data Representations
Extended CDR XML
Copyright © 2010 RTI - All rights Reserved 22
CDR Data Representation (No Change)
Requirements: Continue supporting existing (compact) CDR used by user-
defined types (6.5.16) Continue supporting existing (extensible, parameterized) CDR
used by discovery types (6.5.16) Provide extensibility to user-defined types (6.5.2, 6.5.3, 6.5.17)—
all Data Representations should have equivalent expressiveness
Solution: Make existing extensible, parameterized CDR available to all
Same rules for every type, built-in or user-defined 100% compatible with existing types and serialization
Parameterized CDR: structs & unions marked “mutable”
Each member == parameter Member ID == parameter ID
Compact CDR: everything else
Copyright © 2010 RTI - All rights Reserved 23
XML Data Representation (No Change)
Optional Requirement (6.6.1):Dynamic data access based on XML or JSON
Given: XSD Type Representation
Data object == XML document
Validated by XSD Representation of its type
SAX, DOM-based access follow naturally
New RTPS encapsulation ID lets you use XML on the network
struct OriginalLandData { long x; long y;};
<OriginalLandData> <x>5</x> <y>42</y>;</OriginalLandData>
Language Bindings
Plain Language Binding Dynamic Language Binding
Copyright © 2010 RTI - All rights Reserved 24
Copyright © 2010 RTI - All rights Reserved 25
Plain Language Binding (No Change)
Defined by existing IDL-to-language mappings:1. Start with IDL Representation of a type
2. Apply mapping to desired programming language
Mappings expanded to cover new concepts e.g. maps, annotations, optional members For C, C++, Java as required by RFP (6.5.8)
// IDLstructOrigLandData { long x; long y;};
// Javapublic class OrigLandData { public int x; public int y;}
Copyright © 2010 RTI - All rights Reserved 26
Dynamic Language Binding (No Change)
Classes DynamicTypeFactory: Creates new types (6.5.20) DynamicType: Introspect type definitions (6.5.13)
Informed by CORBA TypeCode API
DynamicData: Introspect objects (6.5.18) Informed by CORBA DynAny, JMS MapMessage,
TIBCO Rendezvous self-describing messages
DynamicTypeSupport: Registers types with DDS (6.5.20) DynamicDataWriter: Writes objects of any (single) type,
each represented by a DynamicData object (6.5.21) DynamicDataReader: Reads objects of any (single) type,
each represented by a DynamicData object (6.5.21)
Copyright © 2010 RTI - All rights Reserved 27
Use by DDS
Data Representation Compatibility Type Compatibility Built-in Types
Copyright © 2010 RTI - All rights Reserved 28
Data Representation Compatibility
Reader and writer must agree on which Data Representation to use
6.5.14: Encapsulation specification, negotiation 6.5.15: Is encapsulation request-offer?
6.6.2: Optional: Custom encapsulation API
Largely unchanged from previous submission
Solution: DataRepresentationQosPolicy List of representation IDs (standard = XCDR, XML) Applies to both readers and writers Included in corresponding built-in topic data types Request-Offer semantics
Writer offers single representation it will use Reader requests a list of representations Compatible iff writer’s representation is in reader’s list
Can describe proposed representations, future additions, and vendor extensions
Copyright © 2010 RTI - All rights Reserved 29
Type Compatibility
If I write type “Foo” and you read type “Bar”, can we communicate?
New submission clearer, more flexiblebased on feedback on previous submission
3 parts:1. Physical: type assignability
2. Intentional: type signature
3. Enforcement: QoS policy governs how above are applied (NEW in this submission)
Copyright © 2010 RTI - All rights Reserved 30
Type Compatibility Example: What I Want
struct Coord2D
{
long x;
long y;
};
struct Coord3D :
Coord2D
{
long z;
};
Read Write
Write Read
Copyright © 2010 RTI - All rights Reserved 31
1. Type Assignability
Structural compatibility:Can sample serialized according to one type be deserialized according to another? Reminder: Rules designed to require no writer-specific
runtime checks at deserialization time
Intrinsic property of types’ definitions Governed in part by extensibility kind No action required by application programmer
Copyright © 2010 RTI - All rights Reserved 32
2. Type Signature
Do I intend the types to be compatible?
Lists of equivalent types, base types provided at registration time
Clarification to previous submission: both reader and writer declarations taken into account
New since previous submission: lists can contain patterns, not just literal names
Copyright © 2010 RTI - All rights Reserved 33
3. Type Consistency Enforcement Policy
QoS policy on Topic, DataReader, DataWritergoverns how (1) and (2) take effect RxO: reader and writer must have the same setting
EXACT_TYPE: Types must also be equal(as if extensibility were “final”)
Registered names must be equal Like DDS of today but with extra safety Assumed during endpoint matching if remote policy
is unspecified
EXACT_NAME: Assignability assumed without inspection Registered names must be equal This is what DDS guarantees today.
DECLARED: Types must be assignable Type signatures must be compatible Default for conformant implementations
ASSIGNABLE: Types must be assignable Type names, signatures ignored: talk to any
compatible type
Copyright © 2010 RTI - All rights Reserved 34
Type Compatibility Example: How to Get It
1. Is Coord2D assignable from Coord3D? Yes.
2. Type Signature:
Coord3DTypeSupport.register_type( my_participant, "Coord3D : Coord2D")
3. Type Consistency Enforcement: DECLARED(default)
struct Coord2D
{
long x;
long y;
};
struct Coord3D :
Coord2D
{
long z;
};
Copyright © 2010 RTI - All rights Reserved 35
Built-in Types (No Change)
Goal: Improve out-of-box experience for new users, those with simple needs
Background: Usability gap vs. competition Other middlewares allow sending textual or binary data
without explicit type definition, registration
Response: Provide simple types built in Pre-defined, pre-registered String: single string payload Bytes: octet sequence payload KeyedString: string payload + string key KeyedBytes: octet sequence payload + string key
Copyright © 2010 RTI - All rights Reserved 36
Summary
Copyright © 2010 RTI - All rights Reserved 37
AB Feedback
Incorporates feedback from 3 AB reviewers:
Define IDL pragma to delimit extensions: Done
Clarify relationships to other specs: Done
Provide more detail about modules and namespaces: Done
Make annotation type definition syntax more natural: Done
Standardize VerbatimText behavior for C, C++, Java languages: Done
Assorted typos, wording changes, organizational suggestions: Done
Copyright © 2010 RTI - All rights Reserved 38
Summary
The proposal:
Satisfies all mandatory and optional requirements andsupports identified real-world use cases
Without sacrificing backwards compatibility Without sacrificing performance
Is easily extensible w/ new type, data representations to handle additional use cases
Only small changes since previous submission
Incorporates all AB feedback
We will be asking for a vote to recommend adoption.
Q & A
Copyright © 2010 RTI - All rights Reserved 39