Pat Helland Partner Architect Microsoft Corporation

Data on the OutsideversusData on the InsidePat HellandPartner ArchitectMicrosoft Corporation

Session Objectives And Takeaways

Understand the use and interpretation of dataIt is different inside services than outside services

Many new issues arise in the world between services

Versioning, meta-data (schema), immutability, idempotence, and much moreThink carefully about the interactions across services

XML, SQL, and Objects all have their placeUnderstand their strengths and weaknesses and use them together to create the best services

Slide 2

Outline Introduction: The Shift Towards Services

Behavior: Encapsulation and Trust

Data: Then and Now

Outside Data: Reference Data

Outside Data: Sending Messages

Outside Data: XML and Schema

Outside Data: Commonality of Schema and Understanding

Inside Data

Inside/Outside: Representations of Data

Inside/Outside: Tying It All Together

ConclusionSlide 3



Data: Then and Now





Inside Data



ConclusionSlide 4

Service

PolicyPolicySchema and

Contract

Schema and Contract

Service-Oriented ArchitectureService-orientation

Independent servicesChunks of code and data

Interconnected via messaging

Four basic tenets:Boundaries are explicitServices are autonomousServices share schema and contract

Not implementationService compatibility is based on policy

Slide 5

Services Communicate With Messages

Services communicate with messagesNothing else

No other knowledge about partnerMay be heterogeneous

Service-A Service-B

Slide 6

Data Inside and Outside ServicesData is different inside from outside

Outside the servicePassed in messagesUnderstood by sender and receiverIndependent schema definition importantExtensibility important

Inside the servicePrivate to serviceEncapsulated by service code

DataSQL

MSG

MSG

Data Outside the Service

Data Inside the Service

Slide 7



Data: Then and Now





Inside Data



ConclusionSlide 8

Bounding Trust via Encapsulation

Services only do limited things for their partnersThis is how they bound their trust

Encapsulation is about bounding trustBusiness logic ensures only the desired operations happenNo changes to the data occur except through locally controlled business logic!

Service

Things I’ll Do for Outsiders• Deposit• Withdrawal• Transfer• Account Balance Check

Things I’ll Do for Outsiders• Deposit• Withdrawal• Transfer• Account Balance Check

Slide 9

Encapsulating Both Change and ReadsEncapsulating change

Ensures integrity of the service’s workEnsures integrity of the service’s data

Encapsulating exported data for readEnsures privacy by controlling what’s exportedAllows planning for loose coupling and expirations

E.G. Wednesday’s price-list

Business Request

Exported Data

Sanitized Datafor Export

PrivateInternal

Data

Data

Trust and TransactionsSome propose atomic transactions across services

E.G. WS-Transactions

Requires holding locksLots of trust in timely unlockDoesn’t sound autonomous and independent to me…

Debate is the definition of the word serviceRequires autonomy and independence?Allows intimacy across service boundaries?There will be code connected by 2-phase commit

Same service or in different services?

For this talk, I presume no cross-service txsSimply the definition of the word “service”

Slide 11

Interconnecting with Independent ServicesServices are connected by messaging

The only interaction between two services is by the messages that they exchange

Schema: the formats of the individual messagesContracts :the allowable sequences of messages

Service

Contract

Reject-Order

Accept-OrderOneOf

TentativePlace-Order

ConfirmPlace-Order

OneOf

CancelPlace-Order

Slide 12

Operators and OperandsMessages contain operators

Requests a business operationOperators provide business semanticsPart of the contract between the two services

Operator messages contain operandsDetails needed to do the business operationThe sending service must put them into the message

Service

Deposit

OperatorOperands

Slide 13

Where Do Operands Come From?

Operands come from reference dataNew kind of data in SOA

Except it’s not new; we’ve done variations of SOA for decades… We’re just getting better at it!

Reference data is versioned and each version is immutable

Immutable images are shared across many servicesWe will talk about the creation, publication, and management of reference data

Slide 14



Data: Then and Now





Inside Data



ConclusionSlide 15

Transactions and Inside DataTransactions make you feel alone

No one else manipulates the data when you are

Transactional serializabilityThe behavior is as if a serial order exists

TkTl

Tm

Tn

ToTh

Tg TjTe

Tf

Tb

Ta

Tc

Td

Ti

These TransactionsPrecede Ti

These TransactionsFollow Ti

Ti Doesn’t Know About TheseTransactions and They Don’t

Know About Ti

TransactionSerializability

TkTl

Tm

Tn

ToThTh

Tg TjTe

Tf

Tb

Ta

Tc

Td

Ti

These TransactionsPrecede Ti

These TransactionsFollow Ti

Ti Doesn’t Know About TheseTransactions and They Don’t

Know About Ti

TransactionSerializability

Slide 16

Life in the “Now”Transactions live in the “now” inside services

Time marches forwardTransactions commit Advancing timeTransactions see the committed transactions

A service’s biz-logic lives in the “now”

Service

Each Transaction Only Sees a Simple Advancing of Time with a Clear Set of

Preceding Transactions

ServiceServiceService

Each Transaction Only Sees a Simple Advancing of Time with a Clear Set of

Preceding Transactions

Slide 17

Sending Unlocked Data Isn’t “Now”Messages contain unlocked data

Assume no shared transactions

Unlocked data may changeUnlocking it allows change

Messages are not from the “now”They are from the past

There is no simultaneity at a distance!• Similar to speed of light• Knowledge travels at speed of light• By the time you see a distant object it may have changed!• By the time you see a message, the data may have changed!

Services, transactions, and locks bound simultaneity!• Inside a transaction, things appear simultaneous (to others)• Simultaneity only inside a transaction!• Simultaneity only inside a service!

Slide 18

Outside Data: a Blast from the Past

All data seen from a distant service is from the “past”By the time you see it, it has been unlocked and may change

Each service has its own perspectiveInside data is “now”; outside data is “past”My inside is not your inside; my outside is not your outside

All data from distant stars is from the past• 10 light years away; 10 year old knowledge• The sun may have blown up 5 minutes ago

• We won’t know for 3 minutes more…

Going to SOA is like going from Newtonian to Einstonian physics• Newton’s time marched forward uniformly

• Instant knowledge• Before SOA, distributed computing many systems look like one

• RPC, 2-phase commit, remote method calls…• In Einstein’s world, everything is “relative” to one’s perspective• SOA has “now” inside and the “past” arriving in messages

Slide 19

Versioned Images of a Single Source

A sequence of versions describing changes to dataUpdates fromone service

Owner controlledOwner changes the dataSends changes as messages

Data is seenas advancingversions

ListeningPartner

Service-1

ListeningPartner

Service-5

ListeningPartner

Service-7

ListeningPartner

Service-8

Tuesday’sPrice-List

Wednesday’sPrice-List



Monday’sPrice-List



Monday’sPrice-List


Data Owning Service

Price-List

ListeningPartner

Service-1

ListeningPartner

Service-5

ListeningPartner

Service-7

ListeningPartner

Service-8

Tuesday’sPrice-ListTuesday’sPrice-ListTuesday’sPrice-List










Monday’sPrice-ListMonday’sPrice-ListMonday’sPrice-List





Monday’sPrice-ListMonday’sPrice-ListMonday’sPrice-List


Data Owning Service

Price-List

Data Owning Service

Price-List

Slide 20

Operators: Hope for the FutureMessages may contain operators

Requests for business functionality part of the contractService-B sends an operator to Service-A

If Service-A accepts the operator, it is part of its future

It changes the state ofService-A

Service-B is hopefulIt wants Service-A to dothe workWhen it receives a reply,its future is changed!

OperatorResponse

OperatorRequest

InvokingPartner

Service-B

InvokedPartner

Service-A

Hopeful for the Future…Decides to Issue Request

Ever Hopeful,Waiting for aResponse

Hopes Fulfilled,the Future Is Now

BlithelyIgnorant andMinding Its Own Business

A Future ForeverAltered by theProcessing of theRequest fromService-B

OperatorResponseOperator

ResponseOperator

ResponseOperator

Response

OperatorRequestOperatorRequestOperatorRequestOperatorRequest

InvokingPartner

Service-B

InvokedPartner

Service-A

Hopeful for the Future…Decides to Issue Request

Ever Hopeful,Waiting for aResponse

Hopes Fulfilled,the Future Is Now

BlithelyIgnorant andMinding Its Own Business

A Future ForeverAltered by theProcessing of theRequest fromService-B

Slide 21

Operands: Past and FutureOperands may live in the past

Values published as reference dataCome from Service-A’s past

Operands may live in the futureThey may contain a proposed value submitted to Service-A

Service-B Preparing a Request for Service-A

Friday’s Price-ListPublished:11PM Thursday

OperatorOperands

On Friday, Operands Are Extracted from

the Price-List Publishedon Thursday

Deposit

Service-B Preparing a Request for Service-A



OperatorOperands

On Friday, Operands Are Extracted from

the Price-List Publishedon Thursday

DepositDeposit

Slide 22

Between Services: Life in the “Then”Everything between services lives in the past or future

Operators live in the futureOperands live in the past or the future

It’s not meaningful to speak of “now” between servicesNo shared transactions no simultaneity

Life in the “then”Past or futureNot now

Each service hasa separate “now”

Different temporalenvironments!

Service-1

Service-2

Service-4

Service-3No Notion No Notion of of ““NowNow””

in Betweenin BetweenServices!Services!

Service-1Service-1Service-1

Service-2Service-2

Service-4Service-4

Service-3Service-3No Notion No Notion of of ““NowNow””

in Betweenin BetweenServices!Services!

Slide 23

Services: Dealing with “Now” and “Then”Services Make the “Now” Meet the “Then”

Each Service Lives in Its Own “Now”Messages Come and Go Dealing with the “Then”The Business-Logic of the Service Must Reconcile This!!

The world is no longer flat!• SOA is recognizing that there is more than one computer• Multiple machines mean multiple time domains• Multiple time domains mandate we cope with ambiguity to

allow coexistence, cooperation, and joint work

Example: accepting an order• A biz publishes daily prices• Probably want to accept yesterday’s prices for a while• Tolerance for time differences must be programmed

Example: “Usually ships in 24 hours”• Order processing has old info• Available inventory not accurate• Deliberately “fuzzy”• Allows both sides to cope with difference in time domains!

Slide 24



Data: Then and Now





Inside Data



ConclusionSlide 25

What Is Reference Data?

Reference data is published across service boundaries

For each collection of reference data:One service creates and publishes the dataOther services receive periodic versions of the data

Purposes for Reference Data

Historic Artifacts

Shared Collections of Data

Operands

Slide 26

Reference Data: Operands for the Operators

As discussed above, messages across services invoke business operations…

Each service-to-service message is an operator

Each operator message is filled with operandsParameters, options, customer-id, parts-being-ordered, etc

The data for these operands is published as reference data

Service

Deposit

OperatorOperands

Slide 27

Reference Data: Historic ArtifactsHistoric artifacts report on what happened in the

pastSometimes these snapshots need to be sent to other services

Examples:Sales quarterly resultsMonthly bank statementsAny and all monthly bills

Well… Both requests for payment (operations) and the historic artifact of how much power you used…

Inventory status at end of quarter

Service(Bank)

BankStatementJul-2007

Slide 28

Ref Vers#23of Employee

Data

AuthoritativeEmployee

Data – Vers#23 Update!Ref Vers#24of Employee

Data

AuthoritativeCustomer

Data

AuthoritativeEmployee

Data – Vers#24

Reference Data: “Shared Collections of Data”

Many services may need access to the same dataThe data is changing…Someone owns updating and distributing the data…

Examples:Customer databaseEmployee databaseParts database and price-list

Sales ServiceHR Service

Ref Vers#24of Employee

DataUpdateEmployees

Vers#24

Slide 29

Publishing Versioned Reference Data

The owner of data periodically publishesUsing whatever messaging technique it wants

Publications are always versionedThe version numbers increase

4Request

Uses: Vers-ZService-B

3A’s-DataVers-Z

2A’s-Data

Vers-Y

A’s-DataVers-X

1

Service-A

Slide 30

2RequestUses: Vers-XPlease MakeData Change

Business Operations May Request ChangesIf a non-owner wants a change it must do a biz-operation

This is a request sent to the owning serviceThe owning service may agree to the operation causing changeto the data in questionIf it changes, this affects the next version`

Service-B3A’s-Data

Vers-Y

A’s-DataVers-X

1

OwningService-A

Slide 31

Optimistic Concurrency Control:Anti-EncapsulationWhat is optimistic concurrency control?

Data is readChanges are made and submitted to the data’s ownerIf the original data hasn’t changed, the new changes are applied

This assumes the remote system should be able to write directly on the data

This is a trusting relationship… not autonomous!

Autonomy and updates to dataAutonomy means independent control

My local biz-logic decides how my data changes!If you want a change, ask me to do a business opIt’s my data…I’ll decide how it changes!

Slide 32

Example: Updating the Customer’s Address

What about a salesperson updating a customer’s address?

Shouldn’t that just be optimistic concurrency control?

No! It should invoke business logic with a request!Not all fields of the customer record should be updated by sales peopleRequests across service boundaries invoke business logic when the customer address is changed

Slide 33



Data: Then and Now





Inside Data



ConclusionSlide 34

Immutable And/Or Versioned Data

Data may be immutableOnce written, it is unchangeable

Immutable data needs an IDFrom the ID, comes the same dataNo matter when, no matter where

Versions are immutableEach new version is identifiedGiven the identifier, the same data comes

Version independent identifiersLet you ask for a recent version

•Windows Vista, SP1• The Same Set of Bits Every Time

New York Times; 7/24/07 • Specific Version of the Paper -- Contents Don’t Change

Latest SP of Vista• Definitely Vista, Results Vary Over Time

Recent NY Times• Maybe Today’s, Maybe Yesterday’s

VersionIndependent

Slide 35

Immutability of MessagesRetries are a fact of life

Zero or more delivery semanticsMessages must be immutable

Retries must not see differences…Once it’s sent, you can’t un-send!

Service-A

Once It’s Outside,It’s Immutable!

Slide 36

To Cache Or Not To Cache

OK to cache immutable dataIt’s never wrongNever have to invalidate!

Caching should only be used for immutable dataCaching data that changes may lead to anomalies

Consider caching data labeled with a version dependent ID

Because versions are immutable it will work

Store the mapping from version independent to version dependent in an accurate location

Slide 37

Normalization And Immutable Data

Databases design for normalized dataCan be changed without “funny behavior”Each data item lives in one place

Sometimes data should be de-normalizedIf data is immutable it’s OK

Emp #Emp

NameMgr #Mgr NameEmp Phone Mgr Phone

47 Joe 13 Sam5-1234 6-987618 Sally 38 Harry3-3123 5-6782

91 Pete 13 Sam2-1112 6-9876

66 Mary 02 Betty5-7349 4-0101

Classic problemwith de-normalization

Can’t updateSam’s phone #since there aremany copies

De-normalization isOK if you aren’t going to

update!

Slide 38

Stability Of Data

Immutability isn’t enough!We need a common understandingPresident Bush 1990 vs. President Bush 2007

Stable data has a clearly understood meaningThe schema must be clearly understoodThe interpretation of values must be unambiguous

Advice• Don’t recycle customer-ids

Observation• Anything called “current” is not stable

Suggestion• Timestamping or versioning makes stable data

Observation• A monthly bank statement is stable data

Slide 39

A Few Thoughts on Stable Data

Outside data must be stableConsistent interpretation across valid spaces and times

Inside data may be stableNotably, when it is the same data as outside data…

Sometimes data inside is not stableClassic normalization for vibrant updateNeeds to be cast into a stable shape to send outside

Slide 40

Validity Of Data In Bounded Space And Time

Bounding the valid timesIt may have an expiration

Bounding the valid locationsRestrictions on where the data is valid

When valid, the data should be:Immutable (the ID yields the same bits)Stable (the meaning is clear)

“Offer Good to Washington State Residents Only”

“Offer Good Until Next Tuesday”

Data Valid For Service-X OnlyPrice-List Valid Until Dec 31st

Slide 41

Rules For Sending Data In MessagesIdentify the

MessagePut Unique ID in All MessagesPart of the Unique ID May Be a Version…

ImmutableData

Don’t Change the Data Associated withthe Unique ID; Never Return Different Bits

OK toCache

The Same Bits Will Always Be Returned

Define ValidRanges

Valid for a Certain Time Period and OverSome Space; OK to Always Be Valid

Must BeStable

Must Ensure There Is Never Any ConfusionAbout the Meaning (Within Valid Range)

Slide 42



Data: Then and Now





Inside Data



ConclusionSlide 43

SQL, DDL, and SerializabilitySQL’s DDL (data definition language) is transactional

Changes are made using transactionsThe structure of the data may be changed

The interpretation after the DDL change is different

DDL lives within the time scope of the databaseThe database’s shape evolves over timeDDL is the change agent for this evolution

SQL lives in the “now”Each transaction’s execution is meaningful only within the schema definition at the moment of its executionSerializability makes this crisp and well-defined

Slide 44

Schema and Immutable MessagesWhen a message is sent, it must be immutable

It is crossing temporal boundariesRetries mustn’t give different results

The message’s schema must be immutableIt makes a mess if the interpretation of the message changes

Service-AMessage

MessageSchema

Immutable Message

Immutable Schema for the Message

Slide 45

Immutable Schema and Its Identifiers

Immutable schema needs an identifierIt must be possible to unambiguously identify the schemaThis must occur across the namespaces of sender and receiverThe schema definition must never change

Given the identifier, the same schema is returned

URIs (Universal Resource Identifiers) work wellGuaranteed to be unique

If you follow the rules

URLs (Universal Resource Locators) are coolURLs are URIsAlso tell you a location to get the stuff (e.g. the schema)

Slide 46

Composition of Schema as a DAGSchema make contain sub-schema

Inside the message are chunks of dataA purchase-order may contain customer information

They have their own definitions

The sub-schema are referenced by identifierThis leads to a tree of references to immutable schema

It’s really a DAG (Directed Acyclic Graph)Sometimes, different sub-schema reference the same stuff

Purchase OrderCustomerDelivery AddrSKUs

Customer NameAddressCredit Rating

Address Number/StreetCity/StatePostal CodeCountry

SKU Part Color Size

Slide 47

Versioning and SchemaFrequently, schema is versioned

A new format of the schema is createdIt is given a new identifier

Version independent schema identifiersSpecify a set of versions for a type of schemaThe set may evolve over time

Version dependent schema identifiersSpecify a specific version of a specific schemaThe version-dependent schema is immutable

Messages should always specify a version-dependent schema

This ensures no ambiguity

Slide 48

Extensibility and SchemaExtensibility is the addition of non-schema specified information into the message

The schema does not specify the additional stuffThe sender wanted to add it anyway

Adding extensions is like scribbling in the marginsSometimes adding notes to a form helps!Sometimes it does no good at all!

Service-A

Purchase Order CustomerDelivery Addr

SKUsDon’t Deliver in Morning

Purchase OrderCustomerDelivery AddrSKUs

Slide 49

Infosets, XML-Schema, And PSVI

XML-InfosetSemantics of XML, not syntaxTree: parents, children, elements, & attributesAllows (encourages) schemaAny representation OK

XML-SchemaDatatype library and schema definitionComposed schema uniquely identified (URI)

PSVI – Post Schema Validated InfosetInfoset after validation against schemaCan leverage schema knowledge

Slide 50

Messages Need Immutable Contents and Immutable SchemaMessages must be immutable

Retries must not see differences…

Message schema must be immutableMust be able to unambiguously interpret the message

XML-Infosets and XML-Schema work well for messaging!

Service-A

Immutable MessagesImmutable Schema

Schema for the outside

is very differentfrom schema for

the inside!!!

Slide 51



Data: Then and Now





Inside Data



ConclusionSlide 52

Coping with Different SchemaDifferent services frequently have different

schemaHow do you represent a customer?What information about a customer is relevant?What special info does your business-unit keep about customers?

Is there an enterprise-wide customer-id?Just disambiguating the identity of a unique customer is fun!

How do you represent:Address?Status level (e.g. Gold-customer)?Special considerations?

Customers for General Electric:• Purchasers of aircraft engines• Purchasers of refrigerators• Purchasers of nuclear power plants• Purchasers of light bulbs• Purchasers of locomotives

Slide 53

ReadingService

Reader-Makes-Right And N-Squared Combinatorics

Reader-makes-rightReceiver fixes messageBased on both schemasBest possible mapping

N-squared combinatoricsEverybody knows everybodyN*(N-1) mappingsGets big fast!

Some datarequires

transformation

12 Services12 * 11 = 132

message transformers

Other datacopies through

Svc

Svc

Svc

Svc Svc Svc

Svc

Svc

Svc

SvcSvcSvc

Slide 54

Svc

Svc

Svc

Svc Svc Svc

Svc

Svc

Svc

SvcSvcSvc

Canonical

SourceService

DestinationServiceCanonical

SchemaTransforming

Service

Canonical Schema And Doubly-Lossy Transformations

Canonical schema is a standard representationMap everything to the canonical and back outFar fewer transformers required

Using a canonical schema is double-lossyInfo loss going in and info loss going out

Twotransformations

gives twicethe data loss

12 Services2 * N; 2*12=24

message transformersSlide 55

The Typical Approach To Integrating Schema

Classic Solution:Implement Canonical Schema

Practical to Build the Transformers2 * N (for N Different Schemas)

Specialize When NeededIf Doubly-Lossy Hurts, Then Reader-Makes-RightSpecialize Only Where Important

Hybrid Trades Work for Fidelity

The cool thing about services:• Rationalizing schema occurs for outside data• Inside a service, you have what you want!• You only have to cooperate on the outside!

Slide 56



Data: Then and Now





Inside Data



ConclusionSlide 57

Storing Incoming Data

When data arrives from the outside, you store it inside

Most services keep incoming dataKeep for processingKeep for auditing

Incoming Data

Internal Data

Slide 58

Extensibility versus ShreddingShredding the message

The incoming data is broken down to relational formEmpowers query and business intelligenceAuditing considerations

Typically, don’t want to change the message imagePreserve for auditingMay keep unshredded version also for non-repudiation

ExtensibilityThe sender added stuff you didn’t expectMay or may not know how utilize extensions

Extensibility fights shredding!Hard to map extensions to planned relational tablesOK to partially shred

Yields partial query benefits

Slide 59

Atomically Processing MessagesAtomic transactions consume messages

Each message is consumedData may be changedRecords outgoing responses

All of a service’s state lives in the database!It is transactionally modified

TransactionService Logic

Service-PrivateData

Service

Slide 60

Resource-Oriented-Data and Activity-Oriented-Data

Resource-Oriented-DataLives Longer Than One Long-Running-OperationChanged by Long-Running-Operations

Activity-Oriented-DataGets Modified byLong-Running-OperationsRetires On Completion ofLong-Running-Operation

Inventoryof SKU#71946


####

CustomerInfo for

Cust #8319“Sally’s Auto”

CustomerInfo for

Cust #8319“Sally’s Auto”

#Info#

PO#307654-03Order for

Joe’s Flowers

PO#307654-03Order for

Joe’s Flowers

#Master##LineItems#

ShoppingBasket

#1834953

ShoppingBasket

#1834953

####

Bank Balanceof Account#01600-18653

Bank Balanceof Account#01600-18653

#$$$#

Account-RecvInv#173-45 forPO#307654-03

Account-RecvInv#173-45 forPO#307654-03

#Info##Payments#

Shipping Order#86403 tied toPO#307654-03

Shipping Order#86403 tied toPO#307654-03

#Info##Shipped#

These classes of data have different characteristicsWe’re gonna examine some of the

consequences of these differences…

Slide 61

Containment of Activity-Oriented and Resource-Oriented Data

Activity-oriented and resource-oriented data live inside services

They are encapsulated within the serviceAccess is mediated by business logic


ResourceOriented

Data

####

ShoppingBasket

#1834953

ActivityOriented

Data

####

Internal Data

Slide 62

Retirement of DataData sometimes retires

It becomes read-onlyIt is referenced less and less frequently

It may get archived to tapeRarely, it is deleted

New regulations will make this very rare

Activity-oriented-data retires when the long-running operation completes

This may take weeks or months

Resource-oriented-data sometimes retiresThe resource-oriented-data may live a long time

Skus, accounts, employees, customers, etcThese may live for years Sometimes they leave

Some resources have bounded lifeHotel occupancy info; airplane seat assignments

Slide 63

High and Low Concurrency Updates

Updates of only activity-oriented dataThis is partitioned by long-running work

Almost never have concurrent work!!You can have it but it’s not a performance issue…

Activity-oriented data have low concurrency updates!

Updates affecting resource-oriented dataThe resource-oriented data MAY be in high demand

Possible that it is so desired that there is a queue for update

Resource-oriented data MAY have high concurrency updates

Slide 64



Data: Then and Now





Inside Data



ConclusionSlide 65

XML, SQL, and Objects

XmlSchematized representation of messagesSchema supports independent definition and extensibility

SQLStores relational data by valueAllows you to “relate” fields by valuesIncredibly query capabilitiesWorld-class management of data

ObjectsVery powerful software engineering toolBased on encapsulation

DataSQL

Slide 66

Bounded And Unbounded Data Representations

Relational is boundedOperations within the databaseValue comparisons only meaningful insideTightly managed schema

XML-Infoset is unboundedOpen (extensible) schema

Contributions to schema from who-knows-where

References (not just values)Uris known to be unique

XML-Infosets can be interpreted anywhere

Slide 67

Encapsulation and Anti-Encapsulation

SQL is anti-encapsulatedUPDATE WHERE

Query/update by joining anything with anything

Triggers/stored-procs are not strongly tied to protected data

XML is anti-encapsulatedPlease examine my public schema!

Components/objects offer encapsulationLong tradition of cheating

Reference passing to shared objectsWhacking on shared database

Slide 68

The ServiceIs a

Black Box!

A Service’s View of EncapsulationAnti-encapsulation is OK in its place

SQL’s anti-encapsulation is only seen by the local biz-logicXML’s anti-encapsulation only applies to the “public” behavior and data of the service

Encapsulation is strongly enforced by the serviceNo visibility is allowed to the internals of the service!

Business Request

Exported Data

Sanitized Datafor Export

PrivateInternal

Data

Data

Slide 69

Persistent ObjectID=Y

Persistent Objects and SQL StoragePersistent Objects

Encapsulated by LogicKept in SQLUses Optimistic Concurrency (Low Update)

Stored as Collection of RecordsMay Use Records in Many TablesKeys of Records Prefixed with Unique ID

This is the Object ID

Encapsulation by Convention

SQL

<record>ID-X <key1>Table-B

<record>ID-X <key2><record>ID-X <key3>

<record>ID-Y <key1><record>ID-Y <key2>

Database-Key<record>ID-X <key>

Table-A

<record>ID-Y <key><record>ID-Z <key>

Database-Key

Persistent ObjectsWork Great for

Activity-Oriented-Data

Slide 70

Interesting!

Maybe copied using tools like XML SQL stuff; shredding what you wantto do BI Analysis on. Maybe transformed in and out using business logic.

Kinds of Data

Probably ProbablyFodder for Business

Intelligence Analysis Definitely Definitely

Encapsulation Useful No No Yes Yes

Immutable orLow Concurrent Update orHighly Concurrent Update

Immutable Immutable(versions)

LowConcurrent

Update

HighlyConcurrent

Update

Requires Open Schemafor Interoperability Yes Yes No No

Durable storage in SQL? XML copyin SQL

XML copyin SQL Definitely Definitely

Represent in XML? Yes YesProbably

NotProbably

Not

Encapsulated Access? No NoYes: Use

Obj PersistYes:

Stateless

RequestResponse

ReferenceData

ActivityOriented

ResourceOriented

The Ruling Triumvirate

Impossible:Can’t see the data!

Problematic:Schema inconsistency

Outstanding

ArbitraryQueries

ImpossibleCan’t see the data!

Outstanding

Impossible:Centralized Schema

Independent Data Definition

Outstanding

Impossible:Open Schema

Not via SQLEnforced by DBA

Encapsulation(Controls Data)

SQL It is fantastic to compare anything to anything and combine anything with anything in Relational (within the bounded database)

XML It is possible to have independent definition of schema and data in XML-Infosets. You can independently extend, too.

Components/Objects

Provide encapsulation of data behind logic. Ensure enforcement ofbusiness rules. Eases composition of logic.

ObjectsEncapsulated Data

XMLUnbounded Schema

SQLBounded Schema

Strengths andWeaknesses

Each model’s strength is simultaneously its weakness!You can’t enhance one to add features of the other without breaking it!



Data: Then and Now





Inside Data



ConclusionSlide 73

Talking To The WorldUse XML-Infosets across services

Standard or proprietary marshaling formatsUse ML-Schema

Interactions across services include:Requests,Responses, andReference-data

Service-A

Slide 74

Storing The DataEnterprise-class services

Encapsulate a collection of private data

Store your data in a SQL database!Industrial strength store, performance, and management

Store incoming and outgoing messages in SQL

Remember incoming and outgoing for auditingRemember them for Biz IntelligenceTransactionally record outgoing messages for retry

DataSQL

Slide 75

Implementing the Code

Build your services with objectsOffer encapsulation of parts within the serviceThey are the best for software engineering!

EncapsulationServices are a big chunk

Completely encapsulate some durable dataObjects make littler chunks

Inside the serviceMake it easier to build

Slide 76

Putting It All Together!Services Need All Three!

XML-Infosets: Between the servicesObjects: Implementing the business logicSQL: Storing private data and messages

DataSQL

Objects Implementthe Biz Logic

SQL Holds the Data

XML-InfoSets forMessages Between Services



Data: Then and Now





Inside Data



ConclusionSlide 78

Data Inside and Outside Services

Data is different inside from outside

Outside the servicePassed in messagesUnderstood by sender and receiverIndependent schema definition importantExtensibility important

Inside the servicePrivate to serviceEncapsulated by service code

DataSQL

MSG

MSG

Data Outside the Service

Data Inside the Service

Slide 79

Session Objectives And Takeaways

Understand the use and interpretation of dataIt is different inside services than outside services

Many new issues arise in the world between services

Versioning, meta-data (schema), immutability, idempotence, and much moreThink carefully about the interactions across services

XML, SQL, and Objects all have their placeUnderstand their strengths and weaknesses and use them together to create the best services

Slide 80

Documents

Pat Helland Partner Architect Microsoft Corporation