Data on the Inside versus Data on the Outside Session Delivered By: Lars Laakes Architect Specialist, Financial Services Microsoft Canada [email protected]

Data on the Inside Data on the Inside versus versus Data on the OutsideData on the Outside

Session Delivered By:Session Delivered By:Lars LaakesLars LaakesArchitect Specialist, Financial ServicesArchitect Specialist, Financial ServicesMicrosoft CanadaMicrosoft [email protected]@microsoft.com

Source: Gartner Research (September 2004)

Magic Quadrant for Web-ServicesEnabled Software, 3Q04

Developed and Written By:Developed and Written By:Pat HellandPat HellandArchitectArchitectArchitecture Strategy TeamArchitecture Strategy TeamDeveloper and Platform EvangelismDeveloper and Platform EvangelismMicrosoft CorporationMicrosoft Corporation

Slide 2

What Is Hubble?What Is Hubble?Two Names for Architectural Thoughts:Two Names for Architectural Thoughts:

MetropolisMetropolisLooking at the History of Urban DevelopmentLooking at the History of Urban DevelopmentExplaining Service Oriented Architecture TrendsExplaining Service Oriented Architecture TrendsPredicting Upcoming ChangesPredicting Upcoming ChangesThree Talks So Far; More PlannedThree Talks So Far; More Planned

HubbleHubbleHard-Core Architectural Discussion of Issues Related to Hard-Core Architectural Discussion of Issues Related to Service Oriented ArchitecturesService Oriented ArchitecturesMany New Talks to Come… (Services, Data, Pub-Sub, etc)Many New Talks to Come… (Services, Data, Pub-Sub, etc)

Why “Hubble”? Edwin Hubble (1889-1953) showed the universe is expanding. As CPUs get faster, latency for communications is not dropping much. While a message is moving, more CPU work is lost next year than this.

“Computing is like Hubble’s Universe; everything is getting farther away from everything else” -- Pat Helland (circa 1992)

Slide 3

OutlineOutline Introduction: The Shift Towards ServicesIntroduction: The Shift Towards Services Introduction: The Shift Towards ServicesIntroduction: The Shift Towards Services

Behavior: Encapsulation and TrustBehavior: Encapsulation and Trust Behavior: Encapsulation and TrustBehavior: Encapsulation and Trust

Data: Then and NowData: Then and Now Data: Then and NowData: Then and Now

Outside Data: Reference DataOutside Data: Reference Data Outside Data: Reference DataOutside Data: Reference Data

Outside Data: Sending MessagesOutside Data: Sending Messages Outside Data: Sending MessagesOutside Data: Sending Messages

Outside Data: XML and SchemaOutside Data: XML and Schema Outside Data: XML and SchemaOutside Data: XML and Schema

Outside Data: Commonality of Schema and UnderstandingOutside Data: Commonality of Schema and Understanding Outside Data: Commonality of Schema and UnderstandingOutside Data: Commonality of Schema and Understanding

Inside DataInside Data Inside DataInside Data

Inside/Outside: Representations of DataInside/Outside: Representations of Data Inside/Outside: Representations of DataInside/Outside: Representations of Data

Inside/Outside: Tying It All TogetherInside/Outside: Tying It All Together Inside/Outside: Tying It All TogetherInside/Outside: Tying It All Together

ConclusionConclusion ConclusionConclusion

Slide 4












Slide 5

Service

PolicyPolicyPolicyPolicySchema and

Contract

Schema and Contract

Service-Oriented ArchitectureService-Oriented ArchitectureService-OrientationService-Orientation

IndependentIndependent Services ServicesChunks of Code and DataChunks of Code and Data

InterconnectedInterconnected via Messaging via Messaging

Four Basic Tenets:Four Basic Tenets:Boundaries Are ExplicitBoundaries Are ExplicitServices Are AutonomousServices Are AutonomousServices Share Schema Services Share Schema and Contractand Contract

Not ImplementationNot ImplementationService Compatibility Service Compatibility Is Based on PolicyIs Based on Policy

Slide 6

Services Communicate Services Communicate With MessagesWith Messages

Services Communicate with MessagesServices Communicate with MessagesNothing ElseNothing Else

No Other Knowledge about PartnerNo Other Knowledge about PartnerMay Be HeterogeneousMay Be Heterogeneous

Service-A Service-B

Slide 7

Data Inside and Outside ServicesData Inside and Outside Services

Data Is Different Inside from OutsideData Is Different Inside from Outside

Outside the ServiceOutside the ServicePassed in MessagesPassed in MessagesUnderstood by Sender and ReceiverUnderstood by Sender and ReceiverIndependent Schema Definition ImportantIndependent Schema Definition ImportantExtensibility ImportantExtensibility Important

Inside the ServiceInside the ServicePrivate to ServicePrivate to ServiceEncapsulated by Encapsulated by Service CodeService Code

DataDataSQL

MSG

MSG

Data Outside Data Outside the Servicethe Service

Data Inside Data Inside the Servicethe Service

Slide 8












Slide 9

Bounding Trust via EncapsulationBounding Trust via Encapsulation

Services Only Do Limited Things for Their PartnersServices Only Do Limited Things for Their PartnersThis Is How They Bound Their TrustThis Is How They Bound Their Trust

Encapsulation Is About Bounding TrustEncapsulation Is About Bounding TrustBusiness Logic Ensures Only the Desired Operations HappenBusiness Logic Ensures Only the Desired Operations Happen

No Changes to the Data Occur Except Through Locally No Changes to the Data Occur Except Through Locally Controlled Business Logic!Controlled Business Logic!

Service

Things I’ll Do for Outsiders• Deposit• Withdrawal• Transfer• Account Balance Check

Things I’ll Do for Outsiders• Deposit• Withdrawal• Transfer• Account Balance Check

Slide 10

Encapsulating Both Change and ReadsEncapsulating Both Change and ReadsEncapsulating ChangeEncapsulating Change

Ensures Integrity of the Service’s WorkEnsures Integrity of the Service’s WorkEnsures Integrity of the Service’s DataEnsures Integrity of the Service’s Data

Encapsulating Exported Data for ReadEncapsulating Exported Data for ReadEnsures Privacy by Controlling What’s ExportedEnsures Privacy by Controlling What’s ExportedAllows Planning for Loose Coupling and ExpirationsAllows Planning for Loose Coupling and Expirations

E.g. Wednesday’s Price-ListE.g. Wednesday’s Price-List

Business Request

Exported Data

Sanitized Datafor Export

PrivateInternal

Data

Data

Slide 11

Trust and TransactionsTrust and Transactions

Some Propose Atomic Transactions Across ServicesSome Propose Atomic Transactions Across ServicesE.g. WS-TransactionsE.g. WS-Transactions

Requires Holding LocksRequires Holding LocksLots of Trust in Timely UnlockLots of Trust in Timely Unlock

Doesn’t Sound Autonomous and Independent to me…Doesn’t Sound Autonomous and Independent to me…

Debate Is the Debate Is the Definition Definition of the Word of the Word ServiceServiceRequires Autonomy and Independence?Requires Autonomy and Independence?

Allows Intimacy across Service Boundaries?Allows Intimacy across Service Boundaries?

There Will Be Code Connected by 2-Phase CommitThere Will Be Code Connected by 2-Phase Commit

Same Service or in Different Services?Same Service or in Different Services?

For This Talk, For This Talk, I Presume No Cross-Service TxsI Presume No Cross-Service TxsSimply the Definition of the Word “Service”Simply the Definition of the Word “Service”

Slide 12

Interconnecting with Independent ServicesInterconnecting with Independent Services

Services Are Connected by MessagingServices Are Connected by MessagingThe Only Interaction Between Two Services Is by the Messages that The Only Interaction Between Two Services Is by the Messages that They ExchangeThey Exchange

SchemaSchema: The Formats of the Individual Messages: The Formats of the Individual MessagesContracts Contracts :The Allowable Sequences of Messages:The Allowable Sequences of Messages

Service

ContractContract

Reject-Order

Accept-OrderOneOf

TentativePlace-Order

ConfirmPlace-Order

OneOf

CancelPlace-Order

Slide 13

Operators and OperandsOperators and OperandsMessages Contain Messages Contain OperatorsOperators

Requests a Business OperationRequests a Business OperationOperators Provide Business SemanticsOperators Provide Business SemanticsPart of the Contract between the Two ServicesPart of the Contract between the Two Services

OperatorOperator Messages Contain Messages Contain OperandsOperandsDetails Needed To Do the Business OperationDetails Needed To Do the Business OperationThe Sending Service Must Put Them into the MessageThe Sending Service Must Put Them into the Message

Service

Deposit

OperatorOperands

Slide 14

Where Do Operands Come From?Where Do Operands Come From?

Operands Come from Operands Come from Reference DataReference DataNew Kind of Data in SOANew Kind of Data in SOA

Except It’s Not New; Except It’s Not New; We’ve Done Variations of SOA for Decades… We’ve Done Variations of SOA for Decades…

We’re Just Getting Better at It!We’re Just Getting Better at It!

CataloguesCatalogues

Reference Data is Versioned and Reference Data is Versioned and each Version Is Immutableeach Version Is Immutable

Immutable Images Are Shared Across Many ServicesImmutable Images Are Shared Across Many Services

We Will Talk About the Creation, Publication, and We Will Talk About the Creation, Publication, and Management of Reference DataManagement of Reference Data

Slide 15












Slide 16

Transactions and Inside DataTransactions and Inside DataTransactions Make You Feel AloneTransactions Make You Feel Alone

No One Else Manipulates the Data When You AreNo One Else Manipulates the Data When You Are

Transactional SerializabilityTransactional SerializabilityThe Behavior Is As If a Serial Order ExistsThe Behavior Is As If a Serial Order Exists

TkTl

Tm

Tn

ToTh

Tg TjTe

Tf

Tb

Ta

Tc

Td

Ti

These TransactionsPrecede Ti

These TransactionsFollow Ti

Ti Doesn’t Know About TheseTransactions and They Don’t

Know About Ti

TransactionSerializability

TkTl

Tm

Tn

ToThTh

Tg TjTe

Tf

Tb

Ta

Tc

Td

Ti

These TransactionsPrecede Ti

These TransactionsFollow Ti

Ti Doesn’t Know About TheseTransactions and They Don’t

Know About Ti

TransactionSerializability

Slide 17

Life in the “Now”Life in the “Now”Transactions Live in the “Now” Inside ServicesTransactions Live in the “Now” Inside Services

Time Marches ForwardTime Marches ForwardTransactions Commit Transactions Commit Advancing TimeAdvancing TimeTransactions See Transactions See the Committed the Committed TransactionsTransactions

A Service’s A Service’s Biz-Logic Lives Biz-Logic Lives in the “Now”in the “Now”

Service

Each Transaction Only Sees a Simple Advancing of Time with a Clear Set of

Preceding Transactions

ServiceServiceService

Each Transaction Only Sees a Simple Advancing of Time with a Clear Set of

Preceding Transactions

Slide 18

Sending Unlocked Data Isn’t “Now”Sending Unlocked Data Isn’t “Now”Messages Contain Unlocked DataMessages Contain Unlocked Data

Assume No Shared TransactionsAssume No Shared Transactions

Unlocked Data May ChangeUnlocked Data May ChangeUnlocking It Allows ChangeUnlocking It Allows Change

Messages Are Not From the “Now”Messages Are Not From the “Now”They Are From the PastThey Are From the Past

There Is No Simultaneity At a Distance!• Similar to Speed of Light• Knowledge Travels at Speed of Light• By the Time You See a Distant Object It May Have Changed!• By the Time You See a Message, the Data May Have Changed!

Services, Transactions, and Locks Bound Simultaneity!• Inside a Transaction, Things Appear Simultaneous (to Others)• Simultaneity Only Inside a Transaction!• Simultaneity Only Inside a Service!

Slide 19

Outside Data: a Blast from the PastOutside Data: a Blast from the Past

All Data Seen From a Distant Service Is From the “Past”All Data Seen From a Distant Service Is From the “Past”By the Time You See It, It Has Been Unlocked and May ChangeBy the Time You See It, It Has Been Unlocked and May Change

Each Service Has Its Own PerspectiveEach Service Has Its Own PerspectiveInside Data Is “Now”; Outside Data Is “Past”Inside Data Is “Now”; Outside Data Is “Past”My Inside Is Not Your Inside; My Outside Is Not Your OutsideMy Inside Is Not Your Inside; My Outside Is Not Your Outside

All Data From Distant Stars Is From the Past• 10 Light Years Away; 10 Year Old Knowledge• The Sun May Have Blown Up 5 Minutes Ago

• We Won’t Know for 3 Minutes More…

Going to SOA Is Like Going From Newtonian to Einstonian Physics• Newton’s Time Marched Forward Uniformly

• Instant Knowledge• Before SOA, Distributed Computing Many Systems Look Like One

• RPC, 2-Phase Commit, Remote Method Calls…• In Einstein’s World, Everything Is “Relative” To One’s Perspective• SOA Has “Now” Inside and the “Past” Arriving in Messages

Slide 20

Versioned Images of a Single SourceVersioned Images of a Single Source

A Sequence of Versions Describing Changes to DataA Sequence of Versions Describing Changes to DataUpdates FromUpdates FromOne ServiceOne Service

Owner ControlledOwner ControlledOwner Changes Owner Changes the Datathe DataSends Changes Sends Changes as Messagesas Messages

Data Is SeenData Is SeenAs AdvancingAs AdvancingVersionsVersions

ListeningPartner

Service-1

ListeningPartner

Service-5

ListeningPartner

Service-7

ListeningPartner

Service-8

Tuesday’sPrice-List

Wednesday’sPrice-List



Monday’sPrice-List



Monday’sPrice-List


Data Owning Service

Price-List

ListeningPartner

Service-1

ListeningPartner

Service-5

ListeningPartner

Service-7

ListeningPartner

Service-8

Tuesday’sPrice-ListTuesday’sPrice-ListTuesday’sPrice-List










Monday’sPrice-ListMonday’sPrice-ListMonday’sPrice-List





Monday’sPrice-ListMonday’sPrice-ListMonday’sPrice-List


Data Owning Service

Price-List

Data Owning Service

Price-List

Slide 21

Operators: Hope for the FutureOperators: Hope for the FutureMessages May Contain OperatorsMessages May Contain Operators

Requests for Business Functionality Part of the ContractRequests for Business Functionality Part of the ContractService-B Sends an Operator to Service-AService-B Sends an Operator to Service-A

If Service-A Accepts the Operator, It Is Part of Its FutureIf Service-A Accepts the Operator, It Is Part of Its FutureIt Changes the State ofIt Changes the State ofService-AService-A

Service-B Is HopefulService-B Is HopefulIt Wants Service-A To DoIt Wants Service-A To Dothe Workthe WorkWhen It Receives a Reply,When It Receives a Reply,It’s Future Is Changed!It’s Future Is Changed!

OperatorResponse

OperatorRequest

InvokingPartner

Service-B

InvokedPartner

Service-A

Hopeful for the Future…Decides to Issue Request

Ever Hopeful,Waiting for aResponse

Hopes Fulfilled,the Future Is Now

BlithelyIgnorant andMinding Its Own Business

A Future ForeverAltered by theProcessing of theRequest fromService-B

OperatorResponseOperator

ResponseOperator

ResponseOperator

Response

OperatorRequestOperatorRequestOperatorRequestOperatorRequest

InvokingPartner

Service-B

InvokedPartner

Service-A

Hopeful for the Future…Decides to Issue Request

Ever Hopeful,Waiting for aResponse

Hopes Fulfilled,the Future Is Now

BlithelyIgnorant andMinding Its Own Business

A Future ForeverAltered by theProcessing of theRequest fromService-B

Slide 22

Operands: Past and FutureOperands: Past and FutureOperands May Live in the PastOperands May Live in the Past

Values Published As Reference DataValues Published As Reference DataCome From Service-A’s PastCome From Service-A’s Past

Operands May Live in the FutureOperands May Live in the FutureThey May Contain a Proposed Value Submitted to Service-AThey May Contain a Proposed Value Submitted to Service-A

Service-B Preparing a Request for Service-A

Friday’s Price-ListPublished:11PM Thursday

OperatorOperands

On Friday, Operands Are Extracted from

the Price-List Publishedon Thursday

Deposit

Service-B Preparing a Request for Service-A



OperatorOperands

On Friday, Operands Are Extracted from

the Price-List Publishedon Thursday

DepositDeposit

Slide 23

Between Services: Life in the “Then”Between Services: Life in the “Then”Everything Between Services Lives in the Past or FutureEverything Between Services Lives in the Past or Future

Operators Live in the FutureOperators Live in the FutureOperands Live in the Past or the FutureOperands Live in the Past or the Future

It’s Not Meaningful to Speak of “Now” Between ServicesIt’s Not Meaningful to Speak of “Now” Between ServicesNo Shared Transactions No Shared Transactions No Simultaneity No Simultaneity

Life in the “Then”Life in the “Then”Past or FuturePast or FutureNot NowNot Now

Each Service HasEach Service Hasa Separate “Now”a Separate “Now”

Different TemporalDifferent TemporalEnvironments!Environments!

Service-1

Service-2

Service-4

Service-3No Notion No Notion of of ““NowNow””

in Betweenin BetweenServices!Services!

Service-1Service-1Service-1

Service-2Service-2

Service-4Service-4

Service-3Service-3No Notion No Notion of of ““NowNow””

in Betweenin BetweenServices!Services!

Slide 24

Services: Dealing with “Now” and “Then”Services: Dealing with “Now” and “Then”

Services Make the “Now” Meet the “Then”Services Make the “Now” Meet the “Then”Each Service Lives in Its Own “Now”Each Service Lives in Its Own “Now”Messages Come and Go Dealing with the “Then”Messages Come and Go Dealing with the “Then”The Business-Logic of the Service Must Reconcile This!!The Business-Logic of the Service Must Reconcile This!!

The World Is No Longer Flat!• SOA Is Recognizing That There Is More Than One Computer• Multiple Machines Mean Multiple Time Domains• Multiple Time Domains Mandate We Cope with Ambiguity to Allow Coexistence, Cooperation, and Joint Work

Example: Accepting an Order• A Biz Publishes Daily Prices• Probably Want to Accept Yesterday’s Prices for a While• Tolerance for Time Differences Must Be Programmed

Example: “Usually Ships in 24 Hours”• Order Processing Has Old Info• Available Inventory Not Accurate• Deliberately “Fuzzy”• Allows Both Sides to Cope with Difference in Time Domains!

Slide 25












Slide 26

What Is Reference Data?What Is Reference Data?

Reference DataReference Data Is Published Across Service Boundaries Is Published Across Service BoundariesFor Each Collection of Reference Data:For Each Collection of Reference Data:

One Service Creates and Publishes the DataOne Service Creates and Publishes the Data

Other Services Receive Periodic Versions of the DataOther Services Receive Periodic Versions of the Data

Purposes for Reference Data

Historic Artifacts

Shared Collections of Data

Operands

Slide 27

Reference Data: Reference Data: Operands for the OperatorsOperands for the Operators

As Discussed Above, Messages Across Services Invoke As Discussed Above, Messages Across Services Invoke Business Operations…Business Operations…

Each Service-to-Service Message Is an Each Service-to-Service Message Is an OperatorOperator

Each Operator Message Is Filled with OperandsEach Operator Message Is Filled with OperandsParameters, Options, Customer-Id, Parts-Being-Ordered, etcParameters, Options, Customer-Id, Parts-Being-Ordered, etc

The Data for These Operands Is Published as The Data for These Operands Is Published as Reference DataReference Data

Service

Deposit

OperatorOperands

Slide 28

Reference Data: Historic ArtifactsReference Data: Historic ArtifactsHistoric Artifacts Report on What Happened in the PastHistoric Artifacts Report on What Happened in the Past

Sometimes These Snapshots Need to Be Sent to Other ServicesSometimes These Snapshots Need to Be Sent to Other Services

Examples:Examples:Sales Quarterly ResultsSales Quarterly ResultsMonthly Bank StatementsMonthly Bank StatementsAny and All Monthly BillsAny and All Monthly Bills

Well… Well… Both Requests for Payment (Operations) and the Both Requests for Payment (Operations) and the Historic Artifact of How Much Power You Used…Historic Artifact of How Much Power You Used…

Inventory Status at End of QuarterInventory Status at End of Quarter

Service(Bank)

BankStatementJan-2004

Slide 29

Reference Data: Reference Data: “Shared Collections of Data”“Shared Collections of Data”

Many Services May Need Access to the Same DataMany Services May Need Access to the Same DataThe Data Is Changing…The Data Is Changing…Someone Owns Updating and Distributing the Data…Someone Owns Updating and Distributing the Data…

Examples:Examples:Customer DatabaseCustomer DatabaseEmployee DatabaseEmployee DatabaseParts Database and Price-ListParts Database and Price-List

Ref Vers#23of Employee

DataUpdate!


Data

AuthoritativeEmployee

Data – Vers#23

AuthoritativeEmployee

Data – Vers#24

Sales Service

AuthoritativeCustomer

Data

HR Service


DataUpdateEmployees

UpdateEmployees

Vers#24Vers#24

Slide 30

Publishing Versioned Reference DataPublishing Versioned Reference Data

The Owner of Data Periodically PublishesThe Owner of Data Periodically PublishesUsing Whatever Messaging Technique It WantsUsing Whatever Messaging Technique It Wants

Publications Are Publications Are AlwaysAlways Versioned VersionedThe Version Numbers IncreaseThe Version Numbers Increase

44Request

Uses: Vers-ZService-B

33A’s-DataVers-Z

22A’s-DataVers-Y

A’s-DataVers-X

11

Service-A

Slide 31

22RequestUses: Vers-XPlease MakeData Change

Business Operations May Request Business Operations May Request ChangesChangesIf a Non-Owner Wants a Change It Must Do a Biz-OperationIf a Non-Owner Wants a Change It Must Do a Biz-Operation

This is a Request Sent to the Owning ServiceThis is a Request Sent to the Owning ServiceThe Owning Service May Agree to the Operation Causing ChangeThe Owning Service May Agree to the Operation Causing Changeto the Data in Questionto the Data in QuestionIf It Changes, This Affects the Next VersionIf It Changes, This Affects the Next Version

Service-B33A’s-Data

Vers-Y

A’s-DataVers-X

11

OwningService-A

Slide 32

Optimistic Concurrency Control:Optimistic Concurrency Control:Anti-EncapsulationAnti-Encapsulation

What Is Optimistic Concurrency Control?What Is Optimistic Concurrency Control?Data Is ReadData Is ReadChanges Are Made and Submitted to the Data’s OwnerChanges Are Made and Submitted to the Data’s OwnerIf the Original Data Hasn’t Changed, the New Changes Are AppliedIf the Original Data Hasn’t Changed, the New Changes Are Applied

This Assumes the Remote System Should Be Able to Write Directly on the DataThis Assumes the Remote System Should Be Able to Write Directly on the DataThis Is a Trusting Relationship… This Is a Trusting Relationship… Not Autonomous!Not Autonomous!

Autonomy and Updates to DataAutonomy and Updates to DataAutonomy Means Independent ControlAutonomy Means Independent Control

My Local Biz-Logic Decides How My Data Changes!My Local Biz-Logic Decides How My Data Changes!If You Want a Change, Ask Me To Do a Business OpIf You Want a Change, Ask Me To Do a Business OpIt’s My Data…I’ll Decide How It Changes!It’s My Data…I’ll Decide How It Changes!

Slide 33

Example: Updating the Example: Updating the Customer’s AddressCustomer’s AddressWhat About a Salesperson Updating What About a Salesperson Updating a Customer’s Address?a Customer’s Address?

Shouldn’t That Just Be Optimistic Concurrency Control?Shouldn’t That Just Be Optimistic Concurrency Control?

No! It Should Invoke Business Logic with a Request!No! It Should Invoke Business Logic with a Request!Not All Fields of the Customer Record Should Be Updated Not All Fields of the Customer Record Should Be Updated by Sales Peopleby Sales PeopleRequests Across Service Boundaries Invoke Business Requests Across Service Boundaries Invoke Business Logic when the Customer Address Is ChangedLogic when the Customer Address Is Changed

Slide 34












Slide 35

Immutable And/Or Versioned DataImmutable And/Or Versioned Data

Data May Be Data May Be ImmutableImmutableOnce Written, It Is UnchangeableOnce Written, It Is Unchangeable

Immutable Data Needs an IDImmutable Data Needs an IDFrom the ID, Comes the From the ID, Comes the SameSame Data DataNo Matter No Matter WhenWhen, No Matter , No Matter WhereWhere

VersionsVersions Are Immutable Are ImmutableEach New Version Is IdentifiedEach New Version Is IdentifiedGiven the Identifier, the Same Data ComesGiven the Identifier, the Same Data Comes

Version Independent IdentifiersVersion Independent IdentifiersLet You Ask for a Recent VersionLet You Ask for a Recent Version

•Windows NT4, SP1Windows NT4, SP1• The Same Set of BitsThe Same Set of Bits Every Time Every Time

•Windows NT4, SP1Windows NT4, SP1• The Same Set of BitsThe Same Set of Bits Every Time Every Time

New York Times; 7/3/03 New York Times; 7/3/03 • Specific Version of the Paper Specific Version of the Paper -- Contents Don’t Change -- Contents Don’t Change

New York Times; 7/3/03 New York Times; 7/3/03 • Specific Version of the Paper Specific Version of the Paper -- Contents Don’t Change -- Contents Don’t Change

Latest SP of NT4Latest SP of NT4• Definitely NT4, Definitely NT4, Results Vary Over Time Results Vary Over Time

Latest SP of NT4Latest SP of NT4• Definitely NT4, Definitely NT4, Results Vary Over Time Results Vary Over Time

Recent NY TimesRecent NY Times• Maybe Today’s,Maybe Today’s, Maybe Yesterday’s Maybe Yesterday’s

Recent NY TimesRecent NY Times• Maybe Today’s,Maybe Today’s, Maybe Yesterday’s Maybe Yesterday’s

VersionIndependent

Slide 36

Immutability of MessagesImmutability of Messages

Retries are a Fact of LifeRetries are a Fact of LifeZero or more delivery semanticsZero or more delivery semantics

Messages Must Be ImmutableMessages Must Be ImmutableRetries Must Not See Differences…Retries Must Not See Differences…Once It’s Sent, You Can’t Un-send!Once It’s Sent, You Can’t Un-send!

Service-A

Once It’s Outside,Once It’s Outside,It’s Immutable!It’s Immutable!

Slide 37

To Cache Or Not To CacheTo Cache Or Not To Cache

OK to Cache Immutable DataOK to Cache Immutable DataIt’s Never WrongIt’s Never WrongNever Have to Invalidate!Never Have to Invalidate!

Caching Should Only Be Used for Immutable DataCaching Should Only Be Used for Immutable DataCaching Data that Changes May Lead to AnomaliesCaching Data that Changes May Lead to Anomalies

Consider Caching Data Labeled with Consider Caching Data Labeled with a a Version Dependent IDVersion Dependent ID

Because Versions Are Immutable It Will WorkBecause Versions Are Immutable It Will Work

Store the Mapping from Version Independent to Store the Mapping from Version Independent to Version Dependent in an Accurate LocationVersion Dependent in an Accurate Location

Slide 38

Normalization And Immutable DataNormalization And Immutable Data

Databases Design for Normalized DataDatabases Design for Normalized DataCan Be Changed Without “Funny Behavior”Can Be Changed Without “Funny Behavior”Each Data Item Lives in One PlaceEach Data Item Lives in One Place

Sometimes Data Should Be De-NormalizedSometimes Data Should Be De-NormalizedIf Data Is Immutable It’s OKIf Data Is Immutable It’s OK

Emp # Emp Name Mgr # Mgr NameEmp Phone Mgr Phone

47 Joe 13 Sam5-1234 6-9876

18 Sally 38 Harry3-3123 5-6782

91 Pete 13 Sam2-1112 6-9876

66 Mary 02 Betty5-7349 4-0101

Classic problemwith de-normalization

Can’t updateSam’s phone #since there aremany copies

De-normalization isOK if you aren’t going to update!

Slide 39

Stability Of DataStability Of Data

Immutability Isn’t Enough!Immutability Isn’t Enough!We Need a Common UnderstandingWe Need a Common UnderstandingPresident Bush President Bush 1990 vs. President Bush 1990 vs. President Bush 2004 2004

Stable DataStable Data Has a Clearly Understood Meaning Has a Clearly Understood MeaningThe Schema Must Be Clearly UnderstoodThe Schema Must Be Clearly UnderstoodThe Interpretation of Values Must Be UnambiguousThe Interpretation of Values Must Be Unambiguous

AdviceAdvice• Don’t Recycle Don’t Recycle Customer-IDs Customer-IDs

AdviceAdvice• Don’t Recycle Don’t Recycle Customer-IDs Customer-IDs

ObservationObservation• Anything CalledAnything Called “Current” Is Not Stable “Current” Is Not Stable

ObservationObservation• Anything CalledAnything Called “Current” Is Not Stable “Current” Is Not Stable

SuggestionSuggestion• Timestamping or Timestamping or Versioning Makes Versioning Makes Stable Data Stable Data

SuggestionSuggestion• Timestamping or Timestamping or Versioning Makes Versioning Makes Stable Data Stable Data

ObservationObservation• A Monthly Bank StatementA Monthly Bank Statement Is Stable Data Is Stable Data

ObservationObservation• A Monthly Bank StatementA Monthly Bank Statement Is Stable Data Is Stable Data

Slide 40

A Few Thoughts on Stable DataA Few Thoughts on Stable Data

Outside Data Must Be StableOutside Data Must Be StableConsistent Interpretation Across Valid Spaces and Consistent Interpretation Across Valid Spaces and TimesTimes

Inside Data May Be StableInside Data May Be StableNotably, When It Is the Same Data as Outside Data…Notably, When It Is the Same Data as Outside Data…

Sometimes Data Inside Is Not StableSometimes Data Inside Is Not StableClassic Normalization for Vibrant UpdateClassic Normalization for Vibrant UpdateNeeds to Be Cast Into a Stable Shape to Send OutsideNeeds to Be Cast Into a Stable Shape to Send Outside

Slide 41

Validity Of Data In Validity Of Data In Bounded Space And TimeBounded Space And Time

Bounding the Valid TimesBounding the Valid TimesIt May Have an ExpirationIt May Have an Expiration

Bounding the Valid LocationsBounding the Valid LocationsRestrictions on Where the Data Is ValidRestrictions on Where the Data Is Valid

When Valid, the Data Should Be:When Valid, the Data Should Be:ImmutableImmutable (the ID Yields the Same Bits) (the ID Yields the Same Bits)StableStable (the Meaning Is Clear) (the Meaning Is Clear)

““Offer Good to Washington State Residents Only”Offer Good to Washington State Residents Only”““Offer Good to Washington State Residents Only”Offer Good to Washington State Residents Only”

““Offer Good Until Next Tuesday”Offer Good Until Next Tuesday”

Data Valid For Service-X OnlyData Valid For Service-X OnlyData Valid For Service-X OnlyData Valid For Service-X OnlyPrice-List Valid Price-List Valid Until Dec 31stUntil Dec 31st

Slide 42

Rules For Sending Data In MessagesRules For Sending Data In Messages

Identify theIdentify theMessageMessage

Identify theIdentify theMessageMessage

Put Unique ID in All MessagesPut Unique ID in All MessagesPart of the Unique ID May Be a Version…Part of the Unique ID May Be a Version…Put Unique ID in All MessagesPut Unique ID in All MessagesPart of the Unique ID May Be a Version…Part of the Unique ID May Be a Version…

ImmutableImmutableDataData

ImmutableImmutableDataData

Don’t Change the Data Associated withDon’t Change the Data Associated withthe Unique ID; Never Return Different Bitsthe Unique ID; Never Return Different BitsDon’t Change the Data Associated withDon’t Change the Data Associated withthe Unique ID; Never Return Different Bitsthe Unique ID; Never Return Different Bits

OK toOK toCacheCacheOK toOK toCacheCache The Same Bits Will Always Be ReturnedThe Same Bits Will Always Be ReturnedThe Same Bits Will Always Be ReturnedThe Same Bits Will Always Be Returned

Define ValidDefine ValidRangesRanges

Define ValidDefine ValidRangesRanges

Valid for a Certain Time Period and OverValid for a Certain Time Period and OverSome Space; OK to Always Be ValidSome Space; OK to Always Be ValidValid for a Certain Time Period and OverValid for a Certain Time Period and OverSome Space; OK to Always Be ValidSome Space; OK to Always Be Valid

Must BeMust BeStableStable

Must BeMust BeStableStable

Must Ensure There Is Never Any ConfusionMust Ensure There Is Never Any ConfusionAbout the Meaning (Within Valid Range)About the Meaning (Within Valid Range)Must Ensure There Is Never Any ConfusionMust Ensure There Is Never Any ConfusionAbout the Meaning (Within Valid Range)About the Meaning (Within Valid Range)

Slide 43












Slide 44

SQL, DDL, and SerializabilitySQL, DDL, and Serializability

SQL’s DDL (Data Definition Language) is TransactionalSQL’s DDL (Data Definition Language) is TransactionalChanges Are Made Using TransactionsChanges Are Made Using TransactionsThe Structure of the Data May Be ChangedThe Structure of the Data May Be Changed

The Interpretation After the DDL Change Is DifferentThe Interpretation After the DDL Change Is Different

DDL Lives Within the Time Scope of the DatabaseDDL Lives Within the Time Scope of the DatabaseThe Database’s Shape Evolves Over TimeThe Database’s Shape Evolves Over TimeDDL Is the Change Agent for This EvolutionDDL Is the Change Agent for This Evolution

SQL Lives in the “Now”SQL Lives in the “Now”Each Transaction’s Execution Is Meaningful Only Within the Each Transaction’s Execution Is Meaningful Only Within the Schema Definition at the Moment of Its ExecutionSchema Definition at the Moment of Its ExecutionSerializability Makes This Crisp and Well-DefinedSerializability Makes This Crisp and Well-Defined

Slide 45

Schema and Immutable MessagesSchema and Immutable Messages

When a Message Is Sent, It Must Be ImmutableWhen a Message Is Sent, It Must Be ImmutableIt Is Crossing Temporal BoundariesIt Is Crossing Temporal BoundariesRetries Mustn’t Give Different ResultsRetries Mustn’t Give Different Results

The Message’s Schema Must Be ImmutableThe Message’s Schema Must Be ImmutableIt Makes a Mess If the Interpretation of the Message ChangesIt Makes a Mess If the Interpretation of the Message Changes

Service-AMessage

MessageSchema

Immutable Message

Immutable Schema for the Message

Slide 46

Immutable Schema and Its IdentifiersImmutable Schema and Its Identifiers

Immutable Schema Needs an IdentifierImmutable Schema Needs an IdentifierIt Must Be Possible to Unambiguously Identify the SchemaIt Must Be Possible to Unambiguously Identify the SchemaThis Must Occur Across the Namespaces of Sender and This Must Occur Across the Namespaces of Sender and ReceiverReceiverThe Schema Definition Must Never ChangeThe Schema Definition Must Never Change

Given the Identifier, the Same Schema Is ReturnedGiven the Identifier, the Same Schema Is Returned

URIs (Universal Resource Identifiers) Work WellURIs (Universal Resource Identifiers) Work WellGuaranteed to Be UniqueGuaranteed to Be Unique

If You Follow the RulesIf You Follow the Rules

URLs (Universal Resource Locators) Are CoolURLs (Universal Resource Locators) Are CoolURLs Are URIsURLs Are URIsAlso Tell You a Also Tell You a LocationLocation To Get the Stuff (e.g. the Schema) To Get the Stuff (e.g. the Schema)

Slide 47

Composition of Schema as a DAGComposition of Schema as a DAGSchema Make Contain Sub-SchemaSchema Make Contain Sub-Schema

Inside the Message Are Chunks of DataInside the Message Are Chunks of DataA Purchase-Order May Contain Customer InformationA Purchase-Order May Contain Customer Information

They Have Their Own DefinitionsThey Have Their Own Definitions

The Sub-Schema Are Referenced by IdentifierThe Sub-Schema Are Referenced by IdentifierThis Leads to a Tree of References to Immutable SchemaThis Leads to a Tree of References to Immutable Schema

It’s Really a DAG (Directed Acyclic Graph)It’s Really a DAG (Directed Acyclic Graph)Sometimes, Different Sub-Schema Reference the Same StuffSometimes, Different Sub-Schema Reference the Same Stuff

Purchase OrderPurchase Order

CustomerCustomerDelivery AddrDelivery AddrSKUsSKUs

Customer Customer

NameNameAddressAddressCredit RatingCredit Rating

Address Address

Number/StreetNumber/StreetCity/StateCity/StatePostal CodePostal CodeCountryCountry

SKU SKU PartPart ColorColor SizeSize

Slide 48

Versioning and SchemaVersioning and SchemaFrequently, Schema Is VersionedFrequently, Schema Is Versioned

A New Format of the Schema Is CreatedA New Format of the Schema Is CreatedIt Is Given a New IdentifierIt Is Given a New Identifier

Version Independent Schema IdentifiersVersion Independent Schema IdentifiersSpecify a Set of Versions for a Type of SchemaSpecify a Set of Versions for a Type of SchemaThe Set May Evolve Over TimeThe Set May Evolve Over Time

Version Dependent Schema IdentifiersVersion Dependent Schema IdentifiersSpecify a Specific Version of a Specific SchemaSpecify a Specific Version of a Specific SchemaThe Version-Dependent Schema Is ImmutableThe Version-Dependent Schema Is Immutable

Messages Messages AlwaysAlways Specify a Specify a Version-DependentVersion-Dependent Schema SchemaThis Ensures No AmbiguityThis Ensures No Ambiguity

Slide 49

Extensibility and SchemaExtensibility and SchemaExtensibility Is the Addition of Non-Schema Specified Extensibility Is the Addition of Non-Schema Specified Information Into the MessageInformation Into the Message

The Schema Does Not Specify the Additional StuffThe Schema Does Not Specify the Additional StuffThe Sender Wanted to Add It AnywayThe Sender Wanted to Add It Anyway

Adding Extensions Is Like Scribbling in the MarginsAdding Extensions Is Like Scribbling in the MarginsSometimes Adding Notes to a Form Helps!Sometimes Adding Notes to a Form Helps!Sometimes It Does No Good at All!Sometimes It Does No Good at All!

Service-A

Purchase Order Purchase Order

CustomerCustomerDelivery AddrDelivery Addr

SKUsSKUsDon’t Deliver in MorningDon’t Deliver in Morning

Purchase OrderPurchase Order

CustomerCustomerDelivery AddrDelivery AddrSKUsSKUs

Slide 50

Infosets, XML-Schema, And PSVIInfosets, XML-Schema, And PSVI

XML-InfosetXML-InfosetSemantics of XML, Not SyntaxSemantics of XML, Not SyntaxTree: Parents, Children, Elements, & AttributesTree: Parents, Children, Elements, & AttributesAllows (Encourages) SchemaAllows (Encourages) SchemaAny Representation OKAny Representation OK

XML-SchemaXML-SchemaDatatype Library and Schema DefinitionDatatype Library and Schema DefinitionComposed Schema Uniquely Identified (URI)Composed Schema Uniquely Identified (URI)

PSVI – Post Schema Validated InfosetPSVI – Post Schema Validated InfosetInfoset After Validation Against SchemaInfoset After Validation Against SchemaCan Leverage Schema KnowledgeCan Leverage Schema Knowledge

Slide 51

Messages Need Immutable Messages Need Immutable Contents and Immutable SchemaContents and Immutable Schema

Messages Must Be ImmutableMessages Must Be ImmutableRetries Must Not See Differences…Retries Must Not See Differences…

Message Schema Must Be ImmutableMessage Schema Must Be ImmutableMust Be Able to Unambiguously Interpret the MessageMust Be Able to Unambiguously Interpret the Message

XML-Infosets and XML-Schema Work Well for Messaging!XML-Infosets and XML-Schema Work Well for Messaging!

Service-A

Immutable MessagesImmutable MessagesImmutable SchemaImmutable Schema

Schema for the Outside

Is Very Differentfrom Schema for

the Inside!!!

Slide 52












Slide 53

Coping with Different SchemaCoping with Different Schema

Different Services Frequently Have Different SchemaDifferent Services Frequently Have Different SchemaHow Do You Represent a Customer?How Do You Represent a Customer?What Information about a Customer Is Relevant?What Information about a Customer Is Relevant?What Special Info Does Your Business-Unit Keep about What Special Info Does Your Business-Unit Keep about Customers?Customers?

Is There an Enterprise-Wide Customer-ID?Is There an Enterprise-Wide Customer-ID?Just Disambiguating the Just Disambiguating the IdentityIdentity of a Unique Customer Is Fun! of a Unique Customer Is Fun!

How Do You Represent:How Do You Represent:Address?Address?Status Level (e.g. Gold-Customer)?Status Level (e.g. Gold-Customer)?Special Considerations?Special Considerations?

Customers for General Electric:• Purchasers of Aircraft Engines• Purchasers of Refrigerators• Purchasers of Nuclear Power Plants• Purchasers of Light Bulbs• Purchasers of Locomotives

Slide 54

ReadingService

Reader-Makes-Right And Reader-Makes-Right And N-Squared CombinatoricsN-Squared Combinatorics

Reader-Makes-RightReader-Makes-RightReceiver Fixes MessageReceiver Fixes MessageBased on Both SchemasBased on Both SchemasBest Possible MappingBest Possible Mapping

N-Squared CombinatoricsN-Squared CombinatoricsEverybody Knows EverybodyEverybody Knows EverybodyN*(N-1) MappingsN*(N-1) MappingsGets Big Gets Big FastFast!!

Some datarequires

transformation

12 Services12 * 11 = 132

message transformers

Other datacopies through

Svc

Svc

Svc

Svc Svc Svc

Svc

Svc

Svc

SvcSvcSvc

Slide 55

Svc

Svc

Svc

Svc Svc Svc

Svc

Svc

Svc

SvcSvcSvc

Canonical

SourceService

DestinationServiceCanonical

SchemaTransforming

Service

Canonical Schema And Canonical Schema And Doubly-Lossy TransformationsDoubly-Lossy Transformations

Canonical SchemaCanonical Schema Is a Standard Representation Is a Standard RepresentationMap Everything to the Canonical and Back OutMap Everything to the Canonical and Back OutFar Fewer Transformers RequiredFar Fewer Transformers Required

Using a Canonical Schema is Using a Canonical Schema is Double-LossyDouble-LossyInfo Loss Going In and Info Loss Going OutInfo Loss Going In and Info Loss Going Out

Twotransformations

gives twicethe data loss

12 Services2 * N; 2*12=24

message transformers

Slide 56

The Typical Approach To The Typical Approach To Integrating SchemaIntegrating Schema

Classic Solution:Classic Solution:Implement Canonical SchemaImplement Canonical Schema

Practical to Build the TransformersPractical to Build the Transformers2 * N (for N Different Schemas)2 * N (for N Different Schemas)

Specialize When NeededSpecialize When NeededIf Doubly-Lossy Hurts, Then Reader-Makes-RightIf Doubly-Lossy Hurts, Then Reader-Makes-RightSpecialize Only Where ImportantSpecialize Only Where Important

Hybrid Trades Work for FidelityHybrid Trades Work for Fidelity

The Cool Thing About Services:• Rationalizing Schema Occurs for Outside Data• Inside a Service, You What You Want!• You Only Have to Cooperate on the Outside!

Slide 57












Slide 58

Storing Incoming DataStoring Incoming DataWhen Data Arrives from the Outside, You Store It InsideWhen Data Arrives from the Outside, You Store It Inside

Most Services Keep Incoming DataMost Services Keep Incoming DataKeep for ProcessingKeep for ProcessingKeep for AuditingKeep for Auditing

Incoming Data

Internal Data

Slide 59

Extensibility versus ShreddingExtensibility versus ShreddingShredding the MessageShredding the Message

The Incoming Data Is Broken Down to Relational FormThe Incoming Data Is Broken Down to Relational FormEmpowers Query and Business IntelligenceEmpowers Query and Business IntelligenceAuditing ConsiderationsAuditing Considerations

Typically, Don’t Want to Change the Message ImageTypically, Don’t Want to Change the Message ImagePreserve for AuditingPreserve for AuditingMay Keep Unshredded Version Also for Non-RepudiationMay Keep Unshredded Version Also for Non-Repudiation

ExtensibilityExtensibilityThe Sender Added Stuff You Didn’t ExpectThe Sender Added Stuff You Didn’t ExpectMay or May Not Know How Utilize ExtensionsMay or May Not Know How Utilize Extensions

Extensibility Fights Shredding!Extensibility Fights Shredding!Hard To Map Extensions To Planned Relational TablesHard To Map Extensions To Planned Relational TablesOK To Partially ShredOK To Partially Shred

Yields Partial Query BenefitsYields Partial Query Benefits

Slide 60

Atomically Processing MessagesAtomically Processing MessagesAtomic Transactions Consume MessagesAtomic Transactions Consume Messages

Each Message Is ConsumedEach Message Is ConsumedData May Be ChangedData May Be ChangedRecords Outgoing ResponsesRecords Outgoing Responses

All of a Service’s State lives in the Database!All of a Service’s State lives in the Database!It Is Transactionally ModifiedIt Is Transactionally Modified

TransactionService Logic

Service-PrivateData

Service

Slide 61

Resource-Oriented-Data and Resource-Oriented-Data and Activity-Oriented-DataActivity-Oriented-Data

Resource-Oriented-DataResource-Oriented-DataLives Longer Than One Lives Longer Than One Long-Running-OperationLong-Running-OperationChanged by Changed by Long-Running-OperationsLong-Running-Operations

Activity-Oriented-DataActivity-Oriented-DataGets Modified byGets Modified byLong-Running-OperationsLong-Running-OperationsRetires On Completion ofRetires On Completion ofLong-Running-OperationLong-Running-Operation

InventoryInventoryof SKU#of SKU#7194671946

Inventoryof SKU#71946

########

CustomerCustomerInfo forInfo for

Cust #8319Cust #8319““Sally’s Auto”Sally’s Auto”

CustomerInfo for

Cust #8319“Sally’s Auto”

#Info##Info#

PO#307654-03PO#307654-03Order forOrder for

Joe’s FlowersJoe’s Flowers

PO#307654-03Order for

Joe’s Flowers

#Master##Master#

#LineItems##LineItems#

ShoppingShoppingBasketBasket

#1834953#1834953

ShoppingBasket

#1834953

########

Bank BalanceBank Balanceof Account#of Account#

01600-1865301600-18653

Bank Balanceof Account#

01600-18653

#$$$##$$$#

Account-RecvAccount-RecvInv#173-45 forInv#173-45 forPO#307654-03PO#307654-03

Account-RecvInv#173-45 forPO#307654-03

#Info##Info#

#Payments##Payments#

Shipping OrderShipping Order#86403 tied to#86403 tied toPO#307654-03PO#307654-03

Shipping Order#86403 tied toPO#307654-03

#Info##Info#

#Shipped##Shipped#

These classes of data have different characteristicsWe’re gonna examine some of the

consequences of these differences…

Slide 62

Containment of Activity-Oriented and Containment of Activity-Oriented and Resource-Oriented DataResource-Oriented Data

Activity-Oriented and Resource-Oriented Data Activity-Oriented and Resource-Oriented Data Live Inside ServicesLive Inside Services

They Are Encapsulated within the ServiceThey Are Encapsulated within the ServiceAccess Is Mediated by Business LogicAccess Is Mediated by Business Logic

InventoryInventoryof SKU#of SKU#7194671946

ResourceOriented

Data

########

ShoppingShoppingBasketBasket

#1834953#1834953

ActivityOriented

Data

########

Internal Data

Slide 63

Retirement of DataRetirement of DataData Sometimes Data Sometimes RetiresRetires

It Becomes Read-OnlyIt Becomes Read-OnlyIt Is Referenced Less and Less FrequentlyIt Is Referenced Less and Less Frequently

It May Get Archived to TapeIt May Get Archived to TapeRarely, It Is DeletedRarely, It Is Deleted

New Regulations Will Make This Very RareNew Regulations Will Make This Very Rare

Activity-Oriented-Data Retires When the Activity-Oriented-Data Retires When the Long-Running Operation CompletesLong-Running Operation Completes

This May Take Weeks or MonthsThis May Take Weeks or Months

Resource-Oriented-Data Sometimes RetiresResource-Oriented-Data Sometimes RetiresThe Resource-Oriented-Data May Live a Long TimeThe Resource-Oriented-Data May Live a Long Time

SKUs, Accounts, Employees, Customers, etcSKUs, Accounts, Employees, Customers, etcThese May Live for Years These May Live for Years Sometimes They LeaveSometimes They Leave

Some Resources Have Bounded LifeSome Resources Have Bounded LifeHotel Occupancy Info; Airplane Seat AssignmentsHotel Occupancy Info; Airplane Seat Assignments

Slide 64

High and Low Concurrency UpdatesHigh and Low Concurrency Updates

Updates of Only Activity-Oriented DataUpdates of Only Activity-Oriented DataThis is Partitioned by Long-Running WorkThis is Partitioned by Long-Running Work

Almost Never Have Concurrent Work!!Almost Never Have Concurrent Work!!You Can Have It but It’s Not a Performance Issue…You Can Have It but It’s Not a Performance Issue…

Activity-Oriented Data Have Low Concurrency Updates!Activity-Oriented Data Have Low Concurrency Updates!

Updates Affecting Resource-Oriented DataUpdates Affecting Resource-Oriented DataThe Resource-Oriented Data MAY Be in High DemandThe Resource-Oriented Data MAY Be in High Demand

Possible that It Is So Desired that There Is a Queue Possible that It Is So Desired that There Is a Queue for Updatefor Update

Resource-Oriented Data MAY Have High Concurrency Resource-Oriented Data MAY Have High Concurrency UpdatesUpdates

Slide 65












Slide 66

XML, SQL, and ObjectsXML, SQL, and Objects

XMLXMLSchematized Representation of MessagesSchematized Representation of MessagesSchema Supports Schema Supports Independent DefinitionIndependent Definition and Extensibility and Extensibility

SQLSQLStores Relational Data by ValueStores Relational Data by ValueAllows You to “Allows You to “RelateRelate” Fields by ” Fields by ValuesValuesIncredibly Query CapabilitiesIncredibly Query CapabilitiesWorld-Class Management of DataWorld-Class Management of Data

ObjectsObjectsVery Powerful Software Engineering ToolVery Powerful Software Engineering ToolBased on Based on EncapsulationEncapsulation

DataDataSQLSQL

Slide 67

Bounded And Unbounded Bounded And Unbounded Data RepresentationsData Representations

Relational Is Relational Is BoundedBoundedOperations Within the DatabaseOperations Within the Database

Value ComparisonsValue Comparisons Only Meaningful Inside Only Meaningful Inside

Tightly Managed SchemaTightly Managed Schema

XML-Infoset Is XML-Infoset Is UnboundedUnboundedOpen (Extensible) Schema Open (Extensible) Schema

Contributions to Schema from Contributions to Schema from Who-Knows-WhereWho-Knows-Where

ReferencesReferences (Not Just Values) (Not Just Values)

URIs Known to Be UniqueURIs Known to Be Unique

XML-Infosets Can Be Interpreted AnywhereXML-Infosets Can Be Interpreted Anywhere

Slide 68

Encapsulation and Anti-EncapsulationEncapsulation and Anti-Encapsulation

SQL Is Anti-EncapsulatedSQL Is Anti-EncapsulatedUPDATE WHEREUPDATE WHERE

Query/Update by Joining Anything with AnythingQuery/Update by Joining Anything with AnythingTriggers/Stored-Procs Are Not Strongly Tied to Triggers/Stored-Procs Are Not Strongly Tied to Protected DataProtected Data

XML Is Anti-EncapsulatedXML Is Anti-EncapsulatedPlease Examine My Public Schema!Please Examine My Public Schema!

Components/Objects Offer EncapsulationComponents/Objects Offer EncapsulationLong Tradition of CheatingLong Tradition of Cheating

Reference Passing to Shared ObjectsReference Passing to Shared ObjectsWhacking on Shared DatabaseWhacking on Shared Database

Slide 69

The ServiceIs a

Black Box!

A Service’s View of EncapsulationA Service’s View of Encapsulation

Anti-Encapsulation Is OK in Its PlaceAnti-Encapsulation Is OK in Its PlaceSQL’s Anti-Encapsulation Is Only Seen by the Local Biz-LogicSQL’s Anti-Encapsulation Is Only Seen by the Local Biz-LogicXML’s Anti-Encapsulation Only Applies to the “Public” Behavior XML’s Anti-Encapsulation Only Applies to the “Public” Behavior and Data of the Serviceand Data of the Service

Encapsulation Is Strongly Enforced by the ServiceEncapsulation Is Strongly Enforced by the ServiceNo Visibility Is Allowed to the Internals of the Service!No Visibility Is Allowed to the Internals of the Service!

Business Request

Exported Data

Sanitized Datafor Export

PrivateInternal

Data

Data

Slide 70

Persistent ObjectID=Y

Persistent Objects and SQL StoragePersistent Objects and SQL Storage

Persistent ObjectsPersistent ObjectsEncapsulated by LogicEncapsulated by LogicKept in SQLKept in SQLUses Optimistic Concurrency (Low Update)Uses Optimistic Concurrency (Low Update)

Stored as Collection of RecordsStored as Collection of RecordsMay Use Records in Many TablesMay Use Records in Many TablesKeys of Records Prefixed with Unique IDKeys of Records Prefixed with Unique ID

This is the Object IDThis is the Object ID

Encapsulation by ConventionEncapsulation by Convention

SQL

<record>ID-X <key1>Table-B

<record>ID-X <key2><record>ID-X <key3>

<record>ID-Y <key1><record>ID-Y <key2>

Database-Key<record>ID-X <key>

Table-A

<record>ID-Y <key><record>ID-Z <key>

Database-KeyDatabase-Key

Persistent ObjectsWork Great for

Activity-Oriented-Data

Slide 71

Interesting!

Maybe copied using tools like XML SQL stuff; shredding what you wantto do BI Analysis on. Maybe transformed in and out using business logic.

Kinds of DataKinds of Data

Probably ProbablyFodder for BusinessIntelligence Analysis Definitely Definitely

Encapsulation Useful No No Yes Yes

Immutable orLow Concurrent Update orHighly Concurrent Update

Immutable Immutable(versions)

LowConcurrent

Update

HighlyConcurrent

Update

Requires Open Schemafor Interoperability Yes Yes No No

Durable storage in SQL? XML copyin SQL

XML copyin SQL Definitely Definitely

Represent in XML? Yes YesProbably

NotProbably

Not

Encapsulated Access? No NoYes: Use

Obj PersistYes:

Stateless

RequestResponse

ReferenceData

ActivityOriented

ResourceOriented

Slide 72

The Ruling TriumvirateThe Ruling Triumvirate

Impossible:Can’t see the data!

Problematic:Schema inconsistency

Outstanding

ArbitraryQueries

ImpossibleCan’t see the data!

Outstanding

Impossible:Centralized Schema

Independent Data Definition

Outstanding

Impossible:Open Schema

Not via SQLEnforced by DBA

Encapsulation(Controls Data)

SQL It is fantastic to compare anything to anything and combine anything with anything in Relational (within the bounded database)

XML It is possible to have independent definition of schema and data in XML-Infosets. You can independently extend, too.

Components/Objects

Provide encapsulation of data behind logic. Ensure enforcement ofbusiness rules. Eases composition of logic.

ObjectsEncapsulated Data

XMLUnbounded Schema

SQLBounded Schema

Strengths andWeaknesses

Each model’s strength is simultaneously its weakness!You can’t enhance one to add features of the other without breaking it!

Slide 73












Slide 74

Talking To The WorldTalking To The World

Use XML-Infosets Across ServicesUse XML-Infosets Across ServicesStandard or Proprietary Marshaling FormatsStandard or Proprietary Marshaling FormatsUse XML-SchemaUse XML-Schema

Interactions Across Services Include:Interactions Across Services Include:Requests,Requests,Responses, andResponses, andReference-DataReference-Data

Service-A

Slide 75

Storing The DataStoring The Data

Enterprise-Class ServicesEnterprise-Class ServicesEncapsulate a Collection of Private DataEncapsulate a Collection of Private Data

Store Your Data in a SQL Database!Store Your Data in a SQL Database!Industrial Strength Store, Performance, Industrial Strength Store, Performance, and Managementand Management

Store Incoming and Store Incoming and Outgoing Messages in SQLOutgoing Messages in SQL

Remember Incoming and Remember Incoming and Outgoing for AuditingOutgoing for AuditingTransactionally Record Transactionally Record Outgoing Messages for RetryOutgoing Messages for Retry DataData

SQLSQL

Slide 76

Implementing The CodeImplementing The Code

Build Your Services with ObjectsBuild Your Services with ObjectsOffer Encapsulation of Parts Within the ServiceOffer Encapsulation of Parts Within the ServiceThey Are the Best for Software Engineering!They Are the Best for Software Engineering!

EncapsulationEncapsulationServices Are a Big ChunkServices Are a Big Chunk

Completely Encapsulate Some Durable DataCompletely Encapsulate Some Durable DataObjects Make Littler ChunksObjects Make Littler Chunks

Inside the ServiceInside the ServiceMake It Easier to BuildMake It Easier to Build

Slide 77

Putting It All Together!Putting It All Together!

Services Need All Three!Services Need All Three!XML-Infosets: Between the ServicesXML-Infosets: Between the ServicesObjects: Objects: Implementing the Business Logic Implementing the Business LogicSQL: SQL: Storing Private Data and Messages Storing Private Data and Messages

DataDataSQLSQL

Objects ImplementObjects Implementthe Biz Logicthe Biz Logic

SQL Holds SQL Holds the Datathe Data

XML-InfoSets forXML-InfoSets forMessages Between ServicesMessages Between Services

Slide 78












Slide 79

Data Inside and Outside ServicesData Inside and Outside Services

Data Is Different Inside from OutsideData Is Different Inside from Outside

Outside the ServiceOutside the ServicePassed in MessagesPassed in MessagesUnderstood by Sender and ReceiverUnderstood by Sender and ReceiverIndependent Schema Definition ImportantIndependent Schema Definition ImportantExtensibility ImportantExtensibility Important

Inside the ServiceInside the ServicePrivate to ServicePrivate to ServiceEncapsulated by Encapsulated by Service CodeService Code

DataDataSQLSQL

MSG

MSG

Data Outside Data Outside the Servicethe Service

Data Inside Data Inside the Servicethe Service

Slide 80

ResourcesResources

[email protected]@microsoft.comhttp://www.microsoft.com/patternshttp://www.microsoft.com/patterns

www.PatHelland.comwww.PatHelland.comwww.PatHelland.comwww.PatHelland.com

http://blogs.msdn.com/http://blogs.msdn.com/PatHellandPatHelland

http://blogs.msdn.com/http://blogs.msdn.com/PatHellandPatHelland

http://msdn.microsoft.com/http://msdn.microsoft.com/architecturearchitecture

http://msdn.microsoft.com/http://msdn.microsoft.com/architecturearchitecture

Documents

Data on the Inside versus Data on the Outside Session Delivered By: Lars Laakes Architect Specialist, Financial Services Microsoft Canada [email protected]