Slide 1

Data on the OutsideversusData on the InsidePat HellandPartner ArchitectMicrosoft Corporation11/25/2007 4:22 PM 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

1Session Objectives And TakeawaysUnderstand the use and interpretation of dataIt is different inside services than outside services

Many new issues arise in the world between servicesVersioning, meta-data (schema), immutability, idempotence, and much moreThink carefully about the interactions across services

XML, SQL, and Objects all have their placeUnderstand their strengths and weaknesses and use them together to create the best servicesSlide 211/25/2007 4:22 PM 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.2Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 3Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 4ServicePolicySchema and ContractService-Oriented ArchitectureService-orientationIndependent servicesChunks of code and dataInterconnected via messaging

Four basic tenets:Boundaries are explicitServices are autonomousServices share schema and contractNot implementationService compatibility is based on policySlide 5Services Communicate With MessagesServices communicate with messagesNothing elseNo other knowledge about partnerMay be heterogeneousService-AService-BSlide 66The key thing about a service oriented architecture is that services only communicate with messagesThere is no other interaction besides messagesYou cant tell anything about the other machine except what messages it sendsIt does not have to be the same type of system

Data Inside and Outside ServicesData is different inside from outside

Outside the servicePassed in messagesUnderstood by sender and receiverIndependent schema definition importantExtensibility importantInside the servicePrivate to serviceEncapsulated by service codeDataSQLMSGMSGData Outside the ServiceData Inside the ServiceSlide 77Data lives either:Outside servicesPassed in messagesMust be understood by both:The service creating the inter-service-data, andThe service reading and using the inter-service-dataInside a single servicePrivate to the service and only accessed via encapsulating codeMay be a replica of some data that lives outside servicesThe characteristics of the inside data differ from the outside dataInside, you need tightly organized schema associated with you application codeYou may want to apply business intelligence to itYou may need to adjust your schema for semantic changes or for optimization of your applications database accessOutside, you need to optimize for flexibility, extensibility, and mutual understanding across systemsThis should tolerate changes across time and the evolution of the participating services

Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 8Bounding Trust via EncapsulationServices only do limited things for their partnersThis is how they bound their trustEncapsulation is about bounding trustBusiness logic ensures only the desired operations happenNo changes to the data occur except through locally controlled business logic!ServiceThings Ill Do for Outsiders Deposit Withdrawal Transfer Account Balance CheckSlide 9Encapsulating Both Change and ReadsEncapsulating changeEnsures integrity of the services workEnsures integrity of the services dataEncapsulating exported data for readEnsures privacy by controlling whats exportedAllows planning for loose coupling and expirationsE.G. Wednesdays price-listBusiness RequestExported DataSanitized Datafor ExportPrivateInternalDataDataTrust and TransactionsSome propose atomic transactions across servicesE.G. WS-TransactionsRequires holding locksLots of trust in timely unlockDoesnt sound autonomous and independent to meDebate is the definition of the word serviceRequires autonomy and independence?Allows intimacy across service boundaries?There will be code connected by 2-phase commitSame service or in different services?For this talk, I presume no cross-service txsSimply the definition of the word serviceSlide 11Interconnecting with Independent ServicesServices are connected by messagingThe only interaction between two services is by the messages that they exchangeSchema: the formats of the individual messagesContracts :the allowable sequences of messagesServiceContractReject-OrderAccept-OrderOneOfTentativePlace-OrderConfirmPlace-OrderOneOfCancelPlace-OrderSlide 12Operators and OperandsMessages contain operatorsRequests a business operationOperators provide business semanticsPart of the contract between the two servicesOperator messages contain operandsDetails needed to do the business operationThe sending service must put them into the messageServiceDepositOperatorOperandsSlide 13Where Do Operands Come From?Operands come from reference dataNew kind of data in SOAExcept its not new; weve done variations of SOA for decades Were just getting better at it!Reference data is versioned and each version is immutableImmutable images are shared across many servicesWe will talk about the creation, publication, and management of reference dataSlide 14Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 15Transactions and Inside DataTransactions make you feel aloneNo one else manipulates the data when you areTransactional serializabilityThe behavior is as if a serial order exists

Slide 16Life in the NowTransactions live in the now inside servicesTime marches forwardTransactions commit Advancing timeTransactions see the committed transactionsA services biz-logic lives in the now

Slide 17Sending Unlocked Data Isnt NowMessages contain unlocked dataAssume no shared transactionsUnlocked data may changeUnlocking it allows changeMessages are not from the nowThey are from the pastThere is no simultaneity at a distance! Similar to speed of light Knowledge travels at speed of light By the time you see a distant object it may have changed! By the time you see a message, the data may have changed!Services, transactions, and locks bound simultaneity! Inside a transaction, things appear simultaneous (to others) Simultaneity only inside a transaction! Simultaneity only inside a service!Slide 18Outside Data: a Blast from the PastAll data seen from a distant service is from the pastBy the time you see it, it has been unlocked and may changeEach service has its own perspectiveInside data is now; outside data is pastMy inside is not your inside; my outside is not your outsideAll data from distant stars is from the past 10 light years away; 10 year old knowledge The sun may have blown up 5 minutes ago We wont know for 3 minutes moreGoing to SOA is like going from Newtonian to Einstonian physics Newtons time marched forward uniformly Instant knowledge Before SOA, distributed computing many systems look like one RPC, 2-phase commit, remote method calls In Einsteins world, everything is relative to ones perspective SOA has now inside and the past arriving in messagesSlide 19Versioned Images of a Single SourceA sequence of versions describing changes to dataUpdates fromone serviceOwner controlledOwner changes the dataSends changes as messagesData is seenas advancingversions

Slide 20Operators: Hope for the FutureMessages may contain operatorsRequests for business functionality part of the contractService-B sends an operator to Service-AIf Service-A accepts the operator, it is part of its futureIt changes the state ofService-AService-B is hopefulIt wants Service-A to dothe workWhen it receives a reply,its future is changed!

Slide 21Operands: Past and FutureOperands may live in the pastValues published as reference dataCome from Service-As past

Operands may live in the futureThey may contain a proposed value submitted to Service-A

Slide 22Between Services: Life in the ThenEverything between services lives in the past or futureOperators live in the futureOperands live in the past or the futureIts not meaningful to speak of now between servicesNo shared transactions no simultaneityLife in the thenPast or futureNot nowEach service hasa separate nowDifferent temporalenvironments!

Slide 23Services: Dealing with Now and ThenServices Make the Now Meet the ThenEach Service Lives in Its Own NowMessages Come and Go Dealing with the ThenThe Business-Logic of the Service Must Reconcile This!!The world is no longer flat! SOA is recognizing that there is more than one computer Multiple machines mean multiple time domains Multiple time domains mandate we cope with ambiguity to allow coexistence, cooperation, and joint workExample: accepting an order A biz publishes daily prices Probably want to accept yesterdays prices for a while Tolerance for time differences must be programmedExample: Usually ships in 24 hours Order processing has old info Available inventory not accurate Deliberately fuzzy Allows both sides to cope with difference in time domains!Slide 24Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 25What Is Reference Data?Reference data is published across service boundariesFor each collection of reference data:One service creates and publishes the dataOther services receive periodic versions of the dataPurposes for Reference DataHistoric ArtifactsShared Collections of DataOperandsSlide 26Reference Data: Operands for the OperatorsAs discussed above, messages across services invoke business operationsEach service-to-service message is an operatorEach operator message is filled with operandsParameters, options, customer-id, parts-being-ordered, etcThe data for these operands is published as reference dataServiceDepositOperatorOperandsSlide 27Reference Data: Historic ArtifactsHistoric artifacts report on what happened in the pastSometimes these snapshots need to be sent to other servicesExamples:Sales quarterly resultsMonthly bank statementsAny and all monthly billsWell Both requests for payment (operations) and the historic artifact of how much power you usedInventory status at end of quarter

Service(Bank)BankStatementJul-2007Slide 28Ref Vers#23of EmployeeDataAuthoritativeEmployeeData Vers#23Update!Ref Vers#24of EmployeeDataAuthoritativeCustomerDataAuthoritativeEmployeeData Vers#24Reference Data: Shared Collections of DataMany services may need access to the same dataThe data is changingSomeone owns updating and distributing the dataExamples:Customer databaseEmployee databaseParts database and price-listSales ServiceHR ServiceRef Vers#24of EmployeeDataUpdateEmployeesVers#24Slide 29Publishing Versioned Reference DataThe owner of data periodically publishesUsing whatever messaging technique it wantsPublications are always versionedThe version numbers increase4RequestUses: Vers-ZService-B3As-DataVers-Z2As-DataVers-YAs-DataVers-X1Service-ASlide 302RequestUses: Vers-XPlease MakeData ChangeBusiness Operations May Request ChangesIf a non-owner wants a change it must do a biz-operationThis is a request sent to the owning serviceThe owning service may agree to the operation causing changeto the data in questionIf it changes, this affects the next version`Service-B3As-DataVers-YAs-DataVers-X1OwningService-ASlide 31Optimistic Concurrency Control:Anti-EncapsulationWhat is optimistic concurrency control?Data is readChanges are made and submitted to the datas ownerIf the original data hasnt changed, the new changes are appliedThis assumes the remote system should be able to write directly on the dataThis is a trusting relationship not autonomous!Autonomy and updates to dataAutonomy means independent controlMy local biz-logic decides how my data changes!If you want a change, ask me to do a business opIts my dataIll decide how it changes!Slide 32Example: Updating the Customers AddressWhat about a salesperson updating a customers address?Shouldnt that just be optimistic concurrency control?No! It should invoke business logic with a request!Not all fields of the customer record should be updated by sales peopleRequests across service boundaries invoke business logic when the customer address is changedSlide 33Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 34Immutable And/Or Versioned DataData may be immutableOnce written, it is unchangeableImmutable data needs an IDFrom the ID, comes the same dataNo matter when, no matter whereVersions are immutableEach new version is identifiedGiven the identifier, the same data comesVersion independent identifiersLet you ask for a recent versionWindows Vista, SP1 The Same Set of Bits Every TimeNew York Times; 7/24/07 Specific Version of the Paper -- Contents Dont ChangeLatest SP of Vista Definitely Vista, Results Vary Over TimeRecent NY Times Maybe Todays, Maybe YesterdaysVersionIndependentSlide 3535Immutability of MessagesRetries are a fact of lifeZero or more delivery semanticsMessages must be immutableRetries must not see differencesOnce its sent, you cant un-send!Service-AOnce Its Outside,Its Immutable!Slide 3636To Cache Or Not To CacheOK to cache immutable dataIts never wrongNever have to invalidate!Caching should only be used for immutable dataCaching data that changes may lead to anomaliesConsider caching data labeled with a version dependent IDBecause versions are immutable it will workStore the mapping from version independent to version dependent in an accurate locationSlide 3737Caching immutable data is wonderful!It is never invalid!You never have to shoot down the cacheIt is important to consider the stability of the cached dataWe really dont want the interpretation and meaning of the data to changeMessages are easily cachedThey should be immutable and stableWe recommend well defined versioning schemes for updating reference dataThe version is immutableIt may be cached without difficultySome kinds of data are not immutableThese are inappropriate for cachingThe cache may be out of sync with the truth!

Normalization And Immutable DataDatabases design for normalized dataCan be changed without funny behaviorEach data item lives in one placeSometimes data should be de-normalizedIf data is immutable its OKEmp #Emp NameMgr #Mgr NameEmp PhoneMgr Phone47Joe13Sam5-12346-987618Sally38Harry3-31235-678291Pete13Sam2-11126-987666Mary02Betty5-73494-0101Classic problemwith de-normalizationCant updateSams phone #since there aremany copiesDe-normalization isOK if you arent going to update!Slide 3838In databases, you design for normalized dataNormalized Data is easily changed without causing funny behaviorEach data item lives in exactly one placeEssential design consideration for database schemasSometime data should be de-normalizedIf the data will not be updated, its OK to have it denormalizedIt is sometimes easier to have denormalized dataDont have to do joins to get the needed information

Stability Of DataImmutability isnt enough!We need a common understandingPresident Bush 1990 vs. President Bush 2007Stable data has a clearly understood meaningThe schema must be clearly understoodThe interpretation of values must be unambiguousAdvice Dont recycle customer-idsObservation Anything called current is not stableSuggestion Timestamping or versioning makes stable dataObservation A monthly bank statement is stable dataSlide 3939Just because data is immutable doesnt mean it is always understood the same way!A reference to President Bush in 2003 means something different than a reference to President Bush made in 1990Stable data has a meaning that is clearly understood First, it must have an immutable schema definitionStable meta-data means that you can clearly understand the schemaDistributed data must be both immutable (perhaps versioned) and stableYou must ensure that the bits are as intended whenever and wherever they may be retrievedYou must ensure that the interpretation of the bits is what is intended for the time-scope and geographic scope of the datas life

A Few Thoughts on Stable DataOutside data must be stableConsistent interpretation across valid spaces and timesInside data may be stableNotably, when it is the same data as outside dataSometimes data inside is not stableClassic normalization for vibrant updateNeeds to be cast into a stable shape to send outsideSlide 40Validity Of Data In Bounded Space And TimeBounding the valid timesIt may have an expirationBounding the valid locationsRestrictions on where the data is validWhen valid, the data should be:Immutable (the ID yields the same bits)Stable (the meaning is clear)

Offer Good to Washington State Residents OnlyOffer Good Until Next TuesdayData Valid For Service-X OnlyPrice-List Valid Until Dec 31stSlide 4141It is just fine to put expirations on data validityOffer good until next TuesdayIt is fine to bound the locality of the datas validityOffer good to Washington State residents onlyData valid only on these two serversInformation valid only for the Acme & Sons Company Accounting appWithin the range of its validity, data must be:Immutable (the id yields the same bits), andStable (the meaning of the bits is clear)Some data has bounds on its validity, some does not

Rules For Sending Data In MessagesIdentify theMessagePut Unique ID in All MessagesPart of the Unique ID May Be a VersionImmutableDataDont Change the Data Associated withthe Unique ID; Never Return Different BitsOK toCacheThe Same Bits Will Always Be ReturnedDefine ValidRangesValid for a Certain Time Period and OverSome Space; OK to Always Be ValidMust BeStableMust Ensure There Is Never Any ConfusionAbout the Meaning (Within Valid Range)Slide 4242Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 43SQL, DDL, and SerializabilitySQLs DDL (data definition language) is transactionalChanges are made using transactionsThe structure of the data may be changedThe interpretation after the DDL change is different

DDL lives within the time scope of the databaseThe databases shape evolves over timeDDL is the change agent for this evolution

SQL lives in the nowEach transactions execution is meaningful only within the schema definition at the moment of its executionSerializability makes this crisp and well-definedSlide 44Schema and Immutable MessagesWhen a message is sent, it must be immutableIt is crossing temporal boundariesRetries mustnt give different resultsThe messages schema must be immutableIt makes a mess if the interpretation of the message changesService-A

MessageMessageSchemaImmutable MessageImmutable Schema for the MessageSlide 45Immutable Schema and Its IdentifiersImmutable schema needs an identifierIt must be possible to unambiguously identify the schemaThis must occur across the namespaces of sender and receiverThe schema definition must never changeGiven the identifier, the same schema is returned

URIs (Universal Resource Identifiers) work wellGuaranteed to be uniqueIf you follow the rules

URLs (Universal Resource Locators) are coolURLs are URIsAlso tell you a location to get the stuff (e.g. the schema)Slide 46Composition of Schema as a DAGSchema make contain sub-schemaInside the message are chunks of dataA purchase-order may contain customer informationThey have their own definitionsThe sub-schema are referenced by identifierThis leads to a tree of references to immutable schemaIts really a DAG (Directed Acyclic Graph)Sometimes, different sub-schema reference the same stuffPurchase Order

CustomerDelivery AddrSKUsCustomer

NameAddressCredit RatingAddress

Number/StreetCity/StatePostal CodeCountrySKU

Part Color SizeSlide 47Versioning and SchemaFrequently, schema is versionedA new format of the schema is createdIt is given a new identifierVersion independent schema identifiersSpecify a set of versions for a type of schemaThe set may evolve over timeVersion dependent schema identifiersSpecify a specific version of a specific schemaThe version-dependent schema is immutableMessages should always specify a version-dependent schemaThis ensures no ambiguitySlide 48Extensibility and SchemaExtensibility is the addition of non-schema specified information into the messageThe schema does not specify the additional stuffThe sender wanted to add it anywayAdding extensions is like scribbling in the marginsSometimes adding notes to a form helps!Sometimes it does no good at all!Service-A

Purchase Order CustomerDelivery AddrSKUsDont Deliver in MorningPurchase Order

CustomerDelivery AddrSKUsSlide 49Infosets, XML-Schema, And PSVIXML-InfosetSemantics of XML, not syntaxTree: parents, children, elements, & attributesAllows (encourages) schemaAny representation OKXML-SchemaDatatype library and schema definitionComposed schema uniquely identified (URI)PSVI Post Schema Validated InfosetInfoset after validation against schemaCan leverage schema knowledgeSlide 5050XML-InfosetThe semantics (data model) of XMLNot the marshalling format no impliedAn Infoset is a tree with well defined parent-child relationships using elements and attributesAllows schema (XML-Schema) to be attached at any elementAn Infoset may be represented in any format (binary is just fine!)XML-SchemaDefines a datatype library and schema definition languageAny composed schema references are unambiguously tagged with a URIURI Universal Resource Identifier A guaranteed unique IDEssential for independent, composable, and extensible definitionsPSVI Post Schema Validated InfosetAn XML structure after being validated against its schemaKnown to be compliant with its schema, this PSVI is way more powerful and useful than a non-validated structure

Messages Need Immutable Contents and Immutable SchemaMessages must be immutableRetries must not see differencesMessage schema must be immutableMust be able to unambiguously interpret the messageXML-Infosets and XML-Schema work well for messaging!Service-A

Immutable MessagesImmutable SchemaSchema for the outsideis very differentfrom schema forthe inside!!!Slide 5151When messages live between services, they need to be immutableRetries are a fact of lifeIf we see it again and its different, what is the behavior that we should do based on the difference?When an Infoset lives between services, we should treat it as immutableOnce its been sent from home, you cant recall it and change your mind!

Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 52Coping with Different SchemaDifferent services frequently have different schemaHow do you represent a customer?What information about a customer is relevant?What special info does your business-unit keep about customers?Is there an enterprise-wide customer-id?Just disambiguating the identity of a unique customer is fun!How do you represent:Address?Status level (e.g. Gold-customer)?Special considerations?Customers for General Electric: Purchasers of aircraft engines Purchasers of refrigerators Purchasers of nuclear power plants Purchasers of light bulbs Purchasers of locomotivesSlide 53ReadingServiceReader-Makes-Right And N-Squared CombinatoricsReader-makes-rightReceiver fixes messageBased on both schemasBest possible mappingN-squared combinatoricsEverybody knows everybodyN*(N-1) mappingsGets big fast!Some datarequirestransformation12 Services12 * 11 = 132message transformersOther datacopies throughSvcSvcSvcSvcSvcSvcSvcSvcSvcSvcSvcSvcSlide 5454Reader-Makes-RightWhen an incoming message arrives, the receiver fixes itThe source schema and destination schema are comparedStuff that easily maps is movedThe best fit to transform the mismatch is hand-craftedN-Squared CombinatoricsEvery reader must know about all of its partnersFor N partners, N*(N-1) hand-crafted transformations must existAs N gets large, this gets too large

SvcSvcSvcSvcSvcSvcSvcSvcSvcSvcSvcSvcCanonicalSourceServiceDestinationServiceCanonical SchemaTransforming ServiceCanonical Schema And Doubly-Lossy TransformationsCanonical schema is a standard representationMap everything to the canonical and back outFar fewer transformers requiredUsing a canonical schema is double-lossyInfo loss going in and info loss going outTwotransformationsgives twicethe data loss12 Services2 * N; 2*12=24message transformersSlide 5555A canonical schema is a standard representation for a piece of the enterprises dataYou shoe-horn every other version of that data into the enterprise standardFor example, you may have a canonical schema for the customer recordConverting to and from a canonical schema is double-lossyYou lose some information going in and lose some information going outReader-Makes-Right will frequently give greater fidelity information transfer

The Typical Approach To Integrating SchemaClassic Solution:Implement Canonical SchemaPractical to Build the Transformers2 * N (for N Different Schemas)Specialize When NeededIf Doubly-Lossy Hurts, Then Reader-Makes-RightSpecialize Only Where ImportantHybrid Trades Work for FidelityThe cool thing about services: Rationalizing schema occurs for outside data Inside a service, you have what you want! You only have to cooperate on the outside!Slide 5656The classic solution to this conundrum is to:Implement a canonical schema for most of the transformationsThis has only 2-N transformations for N different representationsWhen the double-lossy problem strikes, build a special caseThis special case will eliminate the double-lossy for that single source and single destinationMost systems use a hybrid!

Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 57Storing Incoming DataWhen data arrives from the outside, you store it insideMost services keep incoming dataKeep for processingKeep for auditingIncoming DataInternal DataSlide 58Extensibility versus ShreddingShredding the messageThe incoming data is broken down to relational formEmpowers query and business intelligenceAuditing considerationsTypically, dont want to change the message imagePreserve for auditingMay keep unshredded version also for non-repudiation

ExtensibilityThe sender added stuff you didnt expectMay or may not know how utilize extensions

Extensibility fights shredding!Hard to map extensions to planned relational tablesOK to partially shredYields partial query benefitsSlide 59Atomically Processing MessagesAtomic transactions consume messagesEach message is consumedData may be changedRecords outgoing responsesAll of a services state lives in the database!It is transactionally modifiedTransactionService LogicService-PrivateDataServiceSlide 6060Resource-Oriented-Data and Activity-Oriented-DataResource-Oriented-DataLives Longer Than One Long-Running-OperationChanged by Long-Running-Operations

Activity-Oriented-DataGets Modified byLong-Running-OperationsRetires On Completion ofLong-Running-OperationInventoryof SKU#71946Inventoryof SKU#71946

####CustomerInfo forCust #8319Sallys AutoCustomerInfo forCust #8319Sallys Auto

#Info#PO#307654-03Order forJoes FlowersPO#307654-03Order forJoes Flowers

#Master##LineItems#ShoppingBasket#1834953ShoppingBasket#1834953

####Bank Balanceof Account#01600-18653Bank Balanceof Account#01600-18653

#$$$#Account-RecvInv#173-45 forPO#307654-03Account-RecvInv#173-45 forPO#307654-03

#Info##Payments#Shipping Order#86403 tied toPO#307654-03Shipping Order#86403 tied toPO#307654-03

#Info##Shipped#These classes of data have different characteristicsWere gonna examine some of the consequences of these differencesSlide 6161Resource-Oriented DataManages stuff that lives longer than a single long-running operationChanged by long-running-work but lives longer than an individual long-running operationActivity-Oriented DataGets modified only during a single long-running operationGets retired after the completion of its long-running operation

Containment of Activity-Oriented and Resource-Oriented DataActivity-oriented and resource-oriented data live inside servicesThey are encapsulated within the serviceAccess is mediated by business logicInventoryof SKU#71946ResourceOrientedData

####ShoppingBasket#1834953ActivityOrientedData

####Internal DataSlide 62Retirement of DataData sometimes retiresIt becomes read-onlyIt is referenced less and less frequentlyIt may get archived to tapeRarely, it is deletedNew regulations will make this very rareActivity-oriented-data retires when the long-running operation completesThis may take weeks or monthsResource-oriented-data sometimes retiresThe resource-oriented-data may live a long timeSkus, accounts, employees, customers, etcThese may live for years Sometimes they leaveSome resources have bounded lifeHotel occupancy info; airplane seat assignmentsSlide 63High and Low Concurrency UpdatesUpdates of only activity-oriented dataThis is partitioned by long-running workAlmost never have concurrent work!!You can have it but its not a performance issueActivity-oriented data have low concurrency updates!

Updates affecting resource-oriented dataThe resource-oriented data MAY be in high demandPossible that it is so desired that there is a queue for updateResource-oriented data MAY have high concurrency updatesSlide 64Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 65XML, SQL, and ObjectsXmlSchematized representation of messagesSchema supports independent definition and extensibility

SQLStores relational data by valueAllows you to relate fields by valuesIncredibly query capabilitiesWorld-class management of data

ObjectsVery powerful software engineering toolBased on encapsulationDataSQL

Slide 66Bounded And Unbounded Data RepresentationsRelational is boundedOperations within the databaseValue comparisons only meaningful insideTightly managed schema

XML-Infoset is unboundedOpen (extensible) schema Contributions to schema from who-knows-whereReferences (not just values)Uris known to be uniqueXML-Infosets can be interpreted anywhereSlide 6767Relational is a bounded representation of dataContained within a DatabaseDefinitions in the database are only meaningful in the fullness of the databaseCan do complex joins across anything within the bounded databaseThis is easier when the universe is containedNot meaningful to reference data outside the databaseThe contents of the database itself are not cleanly understandable outside of the local system (without additional description)XML-Infosets are an unbounded representationEverything is tagged with a namespaceThe schema is globally identifiedReferences are via URIsThey are meaningful anywhereXML-Infosets can be interpreted anywhereThe semantic is designed to be understood everywhere

Encapsulation and Anti-EncapsulationSQL is anti-encapsulatedUPDATE WHEREQuery/update by joining anything with anythingTriggers/stored-procs are not strongly tied to protected data

XML is anti-encapsulatedPlease examine my public schema!

Components/objects offer encapsulationLong tradition of cheatingReference passing to shared objectsWhacking on shared databaseSlide 68The ServiceIs aBlack Box!A Services View of EncapsulationAnti-encapsulation is OK in its placeSQLs anti-encapsulation is only seen by the local biz-logicXMLs anti-encapsulation only applies to the public behavior and data of the serviceEncapsulation is strongly enforced by the serviceNo visibility is allowed to the internals of the service!Business RequestExported DataSanitized Datafor ExportPrivateInternalDataDataSlide 69Persistent ObjectID=YPersistent Objects and SQL StoragePersistent ObjectsEncapsulated by LogicKept in SQLUses Optimistic Concurrency (Low Update)Stored as Collection of RecordsMay Use Records in Many TablesKeys of Records Prefixed with Unique IDThis is the Object IDEncapsulation by ConventionSQL

ID-X

Table-B

ID-X

ID-X

ID-Y

ID-Y

Database-Key

ID-X

Table-A

ID-Y

ID-Z

Database-KeyPersistent ObjectsWork Great forActivity-Oriented-DataSlide 7070Sometimes people implement persistent objectsThese are kept in the SQL databaseThey are pulled into memory on demand, worked on through a transaction, and stuffed back into SQL on completionTypically, optimistic concurrency control is usedRather than lock the data, the in-memory object remembers all the details of what was read from the databaseWhen the object changes are completed, the database is checked to ensure no one has fiddled with the contentsIf no changes have happened, the updates are made to the SQL copyPersistent objects are implemented as a collection of records in a collection of tablesThe structure of the persistent object in its database storage may be quite rich and complexThe persistent object is identified by a unique keyRequests to work with this object must supply the key valueThe key value is the equivalent of an object reference and may be stored in other objects as a persistent referenceAll the records for a single persistent object are indexed by this keyThe key is the prefix of the actual primary key used for the record in SQLEach object is a slice across a set of tables in SQLMany objects live in the set of tablesThe usage pattern for these objects ensures that the set of records is never updated except through the object codeThe collection of records across tables is an intact object

Interesting!Maybe copied using tools like XML SQL stuff; shredding what you wantto do BI Analysis on. Maybe transformed in and out using business logic.Kinds of DataProbablyProbablyFodder for BusinessIntelligence Analysis DefinitelyDefinitelyEncapsulation UsefulNoNoYesYesImmutable orLow Concurrent Update orHighly Concurrent UpdateImmutableImmutable(versions)LowConcurrentUpdateHighlyConcurrentUpdateRequires Open Schemafor InteroperabilityYesYesNoNoDurable storage in SQL?XML copyin SQLXML copyin SQLDefinitelyDefinitelyRepresent in XML? YesYesProbablyNotProbablyNotEncapsulated Access? NoNoYes: UseObj PersistYes:StatelessRequestResponseReferenceDataActivityOrientedResourceOrientedThe Ruling TriumvirateImpossible:Cant see the data!Problematic:Schema inconsistencyOutstandingArbitraryQueriesImpossibleCant see the data!OutstandingImpossible:Centralized SchemaIndependent Data DefinitionOutstandingImpossible:Open SchemaNot via SQLEnforced by DBAEncapsulation(Controls Data)SQLIt is fantastic to compare anything to anything and combine anything with anything in Relational (within the bounded database)XMLIt is possible to have independent definition of schema and data in XML-Infosets. You can independently extend, too.Components/ObjectsProvide encapsulation of data behind logic. Ensure enforcement ofbusiness rules. Eases composition of logic.ObjectsEncapsulated DataXMLUnbounded SchemaSQLBounded SchemaStrengths andWeaknessesEach models strength is simultaneously its weakness!You cant enhance one to add features of the other without breaking it!Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 73Talking To The WorldUse XML-Infosets across servicesStandard or proprietary marshaling formatsUse ML-SchemaInteractions across services include:Requests,Responses, andReference-dataService-A

Slide 7474We recommend that interactions across services use XML-InfosetsThe Infoset provides independent definition and extensibilityThese may be marshaled into ASCII with angle-brackets or into some other mutually agreed formatThese interactions are messages holding:Requests,Responses, andReference Data

Storing The DataEnterprise-class servicesEncapsulate a collection of private dataStore your data in a SQL database!Industrial strength store, performance, and managementStore incoming and outgoing messages in SQLRemember incoming and outgoing for auditingRemember them for Biz IntelligenceTransactionally record outgoing messages for retryDataSQLSlide 7575Recall the notion of an Enterprise-Class Service It implements one or more services that may be invoked from outsideAn Enterprise-Class Service encapsulates a collection of private dataPartitions over a set of tables in a single SQL engineWe strongly recommend that you store your business data in SQLIt is the industrial strength data storeWe strongly recommend that you store the incoming and outgoing messages in SQLSQL offers the reliability needed to manage any potential retriesMost enterprises need to archive the interactions of their application services for legal reasons

Implementing the CodeBuild your services with objectsOffer encapsulation of parts within the serviceThey are the best for software engineering!EncapsulationServices are a big chunkCompletely encapsulate some durable dataObjects make littler chunksInside the serviceMake it easier to buildSlide 7676We recommend you implement your service using object oriented coding techniquesObjects (or components) offer encapsulationThis has a well established record of improving software engineering

Putting It All Together!Services Need All Three!XML-Infosets: Between the servicesObjects: Implementing the business logicSQL: Storing private data and messagesDataSQL

Objects Implementthe Biz LogicSQL Holds the DataXML-InfoSets forMessages Between Services77Implementing a service requires:XML-Infosets for communication between servicesObjects/Components for implementing the business logic SQL for storing the mission-critical data encapsulated by the service (both Resource and Activity) and the messages (both incoming and outgoing)

Outline Introduction: The Shift Towards Services Behavior: Encapsulation and Trust Data: Then and Now Outside Data: Reference Data Outside Data: Sending Messages Outside Data: XML and Schema Outside Data: Commonality of Schema and Understanding Inside Data Inside/Outside: Representations of Data Inside/Outside: Tying It All Together ConclusionSlide 78Data Inside and Outside ServicesData is different inside from outside

Outside the servicePassed in messagesUnderstood by sender and receiverIndependent schema definition importantExtensibility importantInside the servicePrivate to serviceEncapsulated by service codeDataSQLMSGMSGData Outside the ServiceData Inside the ServiceSlide 7979Data lives either:Outside servicesPassed in messagesMust be understood by both:The service creating the inter-service-data, andThe service reading and using the inter-service-dataInside a single servicePrivate to the service and only accessed via encapsulating codeMay be a replica of some data that lives outside servicesThe characteristics of the inside data differ from the outside dataInside, you need tightly organized schema associated with you application codeYou may want to apply business intelligence to itYou may need to adjust your schema for semantic changes or for optimization of your applications database accessOutside, you need to optimize for flexibility, extensibility, and mutual understanding across systemsThis should tolerate changes across time and the evolution of the participating services

Session Objectives And TakeawaysUnderstand the use and interpretation of dataIt is different inside services than outside services

Many new issues arise in the world between servicesVersioning, meta-data (schema), immutability, idempotence, and much moreThink carefully about the interactions across services

XML, SQL, and Objects all have their placeUnderstand their strengths and weaknesses and use them together to create the best servicesSlide 8011/25/2007 4:22 PM 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.80