Data access layer and schema definitions

Data Access Layer and Schema Definitions Luciano Resende [email protected] [email protected] http://lresende.blogspot.com/

About

Luciano Resende •  Staff Software Engineer – Shutterfly, Inc. •  Member – Apache Software Foundation •  Committer: Apache Tuscany, Wink, Nuvem, PhotoArk

December 8, 2011 MongoSV 2011 2

MongoDB Characteristics

•  Scalability •  Simplicity •  Flexible Schema •  Low TCO


MongoDB initial POCs

December 8, 2011

MongoDB

Java Component

.NET Component

Java Component

first_name

firstname

firstName

Floor plan image from : h3p://benchmarkoffices.com/floorplans.htm

MongoSV 2011 4

The Platform

•  Applications will interact with MongoDB persistence via services •  Enforce a consistent schema

across multiple applications and cross-platform

•  Data Access Layer •  Data Access Layer that in

conjunction with UDDL abstracts the persistence layer allowing for more flexible and clean code

•  A cross platform data schema definition (text DSL) •  Simple human (& machine)

readable language to describe our domain model entities

December 8, 2011

Services

Data Access Layer

UDDL

MongoSV 2011 5

Schema Definition

Schema Definition – Why using Text DSL

•  Platform independent data definition. •  (Annotated) POJO (JAXB, JPA, Morphia) has too much noises and

is abuse-prone. We want to enforce the patterns. •  XSD is too complicated and it doesn’t fit all for web 2.0 •  We need a simple human (& machine) readable language to

describe our domain model to provide/promote: •  Service-friendly DTO, Persistence (RDB and NoSQL), Governance,

Reuse, Validation, MVC, Best Practice, Documentation •  We call the DSL as Universal Data Definition Language (UDDL).

•  Unify Data Definition across different platform application domains

MongoSV 2011 7 December 8, 2011

Our decision: Textual DSL

•  Derived from an Apache licensed open source project Sculptor •  http://fornax.itemis.de/confluence/display/fornax/Sculptor+%28CSC%29

•  Easy to learn, intuitive syntax of the textual DSL, based on the concepts from DDD.

•  Textual DSL has a lot of productivity benefits over graphical tools •  Quick development round trip, short feedback loop •  Generation of complete application from a single model, not only fragments

that are hard to fit in to the overall design •  Supports JPA (oracle/mysql/postgresql)/MongoDB/JAXB/Spring/SpringMVC •  Great extensibility and customization •  Based on Eclipse Xtext/Xtend/Xpand code generation framework •  Can be used with text editor or any IDE, but DSL editor with error highlight,

code completion, and outline is provided for Eclipse users •  Documentation w/ diagrams for domain model •  No runtime magic is built into the tools (generated code can be used without

the tooling)


Schema Definition - The idea

December 8, 2011

Domain Model

RDB

MongoDB

EnFty EnFty EnFty

DTO

REST Root

Resource

REST Sub-‐

Resource

DTO DTO

REST Sub-‐

Resource

Locate CRUD

CRUD

CRUD

POJO XSD

JSON Schema

Textual DSL (such as Sculptor)

Protocol Buffer/ThriT

DDL

12

1.  Persistence 2.  DTO 3.  Mapping 4.  IntrospecFon 5.  ValidaFon 6.  MVC/UI

3

metadata

4

uddl

Java code

ConfiguraFon files

Documents (UML diagrams, HTML

docs)

Generated arFfacts

Code generaFon templates

MongoSV 2011 9

Schema Definition - Sample

Application AddressBook { basePackage=com.shutterfly.model Module addressbook { hint="store=mongodb” Entity Contact { dto mappedName="contact" String id key; String userId ! nullable indexed; String prefix; String firstName; String middleName; String lastName; String company; - @ContactType type; - @Gender gender; List<String> groups; //group names - List<@ContactEmail> emails; - List<@ContactAddress> addresses; @'javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class)' Timestamp created mappedName="updtime"; @'javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class)' Timestamp lastUpdated mappedName="instime"; } BasicType ContactEmail { String label; Boolean isPrimary; String emailAddress ! nullable indexed; }

}

Persistence in use…

EnFty

Required Indexes

Embedded and References

Embedded and References


Schema Definition – Generated Java Code @Generated("uddl:AddressBook/addressbook/Contact") @XmlRootElement @XmlAccessorType(javax.xml.bind.annotation.XmlAccessType.FIELD) @Entity(value = "contact", noClassnameStored = true) public class Contact implements Serializable, Cloneable { @Id private String id; @XmlElement() private String userId; @XmlElement() private String prefix; @XmlElement() private String firstName; @XmlElement() private String middleName; @XmlElement() private String lastName; @XmlElement() private String company; @XmlElement() private List<String> groups = new ArrayList<String>(); @XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) @XmlElement(name = "updtime") @Property("updtime") private Date created; @XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) @XmlElement(name = "instime") @Property("instime") private Date lastUpdated; @XmlElement() private ContactType type; @XmlElement() private Gender gender; @XmlElementWrapper(name = "emails") @XmlElement(name = "email") @Embedded private List<ContactEmail> emails = new ArrayList<ContactEmail>(); @XmlElementWrapper(name = "addresses") @XmlElement(name = "address") @Embedded private List<ContactAddress> addresses = new ArrayList<ContactAddress>(); … }


Schema Definition - Documentation


DSL language development

Grammar Language (DSL)

EMF Ecore model

SemanFc model

Parse tree model

Grammar model

Eclipse UI Xtext

EMF Ecore model

Xtend/Check

Load/save Parse/validate

Extend/transform/validate

Java code

ConfiguraFon files

Documents (UML

diagrams, HTML docs)

Xpand

Eclipse Model Workflow Engine

(MWE)

Generated arFfacts

Other DSLs (such as XSD, GPB, ThriT or

Avro)


Schema Definition - Open source tools

Sculptor: •  http://fornax.itemis.de/confluence/display/fornax/Sculptor+%28CSC%29

Eclipse Xtext:

•  http://www.eclipse.org/Xtext/


Data Access Layer

Data Access Layer

•  Provides an Abstraction layer to Persistence API and storage •  Also facilitating migration from one Persistence storage to another

>  Or even Y adaptor pattern to write to multiple Persistence storages •  Promote rapidly adoption of new technologies

•  Provides different concrete APIs •  Query based APIs •  Entity based APIs •  Binary APIs (GridFS)

•  Cache •  Query Abstraction •  Ability to plug key generators •  Promotes best practices across all applications


Data Access Layer

Services

Data Access Layer

UDDL

ORM Cache Persistence Driver

Query Builder


Data Access Layer – Query Builder

•  Abstract how application developers define queries •  Agnostic of Persistence Storage

•  Support the following Operators •  EQUAL, NOTEQUAL, HASTHISONE, HASALLOF, HASANYOF, HASNONEOF,

HASTHISELEMENT,GREATERTHAN,GREATERHANOREQ,LESSTHAN,LESSTHANOREQ,WHERE,EXISTS,SIZEEQ,MOD,PATTERN

•  Support AND and OR compound operators

•  Syntax originally based on the Morphia ‘Query’

December 8, 2011

QueryBuilder query = dal.createQueryBuilder().filter(Group.USERID, QueryOperator.EQUAL, userId) .filter(Group.ID, QueryOperator.HASANYOF, groupNames); QueryBuilder query = dal.createQueryBuilder().orFilter("_id", QueryOperator.EQUAL, contactId) .orFilter(Contact.LEGACYID, QueryOperator.EQUAL, contactId);

MongoSV 2011 18

Data Access Layer - Cache

•  Provides out-of-the-box support for cache annotations for Queries and entities

December 8, 2011

@Cacheable(cacheName = ”contacts") public final List<Contact> getContacts(QueryBuilder query) throws ExcepFon

MongoSV 2011 19

Data Access Layer - ORM

•  Morphia is a lightweight type-safe library for mapping Java objects to/from MongoDB

•  Morphia provides easy support for a strongly-typed

programming model


Data Access Layer – ORM & Partial Updates

•  Scenario : Allow client applications to send partial documents and merge these changes on current document.

•  Regular behavior :

December 8, 2011

MongoDB { "_id" : ObjectId("4ee1b6ac5c31a483c62d2b15"), "firstName" : "Luciano", "lastName" : "Resende", "email" : {

"isPrimary" : true, "emailAddress" : “lresende@...”

} }

update({"_id":ObjectId(”…")}, {lastName:"Updated"}, false)

See : git://gist.github.com/1450628.git MongoSV 2011 21


•  Scenario : Allow client applications to send partial documents and merge these changes on current document.

•  Regular behavior :

December 8, 2011

MongoDB { "_id" : ObjectId("4ee1b6ac5c31a483c62d2b15"), "firstName" : "Luciano", }

update({"_id":ObjectId(”…")}, {lastName:"Updated"}, false)



•  Supporting Partial Updates •  Provides specific operations for explicitly applying partial

updates •  Internally, flatten the update structure into various $set

operations

December 8, 2011

MongoDB { "_id" : ObjectId("4ee1b6ac5c31a483c62d2b15"), "firstName" : "Luciano", "lastName" : ”Resende", "email" : {

"isPrimary" : false, "emailAddress" : “lresende@...”

} }

update({"_id":ObjectId(”…")}, {"$set": {"lastName": "Updated","email.isPrimary": false}}, false)



•  Supporting Partial Updates •  Provides specific operations for explicitly applying partial

updates •  Internally, flatten the update structure into various $set

operations

December 9, 2011

MongoDB { "_id" : ObjectId("4ee1b6ac5c31a483c62d2b15"), "firstName" : "Luciano", "lastName" : ”Updated", "email" : {

"isPrimary" : false, "emailAddress" : “lresende@...”

} }

update({"_id":ObjectId(”…")}, {"$set": {"lastName": "Updated","email.isPrimary": false}}, false)


Data Access Layer - Open source tools

•  Spring Data - MongoDB: •  http://www.springsource.org/spring-data/mongodb

•  Morphia •  http://code.google.com/p/morphia/


Summary

Summary

•  Schema Definition •  Cross Platform, Simple human (& machine) readable language

to describe our domain model entities

•  Data Access Layer •  Data Access Layer that in conjunction with UDDL abstracts the

persistence layer allowing for more flexible and clean code •  Promote rapidly adoption of new technologies


Q & A

Technology

Data access layer and schema definitions