Mastering Differentiated MDSD Requirements at Deutsche Boerse AG

itemis AG 2009 All rights reserved

Mastering differentiated

MDSD requirements at

Deutsche Brse AG

Code Generation 2009

Heiko Behrens (itemis)

Karsten Thoms (itemis)

The project at Deutsche Brse

! Projects delivers a Global Trading System

! First (internal) user is ISE in NY

! Scalable, Distributed System

! Goes live in 2011

Structure of this talk

! 8 timesMotivation ! Applied Approach

! Inbetween: Demo of discussed ideas

! In the end: Some discoveries we made

http://www.bildtankstelle.de

Application ofCode Generation

in this project

Messaging Model

Structures

Messages Docu

HTML,

TextC++

Factories

Messaging Generator

Msg/Struct.

Adapters

Reference Data Model

Reference Data

Generator

Configuration Model

JPA Entities

Report

Java HTML

IML

Structures

Messages

Java

DAOs

Listener

Container Model

Containers

Filters

Boost

Operator

C++

XML

Model

Data

Access

Generator

XML

Transactional

Model

Queries

DAOs

C++

DB

Service

Generator

Hibernate

SQL

DDL Scripts

JAXB

XML

WSFacade

XSD

Val.List.

UnitTest

Validation

Listeners

WSFacade

DTOs

Persistence

Descriptor

IML

Datatypes

Model

UOW Exec.

Bean

Ref Data

Mngr Class

Global key

functionsDDL Scripts

SQL

UnitTest

Templates

DDL Scripts

(MySQL)

SQL

Excel Model

System Test

Generator

Python

Tests

Messages

Structures

Messages

Structures

C++ Python

Scripting Generator

Factories

Structures

Messages

.Net

.Net

Generator

OFi

Editor

Msg Client

Descriptor

Artifact sizes

Reference Impl.Sources (manual)Tests (manual)ModelTransformationGenerated

2.747 units

Actual numbers had to be replaced by

fictive units

Guess what!

Motivation:Code Generation

Why Code Generation?! Positive experience with former projects

! Prime example: mapping between messages and structures sending with a compact binary wire protocols

ComponentComponent

B1

B2

Structure B

S1

S2

SubStructure S

A2A3A1

Component

B1

B2

Structure B

S1

S2

SubStructure S

S2B2S1B1

Why openArchitectureWare?

Former times:

! Home-grown solution based on graphical modeling and proprietary tools

! Many different post-processors such as XSLT and Perl

Evaluation results of oAW for GTS

! Supports textual modeling

! Has mature tool support

! Works with different model types and output formats

UMLXpand

JavaC++Text

Different Input

XMIXML XMIXML

Same Transformation Language, Different Output ,

When in Rome do as the Romans do

Motivation: Address existing experience and perception

Modeling in UML

! Years of experience for modeling entities graphically (ER-Diagram)

! Established Tool (MagicDraw)

! Proven Technology (Profiles, Tagged Values)

! Great for overview perspectives

UML Reference Model

Textual Modeling

! Years of experience ;)

! Established tool support (grep, editor, mail, svn)

! Highly customizable ! DSL

! Inexpensive metamodel evolvement

! Great for detailed modeling

Textual Model

!""#$%&!$'!($)#*"

Motivation:Reduce Complexity of Modeling

Reducing Model Complexity

! Following Convention-over-configuration paradigm for modeling

! Only model necessary information

! Complete model according to conventions by M2M transformation

Convention over Configuration

can be derived

required model information

uml::Model modify (uml::Model this) : ... -> // Search for unnamed association ends and give them a default name this.eAllContents.typeSelect(Property) .select(x|x.association!=null).nameUnnamedEnds() -> ...

/** * This function sets a default name for an association's end property if the property * has no name and it is navigable. */ Void nameUnnamedEnds(Property this) : if !isNamed() && isNavigable() then ( ("setting default name for association end "+opposite.type.name+" -> "+type.name).info() -> setName(computeDefaultName()) ) -> this;

Convention: Navigable association ends withoutname get the name of the referred Entity. Use plural for to

many relationships.

Convention over Configuration

uml::Model modify (uml::Model this) : ... -> // Search for unnamed association ends and give them a default name this.eAllContents.typeSelect(Property) .select(x|x.association!=null).nameUnnamedEnds() -> ...

private create uml::Property createTechnicalId(RDMProfile::Entity entity): ("Creating technical id attribute for entity "+entity.name).debug() -> entity.ownedAttribute.addToFront(this) -> this.setName("id") -> this.setType(findDataType(Long) -> this.setLower(1) -> this.setUpper(1) -> this.setIsUnique(true) -> this.setVisibility(uml::VisibilityKind::^private) -> this.applyStereotype("RDMProfile::Key") -> this.setTaggedValue("RDMProfile::Key", "keyType", RDMProfile::KeyType::PK)-> this.applyStereotype("RDMProfile::Field") -> this.setTaggedValue("RDMProfile::Field", "uniqueGroup", "444")-> this;

Convention: All entities without attribute get a new property with name id of type Long, and Stereotype

is applied.

Combination of model types

Domain modelUML2platform independent

entity Market

{

relatedTo MarketAbstract with EAGER loading and cascade with PERSIST

relatedTo MarketGroup with LAZY loading and cascade with PERSIST

}

Configuration modeltextual DSLplatform dependent

entity Market

{

relatedTo MarketAbstract with EAGER loading and cascade with PERSIST

relatedTo MarketGroup with LAZY loading and cascade with PERSIST

}

public class Market implements RDSEntity, Serializable { ... @OneToMany(mappedBy = "market", cascade = CascadeType.PERSIST) @Basic(fetch = FetchType.EAGER) private List marketAbstracts = new ArrayList(); @ManyToMany(cascade = CascadeType.PERSIST, targetEntity = MarketGroup.class, mappedBy = "markets") @Basic(fetch = FetchType.LAZY) private List marketGroups = new ArrayList();

Combination of model types

Avoidable Accidents

Motivation:Detect errors early

(c) Ernest von Rosen, www.amgmedia.com

context R

DMProfile:

:EntityRel

ation

(memberEnd

.select(e|

e.upper ==

-1).size

== 2)

WARNI

NG "Many t

o many rel

ation " +

memberEnd.

type.name.

info() +

as owner o

f the rela

tion (owne

rOfAssocia

tion is no

t set)!" :

ownerO

fAssociati

on != null

;

context u

ml::Associ

ation if m

emberEnd.t

ype.select

(e|e.isAct

ive()).siz

e>1 && (me

mberEnd.se

lect(e|

e.upper ==

-1).size

== 2) && m

emberEnd.s

elect(e|e.

aggregatio

n.toString

()!="none"

WARNI

NG "Many t

o many rel

ation " +

memberEnd.

type.name.

info() + "

must be o

f type Ent

ityRelatio

n

and define

one of th

e entities

as owner

of the rel

ation (ste

reotype no

t applied)

!"

RDMPro

file::Enti

tyRelation

.isInstanc

e(this);

context P

roperty if

class.isA

ctive() &&

(associat

ion == nul

l)

ERROR

"Property

" + name

+ " in ent

ity "+clas

s.name+" r

eferences

an entity,

but is n

ot an

associatio

n!":

!RDMPr

ofile::Ent

ity.isInst

ance(type)

;

cont

ext Proper

ty if clas

s.isActive

() && asso

ciation ==

null && ^

default!=

WARNI

NG "Proper

ty "+loc()

+" has an

empty stri

ng (not nu

ll) as def

ault value

"

="";

cont

ext Proper

ty if clas

s.isActive

() && RDMP

rofile::En

tity.isIns

tance(clas

s) && asso

ciation!=

ERROR

"Class "+

type.name+

" must be

stereotype

d as Entit

y, since i

t is assoc

iated by E

ntity

"+class.na

me+" in it

s property

"+name :

RDMPro

file::Enti

ty.isInsta

ncef(type)

;

context P

roperty if

class.isA

ctive() &&

RDMProfil

e::Entity.

isInstance

(class) &&

associati

on==

ERROR

"Only Fun

ctional Da

tatypes or

Functiona

l Enumerat

ions are a

llowed as

property t

ype.

Violated f

or "+loc()

:

RDMPro

file::Func

tionalData

Type.isIns

tance(type

) ||

RDMProfile

::Function

alEnumerat

ion.isInst

ance(type)

;

Detecting errors early

! Check constraints implemented for

! Every assumption or modeling restriction

! Modeling error that lead to generator or artifact errors (e.g. Whitespace in names)

! Inter-model consistency checks

! Risk of modeling errors decreased

context uml::Association if memberEnd.type.select(e|e.isActive()).size>1 && (memberEnd.select(e|e.upper == -1).size == 2) && memberEnd.select(e|e.aggregation.toString()!="none").size == 2 WARNING "Many to many relation " + memberEnd.type.name.info() + " must be of type EntityRelation and define one of the entities" +

" as owner of the relation (stereotype not applied)!" : RDMProfile::EntityRelation.isInstance(this);

context Property if class.isActive() && (association == null) ERROR "Property " + name + " in entity "+class.name +

" references an entity, but is not an association!": !RDMProfile::Entity.isInstance(type);

context Property if class.isActive() && association == null && ^default!=null WARNING "Property "+loc() +" has an empty string (not null) as default value" : ^default.trim()!="";

Constraints with Check language

Integration

! Static Typing while developingchecks and transformations

! IDE integration for workflows invocation

! Continuous Generation (server-side)CMake / Ant / Maven

! Automated execution after SVN checkout

Demo! Graphical EMF model

! Enhanced with textual DSL (with Xtext)

! Validation

! Code Generation

! Changes textual DSL

Standing on the shoulders of giants

Motivation:Reusing generator aspects

Cartridge adaption! Using OS cartridges as

closed package

! Required model structure created through M2M transformation

! Required changes added non-invasive through Xpand/Xtend AOP support

! Redefined functionality concentrated in a small adapter cartridge project

Annotation

Template

EntityClass

Template

Mapping

Template

Attribute

Template

Class

Template

Operation

Template

Fornax Hibernate Cartridge

Fornax JavaBasic Cartridge

Aspect

Template

M2M

Transformation

Adapter

Workflow

Cartridge

Workflow

Cartridge

Workflow

Fornax Adapter Cartridge

DTO

Template

WebService

Template

UnitTest

Template

Project Cartridge

Cartridge

Workflow

uml::Model modify (uml::Model this) : this.applyProfile(getPersistenceProfile()) -> // Create real enumerations for FunctionalEnumerations. This must be done first since // we map datatypes after that this.eAllContents.typeSelect(RDMProfile::FunctionalEnumeration).adaptEnumeration() -> entities().adaptEntity() -> adaptEmbeddedKeys() -> this;

RDMProfile::Entity adaptEntity (RDMProfile::Entity entity) : debug("Adapting entity: "+ entity.name) -> entity.applyStereotype("Persistence::Entity") -> entity.setTaggedValue("Persistence::Entity", "tableName", entity.name.asTableName())-> entity.testAndAssignInheritanceStrategy()-> entity.attribute.select(a|a.association==null).adaptAttribute() -> entity.attribute.select(a|a.association!=null).adaptRelation() -> entity;

//In case the entity extends another class, assigne the persistence stereotype for //strategy on the uml::Generalization//TPC="TABLE_PER_CLASS" | J="JOINED" | ST="SINGLE_TABLE";Void testAndAssignInheritanceStrategy(RDMProfile::Entity this): if !general.isEmpty then assignInheritanceStrategy(); Void assignInheritanceStrategy(RDMProfile::Entity this): let gener = generalization.first(): gener.applyStereotype("Persistence::"+getInheritanceStrategyFromDardl().mapToStrategy()) -> if(getInheritanceStrategyFromDardl() == "SINGLE_TABLE") then ( gener.setTaggedValue("Persistence::TablePerClassHierarchy",

"discriminator_value", getDiscriminatorValue())-> gener.setTaggedValue("Persistence::TablePerClassHierarchy",

"discriminator_superClass_column", gener.general.name+"_type")-> gener.setTaggedValue("Persistence::TablePerClassHierarchy",

"discriminator_superClass_type", "INTEGER") ) ;

UML2 model modification with Xtend

around org::fornax::cartridges::uml2::javabasic::extensions::DataType::NormalizedDefaultValue(uml::Property property): internal_getNormalizedDefaultValue(property);

private String internal_getNormalizedDefaultValue (uml::Property property) : JAVA gts.ise.refdata.oaw.fornax.util.Extensions.getNormalizedDefaultValue(org.eclipse.uml2.uml.Property);

AROUND org::fornax::cartridges::uml2::javabasic::templates::Documentation::documentation FOR uml::Classifier targetDef.proceed() EXPAND JPAAnnotationsENDAROUND

DEFINE JPAAnnotations FOR Persistence::Entity @javax.persistence.Entity @javax.persistence.Table(name="asTableName()") EXPAND InheritanceAnnotations @javax.persistence.NamedQueries({ @javax.persistence.NamedQuery(name="findnames", query="SELECT e from name e") })

IF !isAbstract @javax.persistence.EntityListeners (getQualifiedPackageName("ListenerOnce").nameValidationListener.class) ENDIF EXPAND UniqueConstraintsENDDEFINE

AOP with templates and functions

Arange your ideas

Motivation:Managing Manual Code

Sucessfully applied MDSD best practices

! Separate generated from manual code

! Avoid check-in generated code

BaseClass

AbstractA AbstractB

ConcreteBSome

Manual

Code

ConcreteA

src-gen

always generated

src-once

generated once

src

manually written

BaseClass

ConcreteB

AbstractB

ConcreteCSome

Manual

Code

ConcreteA

AbstractD

ConcreteD

Adjusted Folder Layout

Design your tools

Motivation: Managing Generator

Complexity

Reference Implementation

! Manually implemented

! Far more than a prototype!

! Compilable, deployable, executable, testable

! Covers every architectural concept

! Code seperated in to-be-generated and manual code

Reference Model

! Contains every supported modeling concept

! Not necessary excerpt from the real domain, but helps for understanding

! Try to minimize the model size

Testing the reference

! Reference Implementation is tested intensively and automatic

! Unit tests and integration tests with high coverage

! Tests against generated artifacts

! Real application does not need to be tested in the same extend

generated from reference model

manually implemented

reference

EntitiesAttributesAssociationsModel metrics

Motivation: Optimizing Generator Speedand Readability

Refactorings / Profiling

! Reference Implementation/Unit-Tests ensure valid output

! Static Typing of Xpand supports development

HTML report

standard format (gprof)

Most significant problems

! Repeated calls of functions

! Suboptimal algorithms/data structures

! Branches with best-case scenarios

! Profiling reduced generation timefrom 28 min downto 3 min

Discoveries

Value of Cartridges

! Project benefited from cartridges developed by OS community

! But: Largest part of the code generators are totally project dependent

Early adopters

! Using the bleeding edge of Eclipse Modeling was challenging

! But: Continuous improvement of used technologies thanks to direct feedback with OS community

! And: Significant performance and stability gains since early adoption

Learning curve

! Establishing MDSD and tools was hard for unexperienced developers at the beginning

! But: Coaching helped to fill the gap fast

! And: Now developers educate others to use MDSD tooling

Reference Implementation

! Reference Models and Reference Implementation are invaluable

! Reducing the complexity

! Enable refactoring

! But: Effort to create and maintain a real Reference Implementation quite high

Personal blogshttp://www.1160pm.net http://kthoms.wordpress.com http://www.itemis.com