Schema and Data Translation

Schema and Data Translation

Paolo AtzeniUniversità Roma Tre

Based onTutorial at ICDE 2006

Paper in EDBT 2006 (with P. Cappellari and P. Bernstein)Demo in Sigmod 2007 (with P. Cappellari and G. Gianforme)

ADBIS --- Varna, October 2, 2007

P. Atzeni Varna - October 2, 2007 2

Outline

• Introduction• Model management• Schema and data translation: the problem• A metamodel based approach


A ten-year goal for database research

• The “Asilomar report”(Bernstein et al. Sigmod Record 1999 www.acm.org/sigmod):– The information utility:

make it easy for everyone to store, organize, access, and analyze the majority of human information online


A general framework: cooperation

• "The capacity of a system to interact (effectively) with other systems, possibly managed by different organizations"

• Forms of cooperation– Process-centered cooperation:

• the systems offer services one another, by exchanging messages, or by triggering activities, without making remote data explicitly visible

– Data-centered cooperation: • the systems offer data one another; data is distributed,

heterogeneous and autonomous


Databases in the Internet era

• Databases before the Internet– An internal infrastructure, a precious resource, but usually

hidden, with some controlled cooperation• Internet changes the requirements

– More users (not only humans), more diverse cooperating systems (distributed, heterogeneous, autonomous), more types of data

• "Future" Internet changes more– New devices (embedded everywhere), even more users

(many “per person”), real mobility, need for personalization and adaptation


The most studied form of data-centered cooperation: integration

• We are interested in data-centered cooperation, often referred to as integration

“The unification of related, heterogeneous data from disparate sources, for example, to enable collaboration” (Hammer & Stonebraker 2005)

• Some "paradigms" …


Multidatabase

Global Manager

Local mgr

DB

Mediator

Local mgr

DB

Mediator

Local mgr

DB

Mediator


Data Warehousing System

Mediator

Local manager

Mediator

Local manager

Mediator

Local manager

“Integrator”

DB DB DB

Data Warehouse

DW manager


Intermediate solutions in practice

Mediator

Local manager

DB

Local manager

DB

Integrator

Mediator

Local manager

Mediator

Local manager

DB DB

DB

Local Manager

Application


Peer-based architecturePeer mgr

Local mgr

DB

Local mgr

DB

Mediator

Mediator

Peer mgr

Local mgr

DB

Local mgr

DB

Mediator

Mediator

Peer mgr

Local mgr

DB

Local mgr

DB

Mediator

Mediator


Data is not just in databases

• Mail messages• Web pages• Spreadsheets• Textual documents• Palmtop devices, mobile phones• Multimedia annotations (e.g., in digital photos)• XML documents


The same data in the same form?

• Adaptivity:– Personalization: content adapted to the user

• upon system's decision• upon user's request

– Customization: structure adapted to the user• according to the user's role• upon user's request

– Context dependence• User, Device, Network, Place, Time, Rate


Summarizing: a general need

• We have data at various places, and data has to be– exchanged– replicated – migrated– integrated – adapted


A major difficulty

• Heterogeneity– System level– Model level– Structural (different structure for similar data)– Semantic (different meaning for the same structure)

• Many efforts, but current techniques are mostly manual and ad hoc


A direction for the solutions

• Be general! – ad hoc solution could work in-the-small, but they

• are repetitive and time consuming • do not scale• are not maintainable

• Historical notes:– W. C. McGee: Generalization: Key to Successful Electronic

Data Processing. J. ACM 1959• Indeed, databases are the result of generalization applied to

secondary storage management!


Generality requires …

• … high-level descriptions of problems within the family of interest:– Metadata:

• “data about data”• (formal or informal) description of structures and

meaning

• General solutions leverage on metadata (and then operate on data as a consequence)


Outline

• Introduction and motivation• Model management• Schema and data translation: the problem• A metamodel based approach


A wider perspective

• (Generic) Model Management:– A proposal by Bernstein et al (2000 +)– Includes a set of operators on

• schemas and • mappings between schemas


Terminology: a warning

Model Mgmt people Traditional DB people

Meta-metamodel Metamodel

Metamodel Model

Model Schema


Schemas and mappings

• A simplified approach:– Schema:

• a set of elements, related in some way to one another– Mapping:

• a set of correspondences (pair of elements) or• its reification, a third schema related to the other two via

two sets of correspondences


Model mgmt operators, a first set

• map = Match (S1, S2) • S3 = Merge (S1, S2, map)• S2 = Diff (S1, map) • and more

– map3 = Compose (map1, map2)

– S2 = Select (S1, pred)

– Apply (S, f)

– list = Enumerate (S)

– S2 = Copy (S1)

– …


Match

• map = Match (S1, S2)– given

• two schemas S1, S2– returns

• a mapping between them• the “classical” initial step in data integration:

– find the common elements of two schemas and the correspondences between them


Merge

• S3 = Merge (S1, S2, map)– given

• two schemas and a mapping between them– returns

• a third schema (and two mappings)• the “classical” second step in data integration:

– given the correspondences, find a way to obtain one schema out of two


Diff

• S2 = Diff (S1, map) – given

• a schema and a mapping from it (to some other schema, not relevant)

– returns • a (sub-)schema, with the elements that do not participate

in the mapping


DW2

Example

(Bernstein and Rahm, ER 2000)• A database (a “source”), a data warehouse and a mapping

between the two• we get a second source, with some similarity to the first one• and we want to update the DW

DB1 DW1

DB2


DW2

Example, the "solution"

DB1 DW1

DB2

m1

m2m3

DB2’

m2 = Match(DB1,DB2)

m3= Compose(m2,m1)

DB2’=Diff(DB2,m3)

DW2’, m4 user defined

m5 = Match(DW1,DW2’)

DW2 = Merge(DW1,DW2’,m5)

DW2’m4

m5


Magic does not exist

• Operators might require human intervention:– Match is the main case

• Scripts involving operators might require human intervention as well (or at least benefit from it):– a full implementation of each operator might not always

available– a mapping might require manual specification– incomparable alternatives might exist


The “data level”

• The major operators have also an extended version that operates on data, and not only on schemas

• Especially apparent for– Merge


We also have heterogeneity

• Round trip engineering (Bernstein, CIDR 2003)– A specification (for example ER or UML) and an

implementation (for example, relational)– then a change to the implementation: want to revise the

specification• We need a translation from the implementation model to the

specification one

S1

I1 I2

S2


Model management with heterogeneity

• The previous operators have to be “model generic” (capable of working on different models)

• We need a “translation” operator– <S2, map12> = ModelGen (S1)


ModelGen, an additional operator

• <S2, map12> = ModelGen (S1) – given

• a schema (in a model)– returns

• a schema (in a different data model) and a mapping between the two

• A “translation” from a model to another• I should call it “SchemaGen” …• We should better write

– <S2, map12> = ModelGen (S1,mod2)


S2

Round trip engineering

S1

I1

m1

I2

m2 = Match (I1,I2)m3 = Compose (m1,m2)I2’= Diff(I2,m3)<S2’,m4 > = Modelgen(I2’)… Match, Merge

m2

m3

I2’

S2’

m4


Outline



Schema and data translation, a long standing issue

• Schema translation:– given schema S1 in model M1 and model M2– find a schema S2 in M2 that “corresponds” to S1

• Schema and data translation:– given also a database D1 for S1– find also a database D2 for S2 that “contains the same data”

to D1


Schema and data translation, a long standing issue

• Translations from a model to another have been studied since the 1970’s

• Whenever a new model is defined, techniques and tools to generate translations are studied

• However, proposals and solutions are usually model specific


Model specific solutions

• Given an ER schema, find the suitable relational schema that “implements” it – the original paper (Chen 1976) contains the basics– further discussions by many (e.g. Markowitz and Shoshani

1989)– illustrated in every textbook

• Similarly with – any other conceptual model and any other logical one– XML and relational (or object)


Another problem in the picture: data exchange

• Given a source S1 and a target schema S2 (in different models or even in the same one), find a translation, that is, a function that given a database D1 for S1 produces a database D2 for S2 that “correspond” to D1

• Often emphasized with reference to materialized solutions


Integration

• Given two or more sources, build an integrated schema or database


Schema translation

• Given a schema find another one with respect to some specific goal (another model, better quality, …)


Data exchange

• Given a source and a target schema, find a transformation from the former to the latter


Schema translation and data exchange

• Can be seen a complementary:– Data translation = schema translation + data exchange

• Given a source schema and database• Schema translation produces the target schema• Data exchange generates the target database

• In model management terms we could write– Schema translation:

• <S2, map12> = ModelGen (S1,mod2)– Data exchange:

• i2 = DataGen (S1,i1,S2,map12)


Outline



The problem

• ModelGen– given two data models M1 and M2, and a schema S1 of M1

(the source schema and model), • generate a schema S2 of M2 (the target schema and

model), corresponding to S1• and, for each database D1 over S1, generate an

equivalent database D2 over S2


We have been doing this for a while

• Initial work more than ten years ago (Atzeni & Torlone, 1996)• Major novelty recently (Atzeni, Cappellari & Bernstein, 2006)

– translation of both schemas and data– data-level translations generated automatically, from

schema-level ones


A simple example

• An object relational database, to be translated in a relational one

• Source: the OR-model

• Target: the relational model

System managed ids

used as references


Example, 2

• Does the OR model allow for keys? • Assume EmpNo and Name are keys


Example, 3

• Does the OR model allow for keys? • Assume no keys are specified


Many different models (and variations …)

Binary ER w/o gen

N-ary ER w/ gen

Binary ERw/ gen

Relational

Bin ER w/ gen w/o attr on rel

OO w/ gen

N-ary ER w/o gen

Bin ER w/o gen w/o attr on rel

Bin ER w/o gen w/o M:N rel

Bin ER w/ gen w/o M:N rel

OO w/o gen…

OR

…

XSD

…


Heterogeneity

• We need to handle artifacts and data in various models– Data are defined wrt to schemas– Schemas are defined wrt to models – How models can be defined?

Models

Schemas

Data


A metamodel approach

• The constructs in the various models are rather similar:

– can be classified into a few categories (Hull & King 1986):

• Lexical: set of printable values (domain)

• Abstract (entity, class, …)

• Aggregation: a construction based on (subsets of) cartesian products (relationship, table)

• Function (attribute, property)

• Hierarchies

• …

• We can fix a set of metaconstructs (each with variants):

– lexical, abstract, aggregation, function, ...

– the set can be extended if needed, but this will not be frequent

• A model is defined in terms of the metaconstructs it uses


The metamodel approach, example

• The ER model:– Abstract (called Entity)– Function from Abstract to Lexical (Attribute)– Aggregation of abstracts (Relationship) – …

• The OR model:– Abstract (Table with ID)– Function from Abstract to Lexical (value-based Attribute)– Function from Abstract to Abstract (reference Attribute)– Aggregation of lexicals (value-based Table)– Component of Aggregation of Lexicals (Column)– …


The supermodel

• A model that includes all the meta-constructs (in their most general forms)– Each model is subsumed by the supermodel (modulo

construct renaming) – Each schema for any model is also a schema for the

supermodel (modulo construct renaming)


The metamodel approach, translations

• The constructs in the various models are rather similar: – can be classified into a few categories (“metaconstructs'')– translations can be defined on metaconstructs,

• and there are “standard”, accepted ways to deal with translations of metaconstructs

• they can be performed within the supermodel– each translation from the supermodel SM to a target model

M is also a translation from any other model to M:• given n models, we need n translations, not n2


Generic translation environment

1. Copy

2. Translation

3. Copy

Supermodel

Source model

Target model

Translation:composition 1,2 & 3


Translations within the supermodel

• We still have too many models:– Just within simple ER model versions, we have 4 or 5

constructs, and each has several independent features which give rise to variants

• for example, relationships can be– binary or N-ary– with all possible cardinalities or without many-to-

many– with or without the possibility of specifying optionality– with or without attributes– …

– Combining all these, we get hundreds of models!– The management of a specific translation for each model

would be hopeless


Translations, the approach

• Elementary translation steps to be combined• Each translation step handles a supermodel construct (or a

feature thereof) "to be eliminated" or "transformed"• A translation is the concatenation of elementary translation

steps


A complex translation, example(0,N) (0,N)

• Eliminate N-ary relationships • Eliminate attributes from relationships • Eliminate many-to-many relationships• Replace relationships with references• Eliminate generalizations


Complex translations

Binary ER w/o gen

N-ary ER w/ gen

Binary ERw/ gen

Relational

Bin ER w/ gen w/o attr on rel

OO w/ gen

N-ary ER w/o gen

Bin ER w/o gen w/o attr on rel

Bin ER w/o gen w/o M:N rel

Elim. N-ary relationships Elim. Relationship attr.sElim. M:N relationshipsReplace relationships with

referencesElim OO generalizations Elim ER generalizations

Bin ER w/ gen w/o M:N rel

OO w/o gen…


Translations

• Basic translations are written in a variant of Datalog, with OID invention– We specify them at the schema level– The tool "translates them down" to the data level– Some completion or tuning may be needed


A Multi-Level Dictionary

• Handles models, schemas and data• Has both a model specific and a model independent component


Multi-Level Dictionary

model

schema

modelgeneric

modelspecific

des

crip

tion

modelindependence

Supermodel description

(mSM)

Model descriptions

(mM)

Supermodel schemas

(SM)

Model specific schemas

(M)

Supermodel instances

(i-SM)

Model specific instances

(i-M)data


The supermodel description

MSM-Construct

OID Name IsLex

1 AggregationOfLexicals F

2 ComponentOfAggrOfLex T

3 Abstract F

4 AttributeOfAbstract T

5 BinaryAggregationOfAbstracts F

MSM-Property

OID Name Constr. Type

11 Name 1 String

12 Name 2 String

13 IsKey 2 Boolean

14 IsNullable 2 Boolean

15 Type 2 String

16 Name 3 String

17 Name 4 String

18 IsIdentifier 4 Boolean

19 IsNullable 4 Boolean

20 Type 4 String

21 IsFunct1 5 Boolean

22 IsOptional1 5 Boolean

23 Role1 5 String

24 IsFunct2 5 Boolean

25 IsOptional2 5 Boolean

26 Role2 5 String

MSM-Reference

OID Name Construct Target

30 Aggregation 2 1

31 Abstract 4 3

32 Abstract1 5 3

33 Abstract2 5 3

mSMmM

SMM

i-SMi-M


Model descriptions

MSM-Construct

OID Name IsLex

1 AggregationOfLexicals F

2 ComponentOfAggrOfLex T

3 Abstract F


5 BinaryAggregationOfAbstracts F

MM-Construct

OID Model MSM-Constr Name

1 2 3 ER_Entity

2 2 4 ER_Attribute

3 2 5 ER_Relationship

4 1 1 Rel_Table

5 1 2 Rel_Column

6 3 3 OO_Class

MM-Model

OID Name

1 Relational

2 Entity-Relationship

3 Object

MSM-Property

OID Name Construct Type

… … … …

MSM-Reference

OID Name Construct …

… … … …

MM-Property

… … … …

MM-Reference

… … … …

mSMmM

SMM

i-SMi-M


Schemas in a model

ER-Entity

OID Schema Name

301 1 Employees

302 1 Departments

201 3 Clerks

202 3 Offices

ER-AttributeOfEntity

OID Schema Name isIdent isNullable Type AbstrOID

401 1 EmpNo T F Int 301

402 1 Name F F Text 301

404 1 Name T F Char 302

405 1 Address F F Text 302

501 3 Code T F Int 201

… … … … … … …

SM-Construct

OID Name IsLex

… … …

3 ER-Entity F

4 ER-AttributeOfEntity T

… … …

ER schemas

mSMmM

SMM

i-SMi-M

Employees

Departments

EmpNo

Name

Name

Address


Schemas in the supermodel

SM-Abstract

OID Schema Name

301 1 Employees

302 1 Departments

201 3 Clerks

202 3 Offices

SM-AttributeOfAbstract







… … … … … … …

MSM-Construct

OID Name IsLex

… … …

3 Abstract F


… … …

Supermodel schemas

mSMmM

SMM

i-SMi-M

Employees

Departments

EmpNo

Name

Name

Address


i-SM-Abstract

OID AbsOID InstOID

3010 301 1

3011 301 1

… … …

3010 301 2

i-SM-AttributeOfAbstract

OID AttrOfAbsOID InstOID Value i- AbsOID

4010 401 1 75432 3010

4020 402 1 John Doe 3010

…

4021 402 1 Bob White 3011

… … … … …

Instances in the supermodel

SM-Abstract

OID Schema Name

301 1 Employees

302 1 Departments

201 3 Clerks

202 3 Offices








… … … … … … …Supermodel schemasSupermodel instances

mSMmM

SMM

i-SMi-M


Multi-Level Repository, generation and use

model

schema

modelgeneric

modelspecific

des

crip

tion

modelindependence

Supermodel description

(mSM)

Model descriptions

(mM)

Supermodel schemas

(SM)

Model specific schemas

(M)


(i-SM)

Model specific instances

(i-M)data

Structure fixed, content provided by tool

designers

Structure generated by the tool from the content of mSM

Structure generated by the tool from the content of mSM

Structure fixed, content provided by model

designers out of mSM

Structure generated by the tool from the content of mM


Translations

• Basic translations are written in a variant of Datalog, with OID invention– We specify them at the schema level– The tool "translates them down" to the data level– Some completion or tuning may be needed


A basic translation

• From (a simple) binary ER model to the relational model– a table for each entity– a column (in the table for E) for each attribute of an entity E– for each M:N relationship

• a table for the relationship• columns …

– for each 1:N and 1:1 relationship:• a column for each attribute of the identifier …


A basic translation application

Departments

Name Address

EmployeesEmpNo

Name

Affiliation

Departments

1,1

0,N

Name

Address

Employees

EmpNo Name Affiliation


A basic translation (in supermodel terms)

• From (a simple) binary ER model to the relational model– an aggregation of lexicals for each abstract– a component of the aggregation for each attribute of abstract– for each M:N aggregation of abstracts …

• …

• From (a simple) binary ER model to the relational model– a table for each entity– a column (in the table for E) for each attribute of an entity E– for each M:N relationship

• a table for the relationship• columns …

– for each 1:N and 1:1 relationship:• a column for each attribute of the identifier …


"An aggregation of lexicals for each abstract"

SM_AggregationOfLexicals(

OID: #aggregationOID_1(OID),

Name: name)

SM_Abstract (

OID: OID,

Name: name ) ;


Datalog with OID invention

• Datalog (informally):– a logic programming language with no function symbols and

predicates that correspond to relations in a database– we use a non-positional notation

• Datalog with OID invention:– an extension of Datalog that uses Skolem functions to

generate new identifiers when needed• Skolem functions:

– injective functions that generate "new" values (value that do not appear anywhere else); so different Skolem functions have disjoint ranges





Name: n)

SM_Abstract (

OID: OID,

Name: n ) ;

• the value for the attribute Name is copied (by using variable n)

• the value for OID is "invented": a new value for the function #aggregationOID_1(OID) for each different value of OID, so a different value for each value of SM_Abstract.OID



SM-Abstract

OID Schema Name

301 1 Employees

302 1 Departments

… … …





… … … … … … …

EmployeesEmpNoName

11

11

Departments1002

Employees1001



Name: n)

SM_Abstract (

OID: OID,

Name: n ) ;

…

3021002

……

1001

SM-AggregationOfLexicals

Schema NameOID

SM-aggregationOID_1_SK

absOIDOID

301

Employees


"A component of the aggregation for each attribute of abstract"

SM_ComponentOfAggregation… ( OID: #componentOID_1(attOID),Name: name, AggrOID: #aggregationOID_1(absOID),IsNullable: isNullable, IsKey: isIdent, Type : type )

←SM_AttributeOfAbstract(

OID: attOID,Name: name,AbstractOID: absOID, IsIdent: isIdent, IsNullable: isNullable , Type : type ) ;

• Skolem functions

– are functions

– are injective

– have disjoint ranges

• the first function "generates" a new value

• the second "reuses" the value generated by the first rule


A component of the aggregation for each attribute of abstract"

SM-Abstract

OID Schema Name

301 1 Employees

302 1 Departments

… … …

EmployeesEmpNoName

11

11

Departments1002

Employees1001

…

3021002

……

1001

SM-AggregationOfLexicals

Schema NameOID

SM-aggregationOID_1_SK

absOIDOID

301

SM_ComponentOfAggregation… ( OID: #componentOID_1(attOID),Name: name, AggrOID: #aggregationOID_1(absOID),IsNullable: isNullable, IsKey: isIdent, Type : type )


OID: attOID,Name: name,AbstractOID: absOID, IsIdent: isIdent, IsNullable: isNullable , Type : type ) ;

SM-componentOID_1_SK

absOIDOID

Employees

EmpNo Name





… … … … … … …

SM-ComponentOfAggregationOfLexicals

Schema Type AggrOIDisNullableisIdentNameOID

Text 1001FFName111004

1001IntFTEmpNo11

1003 401

1004 402

1003


Generating data-level translations

Schema translation

Data translation

• Same environment

• Same language

• High level translation specificationSupermodel description

(mSM)

Supermodel schemas

(SM)


(i-SM)


Translation rules, data level

i-SM_ ComponentOfAggregation… (OID: #i-componentOID_1 (i-attOID),i-AggrOID: #i-aggregationOID_1(i-absOID),ComponentOfAggregationOfLexicalsOID:

#componentOID_1(attOID),Value: Value )

←i-SM_AttributeOfAbstract(

OID: i-attOID,i-AbstractOID: i-absOID,AttributeOfAbstractOID: attOID,Value: Value ),

SM_AttributeOfAbstract(OID: attOID,AbstractOID: absOID,Name: attName,IsNullable: isNull,IsID: isIdent,Type: type )

SM_ComponentOfAggregation… ( OID: #componentOID_1(attOID), Name: name, AggrOID:

#aggregationOID_1(absOID),IsNullable: isNullable, IsKey: isIdent, Type : type )


OID: attOID,Name: name,AbstractOID: absOID, IsIdent: isIdent, IsNullable: isNullable , Type: type ) ;


Correctness

• Usually modelled in terms of information capacity equivalence/dominance (Hull 1986, Miller 1993, 1994)

• Mainly negative results in practical settings that are non-trivial• Probably hopeless to have correctness in general• We follow an "axiomatic" approach:

– We have to verify the correctness of the basic translations, and then infer that of complex ones


Experiments

• A significant set of models– ER (in many variants and extensions)– Relational– OR– XSD– UML

• Demo (good luck!)


XSD

OR noTab ref, FK, nested

Relational

OO ref, nested

OR Tab, ref, FK, nested

OO ref, flat

OR noTab, FK, nested

OR noTab, FK, flat

OR Tab, genref, FK, nested

OR noTab, gen

ref, FK, nested

OR noTab, gen, FK, nested

37+ Remove generalizations36 Unnest sets (with ref)03 Unnest sets (with FKs)24 Unnest structures (flatten)06 Unnest structures (TT & FKs)01 Unnest structures (TT & ref)43 FKs for references29 Tables for typed tables30 Typed tables for tables13 References for FK31 Nest referenced classes


XSD

OR noTab ref, FK, nested

Relational

OO ref, nested

OR Tab, ref, FK, nested

OO ref, flat

OR noTab, FK, nested

OR noTab, FK, flat

OR Tab, genref, FK, nested

OR noTab, gen

ref, FK, nested

OR noTab, gen, FK, nested

36 Unnest sets (with ref)

24 Unnest structures (flatten)

01 Unnest structures (TT & ref)43 FKs for references29 Tables for typed tables


Summary

• ModelGen was studied a few years ago• New interest on it within the "Model management" framework• New approach

– Translation of schema and data– Visible (and in part self generated) dictionary– Visible and modifiable rules– Skolem functions describe mappings


Conclusion

• The ten-year goal of the Asilomar report:– The information utility:

make it easy for everyone to store, organize, access, and analyze the majority of human information online

• A lot of interesting work has been done but …• …integration, translation, exchange are still difficult…• … 2009 is approaching … we are late!

Documents

Schema and Data Translation