From Spreadsheets to Relational Databases and Backalfa.di.uminho.pt/~jacome/down/pepm09_talk.pdfCreate a Relational Database (RDB) schema (3NF) Use data reﬁnements to perform the

From Spreadsheets to Relational

Databases and Back

Jacome Cunha Joao Saraiva Joost Visser

Departamento de Informatica SIG & CWIUniversidade do Minho The Netherlands

Portugal

PEPM’09, January, 19-20

1.1

Motivation



Portugal

Property Renting System

From Spreadsheets to Relational Databases and Back

Each row represents a renting transaction

It includes information about owners, clients, properties

It also contains dates and prices (formulas)

Row 3 “says” that john has rented a property owned by tony, with

address 5 Novar Dr., price per day 70, between dates 1/7/00 and

8/31/01 and paid 25550 for it

1.2

Motivation



Portugal

Redundancy


This unstructured model is valid and serves its purpose

But, it contains data redundancy

For example, the rent per day is repeated several times

1.3

Motivation



Portugal

Updating Problems


As a result, updates can cause data inconsistence

For example, updating the renting value of property pg4 must be

performed in several places

1.4

Motivation



Portugal

Deleting Problems


Deleting rows can be problematic, too

For example, deleting the row 5 will remove all the information

about property pg36

1.5

Motivation



Portugal

A Database Solution


ClientclientNo VARCHAR(256) PcName VARCHAR(256)

RentingclientNo VARCHAR(256) FPpropertyNo VARCHAR(256) FPrentStart VARCHAR(256) PrentFinish VARCHAR(256) PownerNo VARCHAR(256) FPtotal_rent VARCHAR(256)totalDays VARCHAR(256)

OwnerownerNo VARCHAR(256) PoName VARCHAR(256) U

PropertypropertyNo VARCHAR(256) PpAddress VARCHAR(256)rentPerDay VARCHAR(256)oName VARCHAR(256) F

A normalized (3NF) data representation

No redundancy

No update/delete problems

1.6

Motivation



Portugal

A Database Solution


A normalized (3NF) data representation

No redundancy

No update/delete problems: the price per day occurs only once

Tables can be stored in different sheets

1.6

Motivation



Portugal

A Database Solution


We can use information computed during normalization to pro-

duce a refactored spreadsheet that helps users to introduce cor-

rect data

Spreadsheet warns the user when errors are introduced

2.0

Our Approach



Portugal


s : S Ti

FDs CFDs

≥

fun

to

from choosePKs

synthesize

S spreadsheet schema (header)s spreadsheet data (remaining rows)CFDs compound functional dependenciesT relational database schemai referential integrity constraintsto data conversion from RDB to SSfrom data conversion from SS to RDB

Infer functional dependencies (Fun)

Normalize (3NF) those functional dependencies (synthesize)

Create a Relational Database (RDB) schema (3NF)

Use data refinements to perform the data migration (to and from)

HaExcel combines all these steps creating a complete system

2.1

Our Approach



Portugal

Calculate FDs


A Functional Dependency (FD) denoted A ⇀ B means that an

element of A is uniquely associated with an element of B

The Fun algorithm (Novelli et. al.) calculates the FDs from data

For our example, the Haskell implementation returns:

ownerNo ⇀ oName

totalDays ⇀ clientNo, cName

propertyNo ⇀ pAddress, rentPerDay , ownerNo, oName

...

Formulas also induce FDs:

C0 = f(X0, . . . , Xn) induces X0, . . . , Xn ⇀ C0

2.2

Our Approach



Portugal

A Few Concepts


Each tuple in a DB table is uniquely identified by a set of attri-

butes called Primary Key (PK)

There may be more then one set suitable for becoming PK. They

are designated Candidate Keys (CK)

First Normal Form (1NF) is respected if each element of all tuples

are atomic values

Second Normal Form (2NF) is respected if the 1NF is respected

and its non-key attributes are not functionally dependent on part

of the key

Third Normal Form (3NF) is respected if the 2NF is respected and

if the non-key attributes are only dependent on the key attributes

2.3

Our Approach



Portugal

Normalize FDs


Synthesize algorithm (Maier) calculates 3NF set of FDs

It returns a set of attributes with candidate keys

It is necessary to choose one CK to be the primary key

To produce a schema with the lossless decomposition property

one must add a FD (Maier):

{all atts } \ {atts rep formulas }⇀ {new att , atts rep formulas }

For our example:

clientNo, propertyNo, cName, pAddress, rentStart , rentFinish,

rentPerDay , ownerNo, oName ⇀ totalDays, total rent , new att

2.4

Our Approach



Portugal

Calculate a RDB Schema


Fun produces too many, not necessary and redundant FDs

Also, attributes representing columns with formulas can not be

used in a PK

We use Synthesize to produce a 3NF set of attributes/CK

Its input is the result from Fun without the FDs with formulas

in PKs and the FD that guarantees the losslees decomposition

property

We choose the smallest CK to be the PK

2.5

Our Approach



Portugal


For our example,

owners

ownerNo ⇀ oName

clients

clientNo ⇀ cName

properties

propertyNo ⇀ pAddress, rentPerDay , oName

renting

clientNo, propertyNo, rentStart , rentFinish, ownerNo ⇀ total rent , totalDays

3.1

From a Spreadsheet to a Database



Portugal

Data Refinements


We need to migrate data from spreadsheets to the relational

model

We use data refinements:

A[rr]−tooo 6 B[ll]−fromppaaaaaaaaaaaaaaaaaaaaaaa

to : A→ B is an injective function;from : B → A a surjective function;from · to = idA (identity function on A);

B is a refinement of A witnessed by the to and from functions

In the case where the refinement works in both directions we have

an isomorphism A ∼= B

3.1




Portugal

Data Refinements


Composition

if A[rr]−tooo 6 B[ll]−fromppaaaaaaaaaaaaaaaaaaaaaaa and B[rr]−to′oo 6 C[ll]−from′

ppaaaaaaaaaaaaaaaaaaaaaaaa then A[rr]−from · from′oo 6 C[ll]−to′ · toppaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Propagation

if A[rr]−tooo 6 B[ll]−fromppaaaaaaaaaaaaaaaaaaaaaaa then FA[rr]−Ftooo 6 FB[ll]−Ffrom

ppaaaaaaaaaaaaaaaaaaaaaaaaaa

3.2




Portugal

Examples: Hierarchical-Relational Data Mapping


A? 6 N ⇀ A List elimination2A ∼= A ⇀ 1 Set eliminationA? ∼= 1 ⇀ A Optional elimination

A+B 6 A?×B? Sum eliminationA× (B + C) ∼= (A×B) + (A× C) Distribute product over sumA ⇀ (B + C) 6 (A ⇀ B)× (A ⇀ C) Distribute map over sum (range)(B + C) ⇀ A ∼= (B ⇀ A)× (C ⇀ A) Distribute map over sum (domain)

3.3




Portugal

Data Refinements With Constraints


To represent database constraints we use type constraints

Associate a constraint φ to a type A: Aφ where φ : A→ Bool

Add a second constraint to a constrained datatype: (Aφ)ψ∼= Aφ∧ψ

Functorial pull: F(Aφ)∼= (FA)(Fφ)

For example, a constraint on the elements of a list can be pulled

up to a constraint on the list: (Aφ)∗ ∼= (A∗)listφ

3.3




Portugal

Data Refinements With Constraints


Refining a constrained datatype (source)

if A[rr]−tooo 6 B[ll]−fromppaaaaaaaaaaaaaaaaaaaaaaa then Aφ[rr]

−tooo 6 Bφ·from[ll]−fromppaaaaaaaaaaaaaaaaaaaaaaaaa

A law that introduces a constraint to a constrained datatype

if A[rr]−tooo 6 Bψ[ll]−from

ppaaaaaaaaaaaaaaaaaaaaaaa then Aφ[rr]−tooo 6 (Bψ)φ·from[ll]−from

ppaaaaaaaaaaaaaaaaaaaaaaaaa∼= Bψ∧(φ·from)

3.4




Portugal

Constraint-Aware Rewriting


There is an Haskell implementation of data refinement called 2LT

– Two Level Transformation (http://code.google.com/p/2lt)

We need a type-safe representation for types, so we use a GADT

data Type t where

Int :: Type Int

[·] :: Type a → Type [a ]

· × · :: Type a → Type b → Type (a, b)

Func :: Type a → Type b → Type (a → b)

(·)· :: Type a → PF (Pred a) → Type a (Aφ)

...

type Pred a = a → Bool

3.5




Portugal



For the function representation we also use a GADT

data PF a where

π1 :: PF ((a, b) → a)

δ :: PF ((a ⇀ b) → Set a)

·? :: PF (a → b) → PF ([a ] → [b ])

· × · :: PF (a → b) → PF (c → d) → PF ((a, c) → (b, d))

...

Function representations can be evaluated to the function that is

represented:

eval :: Type (a → b) → PF (a → b) → a → b

3.5




Portugal



Data refinement rules in Haskell

type Rule = ∀ a . Type a → Maybe (View (Type a))

data View a where

View :: Rep a b → Type b → View (Type a)

data Rep a b = Rep{to = PF (a → b), from = PF (b → a)}

3.5




Portugal



Rewriting composition operators

nop :: Rule -- identity

. ::Rule → Rule → Rule -- sequential composition

� ::Rule → Rule → Rule -- left-biased choice

many :: Rule → Rule -- repetition

once :: Rule → Rule -- arbitrary depth rule application

3.6




Portugal

Refining a Relational Table to a Spreadsheet Table


A ⇀ B

Table2sstable

$$

6 (A×B)?fd

Sstable2table

bb

data PF a where

...

Sstable2table :: PF ([(a, b)] → (a ⇀ b))

Table2sstable :: PF ((a ⇀ b) → [(a, b)])

table2sstable :: Rule

table2sstable (a ⇀ b) = return (View rep [a × b]fd)

where

rep = Rep{to = Table2sstable, from = Sstable2table }

3.7




Portugal

Refining Relational Tables where a PK Contains a FK


((A ⇀ B)× (C ⇀ D))πA◦δ◦π1⊆πC◦δ◦π2[dd]−Table2sstable× Table2sstableoo

6

((A×B)?fd × (C ×D)?fd)πA◦list2set◦π?1◦π1⊆πC◦list2set◦π?1◦π2[uu]−Sstable2table× Sstable2table

OO

A Foreign Key (FK) is a set of attributes within one relation that

matches the PK of some relation

3.8




Portugal

Refining Tables where Non-Key Attributes Contain FKs


((A ⇀ B)× (C ⇀ D))πB◦ρ◦π1⊆πC◦δ◦π2[dd]−Table2sstable× Table2sstableoo

6

((A×B)?fd × (C ×D)?fd)πB◦list2set◦π?2◦π1⊆πC◦list2set◦π?1◦π2[uu]−Sstable2table× Sstable2table

OO

3.9




Portugal

The Strategy


rdb2ss :: Rule

rdb2ss = simplifyInv B

(many (aux tables2table)) B

(many ((aux tables2sstables) B (aux tables2sstables ′))) B

(many (aux table2sstable))

where

aux r = ((once r) B simplifyInv) B ((many (once r)) B simplifyInv)

4.0

The HaExcel Framework



Portugal


HaExcel is a framework to manipulate and transform spreadsheets

Haskell library: generic and reusable library to transform spre-

adsheets into RDBs and back

Front-ends: Excel/Gnumeric (XML) formats can be imported and

exported

Tools:

an online tool, available at

http://haskell.di.uminho.pt/jacome/index.html

a batch tool, compilable from sources available at

http://haskell.di.uminho.pt/websvn

http://haskell.di.uminho.pt/jacome/index.html

http://haskell.di.uminho.pt/websvn

5.0

Conclusions



Portugal


We extended the 2LT framework with new constraint-aware two-

level data refinements

We shown how these rules can be combined into a strategic re-

write system

We constructed HaExcel that transforms a relational schema into

a tabular schema and back

Our techniques also migrate the formulas between the models

We connected importers and exporters for SQL and spreadsheet

formats

This allows refactoring of spreadsheets to reduce data redun-

dancy and improve error detection and migration of spreadsheet

applications

An Excel plugin that implements HaExcel is under development

6.0

Questions?



Portugal


Questions?

Documents

From Spreadsheets to Relational Databases and Backalfa.di.uminho.pt/~jacome/down/pepm09_talk.pdfCreate a Relational Database (RDB) schema (3NF) Use data reﬁnements to perform the