Trio: A System for Data, Uncertainty, and Lineage

Martin Theobald

Jennifer Widom

Stanford University

Uncertainty in Databases

• Not a new idea — proposed 20+ years ago• “Classic” probabilistic database systems

• Uncertainty about attribute values

• Modeling of missing values/incompleteness

[Cavallo ‘87, Abiteboul ‘91, Garcia-Molina ‘92, Fuhr/Rölleke ‘97]

• Most initial (18+ years) work largely theoretical; not much systems-building until recently:URank – U Waterloo, Orion – Purdue U, MayBMS -

Saarland U

• But applications weren’t ready anyway

Are they now?

The “Trio” in Trio

1. Data Student #123 is majoring in Econ: (123,Econ)

2. Uncertainty Student #123 is majoring in Econ or CS:

(123, Econ ∥ CS) Major With confidence 60% student #456 is a CS major:

(456, CS 0.6) Major

3. Lineage 456 HardWorker derived from:

(456, CS) Major and“CS is hard” some web page

Original Motivation for Trio

New Application Domains

• Many involve data that is uncertain (probabilistic, imprecise, fuzzy, incomplete...)

• Many of the same ones need to track the lineage (provenance) of their data

Coincidence or Fate?Coincidence or Fate?

Original Motivation for Trio

New Application Domains

• Many involve data that is uncertain (probabilistic, imprecise, fuzzy, incomplete...)

• Many of the same ones need to track the lineage (provenance) of their data

Trio combines uncertainty & lineage for a closed & complete representation modelNeither uncertainty nor lineage is supported in current commercial database systems

Neither uncertainty nor lineage is supported in current commercial database systems

Sample Applications

Information extraction• Find & label entities in unstructured text• Often probabilistic

Information integration• Combine data from multiple sources• Inconsistencies, probabilistic schema mappings

“Human input” data / Scientific experiments• Inexact/incomplete data• Many levels of “derived data products”

Sample Applications

Sensor data management• Approximate readings• Missing readings• Levels of data aggregation

Deduplication (“data cleaning”)• Object linkage, entity resolution• Often heuristic/probabilistic

But *NOT* approximate query processing• Fast but inexact answers

Our Claim

The connection between uncertainty and lineage goes deeper than just a shared need by several applications

Coincidence or Fate?Coincidence or Fate?

Substantiation of Claim

[technical] Lineage:1. Enables simple and consistent

representation of uncertain data

2. Correlates uncertainty in query results with uncertainty in the input data

3. Can make computations over uncertain data more efficient

[fluffy] “Applications may use lineage to reduce or resolve uncertainty”

Our Goal

Develop a new kind of database management system (DBMS) in which:

1. Data2. Uncertainty3. Lineage

are all first-class interrelated concepts

With all the “usual” DBMS features

The “Usual” DBMS Features

1. Efficient,

2. Convenient,

3. Safe,

4. Multi-User storage of and access to

5. Massive amounts of

6. Persistent data

Standard Relational DBMSs

Persistent; Convenient• All data stored in simple tables (relations)

• Queries and updates via simple but powerful declarative language (SQL)

Multi-User; Safe• Transactions, Recovery

Massive; Efficient• Storage and indexing structures

• Query optimization

Trio: What Changes

Persistent; Convenient• All data stored in “uncertain” relations

• Queries and updates via simple but powerful declarative language

Multi-User; Safe• Transactions, Recovery

Massive; Efficient• Storage and indexing structures

Trio: What Changes

Persistent; Convenient• All data stored in “uncertain” relations

• Queries and updates via simple but powerful declarative language

Multi-User; Safe – standard DBMS underneath• Transactions, Recovery

Massive; Efficient – standard DBMS underneath• Storage and indexing structures

Another “Trio” in Trio

1. Data Model Simplest extension to relational model that’s

sufficiently expressive

2. Query Language Simple extension to SQL with well-defined

semantics and intuitive behavior

3. System A complete open-source DBMS that people

want to use

Another “Trio” in Trio

1. Data Model Uncertainty-Lineage Databases (ULDBs)

2. Query Language TriQL

3. System Trio-One — built on top of standard DBMS

Remainder of Talk

1. Data Model Uncertainty-Lineage Databases (ULDBs)

2. Query Language TriQL

3. SystemTrio-One

4. Real-World Applications PharmGKB.org

Running Example: Crime-Solving

Saw (witness, color, car) // may be uncertain

Drives (person, color, car) // may be uncertain

Suspects (person) = πperson(Saw ⋈ Drives)

In Standard Relational DBMS

Drives (person, color, car)

Jimmy red Toyota

Frank red Mazda

Billy blue Honda

Frank green

Saw (witness, color, car)

Amy red Mazda

Betty blue Honda

Carol green Toyota

Suspects

Create Table Suspects asSelect personFrom Saw, DrivesWhere Saw.color = Drives.color And Saw.car = Drives.car

Data Model: Uncertainty

An uncertain database enodes a set of possible instances based on different kinds of uncertainty:

• Jimmy drives either a Toyota or a Mazda, but not both.

• Amy saw either a Honda or a Toyota, or nothing?

• Hank is a suspect with confidence 0.7

• Betty saw either an Acura with confidence 0.5 or a Toyota with confidence 0.3

or nothing with confidence 0.2 ….

Our Model for Uncertainty

1. Alternatives

2. ‘?’ (Maybe) Annotations

3. Confidences

1. Alternatives: uncertainty about value

3. ConfidencesSaw (witness, color, car)

Amy red, Honda ∥ red, Toyota ∥ orange, Mazda

Three possible

instances

Six possibleinstances

1. Alternatives

2. ‘?’ (Maybe): uncertainty about presence

3. Confidences

Betty blue, Acura

1. Alternatives

2. ‘?’ (Maybe): uncertainty about presence

3. ConfidencesSaw (witness, color, car)

Betty blue, Acura?

Betty blue, Acura ∥ NULL, NULL

absent unknownabsent unknown

1. Alternatives

3. Confidences: weighted uncertainty

Six possible instances, each with a probability

Amy red, Honda 0.5 ∥ red, Toyota 0.3 ∥ orange, Mazda 0.2

Betty blue, Acura 0.6

Our Model is Not Closed

Saw (witness, car)

Honda ∥ Mazda

Drives (person, car)

Jimmy, Toyota ∥ Jimmy, Mazda

Billy, Honda ∥ Frank, Honda

Hank, Honda

Suspects

Billy ∥ Frank

Suspects = πperson(Saw ⋈ Drives)

Does not correctlycapture possibleinstances in theresult (not 12)!

CANNOT

to the Rescue

Lineage (provenance): “where data came from”• Internal lineage – from another Trio relation

• External lineage – from an external source

In Trio: A Boolean function λ from data elements to other data elements (or external sources)

Lineage

Example with Lineage

ID Saw (witness, car)

Honda ∥ Mazda

ID Drives (person, car)

Hank, Honda

ID Suspects

Billy ∥ Frank

λ(31) = (11,2)˄(21,2)

λ(32,1) = (11,1)˄(22,1)

λ(33) = (11,1) ˄ 23

; λ(32,2) = (11,1)˄(22,2)

Example with Lineage

ID Saw (witness, car)

Honda ∥ Mazda

ID Drives (person, car)

Hank, Honda

ID Suspects

Billy ∥ Frank

λ(31) = (11,2)˄(21,2)λ(32,1) = (11,1)˄(22,1); λ(32,2) = (11,1)˄(22,2)

λ(33) = (11,1) ˄ 23

Correctly captures possible instances inthe result (7)

Trio Data Model

1. Alternatives

3. Confidences

4. Lineage

ULDBs are closed and complete

Uncertainty-Lineage Databases (ULDBs)Uncertainty-Lineage Databases (ULDBs)

[Databases with Uncertainty and Lineage, VLDB J. Special Issue 2/08]

ULDBs: Lineage

Conjunctive lineage sufficient for most operations (,,⋈)

• Disjunctive lineage for duplicate-elimination ()

• Negative lineage for set difference ()

General case after several queries: • Boolean formula

• Can represent any finite (sub)set of possible instances

• Can capture arbitrary Boolean constraints among tuples

• Can represent any finite (sub)set of possible instances

• Can capture arbitrary Boolean constraints among tuples

ULDBs: Minimality

A ULDB relation R represents a set of possible

instances

• Does every tuple in R appear in some possible instance? (no extraneous tuples)

• Does every maybe-tuple in R not appear in some possible instance? (no extraneous ‘?’s)

• Also

Data-minimalityData-minimality

Lineage-minimalityLineage-minimality

Data Minimality Examples

Extraneous ‘?’

Billy, Honda ∥ Frank, Mazda

Billy ∥ Frank

?λ(40,1)=(10,1)λ(40,2)=(10,2)

extraneous

20 Billy, Honda

30 Frank, Mazda

Data Minimality Examples

Extraneous tuple

extraneous

. . . . . .

Diane Acura ∥ Mazda

. . . . . .

? λ(40,1)=(10,1)˄(10,2)

. . . . . .

20 Diane Acura

. . . . . .

30 Diane Mazda

. . . . . .

• Confidence computation over arbitrary Boolean expressions:

Efficient approximation algorithms (E.g. Monte Carlo simulations)

ULDBs: Complexity Questions• Does a given tuple t appear in some (all) possible

instance(s) of an uncertain relation R ?

• Is a given table T one of (all of) the possible instances of R ?

Polynomial algorithms based on data-minimizationPolynomial algorithms based on data-minimization

NP-hardNP-hard

#P-complete#P-complete

Querying ULDBs

Simple extension to SQL

Formal semantics, intuitive meaning

Query uncertainty, confidences, and lineage

TriQLTriQL

Formal Semantics

Relational (SQL) query Q on ULDB D

D1, D2, …, DnD1, D2, …, Dn

possibleinstances

Q on eachinstance

representationof instances

Q(D1), Q(D2), …, Q(Dn)Q(D1), Q(D2), …, Q(Dn)

implementation of Q

operational semanticsD + ResultD + Result

Operational Semantics: Example

SELECT personFROM Saw, DrivesWHERE Saw.car = Drives.car

Saw (witness, car)

Cathy Honda ∥ Mazda

Jim ∥ Bill Mazda

Hank Honda

(Cathy,Honda,Jim,Mazda)∥(Cathy,Honda,Bill,Mazda)∥(Cathy,Mazda,Jim,Mazda)∥(Cathy,Mazda,Bill,Mazda)

Saw (witness, car)

Jim ∥ Bill Mazda

Hank Honda

Saw (witness, car)

Jim ∥ Bill Mazda

Hank Honda

Saw (witness, car)

Jim ∥ Bill Mazda

Hank Honda

(Cathy,Honda,Hank,Honda) ∥ (Cathy,Mazda,Hank,Honda)

Saw (witness, car)

Jim ∥ Bill Mazda

Hank Honda

Saw (witness, car)

Jim ∥ Bill Mazda

Hank Honda

Saw (witness, car)

Jim ∥ Bill Mazda

Hank Honda

CREATE TABLE Suspects ASSELECT Drives.personFROM Saw, DrivesWHERE Saw.car = Drives.car

Suspects

Jim ∥ Bill

??λ( ) = ...λ( ) = ...

Saw (witness, car)

Jim ∥ Bill Mazda

Hank Honda

• Efficiently implementable on top of standard SQL

• Data computation overhead: O(n log(n))

• Efficiently implementable on top of standard SQL

• Data computation overhead: O(n log(n))

Confidences

Confidences supplied with base data

Trio computes confidences on query results• Default probabilistic interpretation (#P-complete)• Plug in different arithmetics, e.g., Min (linear)

Saw (witness, car)

Honda 0.6 ∥ Mazda 0.4

Jim 0.3 ∥ Bill 0.6 Mazda

Hank 1.0 Honda

Suspects

Jim 0.12 ∥ Bill 0.24

Hank 0.6

0.3 0.4

0.6 ProbabilisticProbabilistic Min

TriQL: Querying Confidences Built-in function: Conf()

SELECT person FROM Saw, Drives WHERE Saw.car = Drives.car AND Conf(Saw) > 0.5 AND Conf(Drives) > 0.8

Similarly: Conf(*)

SELECT person FROM Saw, Drives WHERE Saw.car = Drives.car AND Conf(*) > 0.5

Naïve data-minimization:

WHERE Conf(*) > 0

Naïve data-minimization:

WHERE Conf(*) > 0

TriQL: Querying Lineage

Built-in join predicate: Lineage()

SELECT Suspects.person FROM Suspects, Saw WHERE Saw.witness = ‘Amy’ AND Lineage(Suspects,Saw)

X ==> Y shorthand for Lineage(X,Y)

More TriQL Constructs

• “Horizontal subqueries”• Refer to tuple alternatives as a relation

• Aggregations: • (sum, count, avg, min, max) (low, high,

expected)

• Fetch modifiers:• Merged (eliminate horizontal duplicates)• Flatten, GroupAlts, NoLineage, NoConf, NoMaybe

• Set operators: Union [All], Intersect, ExceptConjunctive, Disjunctive & Negative Lineage

Horizontal Subquery Example

PrimeSuspect (crime#, suspect, accuser)

1 Jimmy, Amy ∥ Billy, Betty ∥ Hank, Cathy

2 Frank, Cathy ∥ Freddy, Betty

Credibility (person,scor

Amy 10

Betty 15

Cathy 5

Suspects

Jimmy 0.33 ∥ Billy 0.5 ∥ Hank 0.166

Frank 0.25 ∥ Freddy 0.75

List suspects with conf values based on accuser credibility

Amy 10

Betty 15

Cathy 5

Suspects

SELECT suspect, score/[sum(score)] as confFROM PrimeSuspect, CredibilityWHERE accuser = person

SELECT suspect, FROM PrimeSuspect, CredibilityWHERE accuser = person

Amy 10

Betty 15

Cathy 5

Suspects

scaled as conf

Remove extraneous ‘?’ if:

[Sum(Conf(*))] = 1

Remove extraneous ‘?’ if:

[Sum(Conf(*))] = 1

Trio-Specific Additional Features

• Lineage tracing

• On-demand confidence computation

• Coexistence checks

• Extraneous data removal

Interrelated algorithms based on lineage

The Trio System

• Version 1 (“Trio-One”)• Layered on top of a standard DBMS

(PostgreSQL)

• Surprisingly easy and complete, reasonably efficient

• Open source as of summer 2007• Supports Linux, Windows Server/XP/Vista

& Mac-OSX

• Constantly updated & maintained

Trio-One Overview

Standard relational DBMS

Trio API and TriQL rewriter(Python)

TrioPlus(Command-line

Client)

TrioPlus(Command-line

Client)

TrioMetadat

TrioExplorer(GUI client)

Trio Stored

Procedures

EncodedData

TablesLineageTables

Standard SQL• Partition and “verticalize”• Shared IDs for alternatives• Columns for confidence,“?”• One per result table• Uses unique IDs• Encodes formulas

• Table types• Schema-level lineage structure• Conf()• Lineage()

• DDL commands• TriQL queries• Schema browsing• Table browsing• Explore lineage• On-demand confidence computation

Postgres SPI functions

Applications

Pharmaceutical / Medical Databases• PharmGKB.org

• Relationships between Drugs, Diseases, Genes

• Evidence from Literature References, Clinical Outcomes, etc.

Applications – PharmGKB.org

DiseaseB

EffectiveA,BEffectiveA,B,C

• Relational Schema • Drugs(DrugID, Name, AltName)

• Diseases(DiseaseID, Name, AltName)

• Genes(GeneID, Name, Symbol, AltSymbol)

• Relationships(RelID, RelatedTo, Evidence)

• Independent Base Data

P[EffectiveA,B,C]

= P[A] P[B] P[C]

Applications – PharmGKB.org

DiseaseB

EffectiveA,BEffectiveA,B,C

• Relational Schema • Drugs(DrugID, Name, AltName)

• Diseases(DiseaseID, Name, AltName)

• Genes(GeneID, Name, Symbol, AltSymbol)

• Relationships(RelID, RelatedTo, Evidence)

• Non-Independent Base Data

P[EffectiveA,B,C]

= P[E|A,B,C]

≠ P[A] P[B] P[C]

ULDBs Vs. Bayesian Nets Factorized representation w/conditional probability tables (CPTs)

E.g.: C = Heart DiseaseB = High Blood PressureA = Warfarin

Applications – SERF

Stanford Entity Resolution Framework

Swoosh: A Generic Approach to Entity ResolutionBenjelloun, Garcia-Molina, Whang et al.

<id>r2<name>John Doe<phone>357-2635<email>jdoe@yahoo.com

<id>r1<name>J. Doe<phone>351-2535

<id>r3<name>John D.<phone>351-2535<email>jdoe@yahoo.com

<id>r4<name>{John Doe||John D.}<phone>{357-2635||351-2535}<email>jdoe@yahoo.com

<id>r5<name>{John Doe||John D.||J. Doe}<phone>{357-2635||351-2535}<email>jdoe@yahoo.com

Iteration 2Iteration 2

Stanford Entity Resolution Framework

Swoosh: A Generic Approach to Entity ResolutionBenjelloun, Garcia-Molina, Whang et al.

<id>r2<name>John Doe<phone>357-2635<email>jdoe@yahoo.com

<id>r1<name>J. Doe<phone>351-2535

<id>r3<name>John D.<phone>351-2535<email>jdoe@yahoo.com

<id>r4<name>{John Doe : 0.4 ||John D. : 0.3}<phone>{357-2635 : 0.5 ||351-2535 : 0.5}<email>jdoe@yahoo.com

<id>r5<name>{John Doe : 0.4 ||John D. : 0.3||J. Doe : 0.3}<phone>{357-2635 : 0.3 ||351-2535 : 0.6}<email>jdoe@yahoo.com

Applications – SERF

Future Topics (sample)

More forms of uncertainty• Continuous uncertainty (intervals, Gaussians)• Correlated uncertainty• Incomplete relations

More efficient confidence-based queries• Top-k by confidence• Confidence bounds & approximation

More forms of lineage• Update propagation w/lineage (versioning)• External lineage

Future Topics (sample)

Theory• AI models/Bayesian Nets & inferencing

• Design theory

System• Full TriQL query language• Version 2: Go native beyond Postgres SPI?

– Storage and indexing structures– Statistics– Query optimization

Search “stanford trio”

Trio contributors, past and present Parag Agrawal, Omar Benjelloun, Julien Chaumond, Ashok Chandra, Anish Das Sarma, Alon Halevy, Chris

Hayworth, Ander de Keijzer, Raghotham Murthy, Michi Mutsuzaki, Shubha Nabar, Tomoe Sugihara,

Martin Theobald, Jeff Ullman, Jennifer Widom

Trio: A System for Data, Uncertainty, and Lineage

Documents

Trio: A System for Integrated Management of Data, …ilpubs.stanford.edu:8090/658/1/2004-40.pdf · 5. Lineage may be queried. For example, “ﬁnd all records whose derivation includes

Databases With Uncertainty And Lineage

Lineage December 2011

Chinese Family Family and Lineage - University of Washingtonfaculty.washington.edu/sangok/JSISA405/Chinese family and lineage.… · Chinese Family Family and Lineage Social Change

Trio-One: Layering Uncertainty and Lineage on a Conventional DBMS Martin Theobald Jennifer Widom Stanford University

RESEARCHARTICLE Lineage … · RESEARCHARTICLE Lineage-SpecificChangesinBiomarkersin GreatApesandHumans ... (Pan troglodytes verus), 50 ... witheither adart

Trio: A System for Data, Uncertainty, and Lineage

Tsongkhapa Lineage

Lineage media profile

Lineage January2013

Trio: A System for Data, Uncertainty, and Lineage Jennifer Widom Stanford University

cdn.admitad-connect.com · Web viewlineage apk lineage os 14.1 lineage персонажи lineage 2 россия lineage 1 lineage 2 apk lineage вещи lineage скачать торрент

Special Interest Activity Trio – A System for Integrated Management of Data, Uncertainty, and Lineage ITK478 Yan Cui

Trio: A System for Data, Uncertainty, and Lineageforum.stanford.edu/events/2006/2006slides/infolab/widom.pdf · 3. Lineage are all first-class interrelated concepts. Trio . 9. The

SYNOPSIS OF DHARMA TALK ON KARMA KAGYU LINEAGE MASTERS Kagyu Lineage Masters.pdf · KARMA KAGYU LINEAGE MASTERS Page 1 of 16 SYNOPSIS OF DHARMA TALK ON KARMA KAGYU LINEAGE MASTERS

Lineage Powerpoint 2

Trio: A System for Data, Uncertainty, and Lineage

The Ancient Lineage

SHORIN LINEAGE - Squarespacestatic1.squarespace.com/static/56426837e4b0ccf90c65f64b/t/56b265bd...Isshin Kempo SHORIN LINEAGE Hard Style, Linear Karate BUDDHIST KEMPO LINEAGE Wu Hsing

Diversification ofthe Wntgenefamilyonthe ancestral lineage ... · lineage on (P