Palantir's Revisioning Database

© 2008 Palantir Technologies Inc. All rights reserved.

Palantir Revisioning DatabaseBob McGrew

Director of Engineering

Introduction

Imagine a community of thousands of analysts all reading and writing to a traditional database.

Problems for analysts:– No history of additions and edits for data– No ability to discover “what we knew when”– No prior review of shared data– No granular sharing

Revisioning Database features

Online object history Separate spaces for analysis Granular sharing of changes



Alternative approach: audit logs

Write every change to audit log– Separate data store (database or flat file)– Separate tools (SQL or grep)

Not integrated with analysis– Offline and inaccessible by design– No security info– No analyst-facing tools– Infeasible to determine “what we knew when”

Audit logging is important, but no substitute for an online history

Integrated object history

Online history of every change

Ability to view “what we knew when”

Modeling data: The object model

Each stored in separate database table

Modeling time: Application events

Model time as a linear sequence of application events Provides atomicity for changes The database ID for an app event always increases with time

– Total ordering for all app events– Wall clock time allows ties

Revisioning Database: the basic idea

Every edit to an object component inserts a new row into the relevant database table

Once inserted, rows are never updated

Down to the database

Revisioning fields:– id: long-living component ID– object_id: ties the component to its object– app_event_id: totally ordered timestamp for row– version: primary key for row, based on ID and app_event_id– deleted: 0 if not deleted; non-zero if deleted

Metadata fields:– time_created: time component was created– created_by: user ID of component creator– last_modified: time row was created

Value fields:– <value fields>: everything else

Modifying an object

op id object_id app_event_id version deleted value

Create 10 10 1 1001 0 Type:Person

Create 101 10 1 1002 0 Name:Mike Fikri

Create 102 10 2 1003 0 Phone#:650-494-1574

Edit 101 10 3 1004 0 Name:Michael Fikri

Delete 102 10 4 1005 1 Phone#:650-494-1574

Loading an object

App Event # Object #1 Property #101 Property #1021 Type: Person Name: Mike Fikri2 Phone#: 650-494-15743 Name: Michael Fikri4 Phone#: 650-494-1574

op id object_id app_event_id version deleted value

Create 10 10 1 1001 0 Type:Person

Create 101 10 1 1002 0 Name:Mike Fikri

Create 102 10 2 1003 0 Phone#:650-494-1574

Edit 101 10 3 1004 0 Name:Michael Fikri

Delete 102 10 4 1005 1 Phone#:650-494-1574

Loading an object


• Choose all rows that • Belong to object 10

Loading an object


Type: Person Name: Michael Fikri Phone#: 650-494-1574

• Choose all rows that • Belong to object 10• Have not been superseded by a later version

Loading an object


Type: Person Name: Michael Fikri

• Choose all rows that • Belong to object 10• Have not been superseded by a later version• Are not deleted

Loading an object from history

App Event # Object #10 Property #101 Property #1021 Type: Person Name: Mike Fikri2 Phone#: 650-494-1574

3 Name: Michael Fikri4 Phone#: 650-494-1574

• To load the version of Mike at app event 2, choose all rows that

• Belong to object 10• Have not been superseded by a later version less than 2• Are not deleted



Type: Person Name: Mike Fikri Phone#: 650-494-15743 Name: Michael Fikri4 Phone#: 650-494-1574





Type: Person Name: Mike Fikri Phone#: 650-494-15743 Name: Michael Fikri4 Phone#: 650-494-1574



Speed of operations

Time efficient– Edits– Fetch current version of object– Fetch past version of object– Fetch all changes in a given time range

Space efficient– No overhead if object never changed– Only one row overhead per change



Separate spaces for analysis

Explore competing hypotheses Edit information without affecting others Control over visibility of others’ edits Solution:

– Let analysts “check out” a copy of the data– Similar to source control systems like SVN or CVS

A realm of one’s own

A realm is a complete view of all objects– Each investigation– Data Repository

Investigations inherit data from the Repository– Load and view Repository data– Search Repository data


Revisioning fields:– id: long-living component ID– object_id: ties the component to its object– realm_id: the realm that this row is in– app_event_id: totally ordered timestamp– version: primary key, based on ID and app_event_id– deleted: 0 if not deleted; non-zero if deleted


Revisioning fields:– id: long-living component ID– object_id: ties the component to its object– realm_id: the realm that this row is in– app_event_id: totally ordered timestamp– version: primary key, based on ID and app_event_id– deleted: 0 if not deleted; non-zero if deleted

Making inheritance work

Copy-on-read approach– Copy on first viewing– Edit as normal– Expensive for large objects or many objects

Copy-on-write– Write a pointer to the right object state– Edits supersede pointed-to rows– Super-space efficient; no redundant info

The realm_object relation

Pointer is relation between objects and realm realm_object row:

– object_id: the object that we are locking– realm_id: the investigative realm– source_realm_id: the Data Repository – source_realm_app_event_id: the app event that the object is

locked into Realm_objects are object components Realm_objects are revisioned!

Object modification

Object addition to realm– Write one realm_object row– source_realm_app_event is the current app event

Edit– Write only changed rows to investigative realm– Need not write full object– Data Repository untouched

Loading an object

If there is no realm_object row,– Load the object from the Data Repository at the latest app event

If there is a realm_object row,– Load the object from the Data Repository at the app event

specified– Load all rows from the investigative realm at the current app

event– Investigative rows supersede Data Repository rows

Loading with realm objects

App Event # Object #10 Property #101 Property #1021 Type: Entity Name: Mike Fikri2 Phone#: 650-494-15743 Name: Michael Fikri4 Phone#: 650-494-15745

Data Repo Type: Entity Name: Michael Fikri

App Event # Object #10 Property #101 Property #1026 Type: Person

Investigation Type: Person

Loading with realm objects

Realm Object #10 Property #101 Property #102Data Repo Type: Entity Name: Michael Fikri

Investigation Type: Person


Speed of operations

Time efficient– Edits– Fetch current version of object– Fetch past version of object– Fetch all changes in a given time range

Space efficient– Almost all data shared between realms– One row written per edit

Revisioning Database Features


Granular data sharing

Changes developed in one realm need to be shared with other analysts

Data sharing must be per-object Update

– Bringing changes from the Data Repository into an investigation Publish

– Moving changes from an investigation into the Data Repository

Update

Edit the realm_object Insert new realm_object row:

– object_id: unchanged– realm_id: unchanged– source_realm_id: unchanged– source_realm_app_event_id: overwrite to the new app event

that the object is locked into– Set other revisioning fields appropriately

Publish

Two-phase process Data Repository

– Copy rows from investigative realm– Only need to copy rows that are still current

Investigative realm– Need to allow Data Repository rows to be authoritative– Insert rows with deleted = -1– Rows with deleted = -1 do not supersede Data Repository rows

Publish

id object_id realm_id app_event_id deleted value

10 10 100 1 0 Type:Person

101 10 100 1 0 Name:Mike Fikri

102 10 100 2 0 Phone#:650-494-1574

101 10 100 3 0 Name:Michael Fikri

102 10 100 4 1 Phone#:650-494-1574

Publish: Copy rows to Repository




102 10 100 2 0 Phone#:650-494-1574


102 10 100 4 1 Phone#:650-494-1574




Publish: Delete rows in investigation




102 10 100 2 0 Phone#:650-494-1574


102 10 100 4 1 Phone#:650-494-1574

10 10 100 6 -1 Type:Person

101 10 100 6 -1 Name:Michael Fikri




Loading after publish

App Event # Object #10 Property #101 Property #1021 Type: Person Name: Mike Fikri2 Phone#: 650-494-15743 Name: Michael Fikri4 Phone#: 650-494-15746 Type: Person Name: Michael Fikri

Investigation

App Event # Object #10 Property #101 Property #1025 Type: Person Name: Michael Fikri

Data Repo Type: Person Name: Michael Fikri

Loading after publish

Realm Object #10 Property #101 Property #102Data Repo Type: Person Name: Michael Fikri

Investigation


Speed of operations

Time efficient– Updates– Publish is bulk in-database insert

Space efficient– Updates– Publish writes no more rows than the original– Often considerably fewer



Other issues not covered

Branching history in investigations Loading links Conflict resolution Synchronizing external indices (search server)

© 2008 Palantir Technologies Inc. All rights reserved.

Palantir Revisioning DatabaseBob McGrew

Director of Engineering

Technology

Palantir's Revisioning Database