Introduction to Datomic

Siva Jagadeesan

“there’s an another database!”

• what do you think of first?

The job of a database

•Coordination •Consistency •Indexing •Storage •Queries

The traditional world

• Place-oriented-programming (PLOP)

• designed decades ago

• tiny RAM, tiny disks

• Collocated components

• Difficult to scale

• Necessitates application-level sharding

The dreaded update-in-place

sharding

• lose

• consistency

• transactions

• queries

key-value stores

• lose

• querying

• really just storage systems

• not “databases”

Deconstruction

• relocate subsystems

• separate services each do one thing

• simplification

Datomic architecture

• Peers

• Transactors

• Storage services

The new world

Benefits

• Separating reads and writes

• applications only read from storage

• transactor

• handles transactions

• provides consistency

• reflects changes to peers

Benefits

• Integrated data distribution

• built-in in-memory caches

• self-tunes to working set

• automatic

Benefits

• Peers each have a query engine

• Datalog

• simple rules and data patterns

• declarative: implicit joins

• declarative: meaning is evident

• locality: datomic db and app data

Benefits

• Elasticity

• as elastic as peers

Benefits

• Cloud ready

• commodity hardware

• resilient to failures

Data model

• Immutable data

• remember everything

• things don’t actually “change”

• audit everything

• automatic

Data model

• Facts-oriented (vs. PLOP)

• Atomic Data

• datoms

• entity, attribute, value, transaction

• Not embedded into strucures

Data model

• Minimal Schema

• at the datom level

• directly supports cardinality

• avoid rigidity

• even other “schema-free” documents impose their structure into apps

• Hickey: “nothing pivots like a datom”

Programming model

• Peer embedded in your application

• Pulls indexes or data segments as needed

• Caches locally, gets updates from transactor

Programming model

• No strings-based query language

• Data-structure-driven

• Lists and maps

• Easy to generate and test

Programming model

• Transactional

• Totally ordered

• First class: peers get a queue of all transactions

• Can use them in queries

• Facilitates event-driven triggers without polling

• Annotated transactions

Programming model

• Datalog

• deductive query system

• facts = datomic db

• can also include other app data

• extensible through custom functions that queries can use

Programming model

• Consistent

• without impeding other threads or peers

• reads/queries without transactions

• through immutability

• db as a value

Programming model

• Time

• as-of queries

• windowed queries

The so-what

• Simplicity

• consistency

• built-in caching

• no manual sharding

• configuration-free

• cloud-scale

The so-what

• Evolvability

• datom-level schema

• application free to change “structural” thinking

The so-what

• Local query power

• each peer has full query engine

• the db is effectively local

• isolation from others:

• e.g. - analytics usage won’t bog down transaction processing

The so-what

• Multiple storage choices

• in-memory for developing, unit-testing

• RDBMS backend possible (behind firewall, etc)

• Distributed storage services for redundancy and unlimited read scaling

• backup/restore tools available

The so-what

• Integrated mem-cache support

• local caches can be backed by OTS mem-cache cluster

• datomic is a “good citizen” user

The so-what

• Leverage datalog against multiple sources

• datomic db

• application data

The so-what

• Travel through time

• audits

• bug-resolution

Schema def

{:db/id {:part :db.part/db, :idx -1000001}, :db/ident :user/login, :db/valueType :db.type/string, :db/cardinality :db.cardinality/one, :db/fulltext true, :db/doc "the login handle of a user", :db.install/_attribute :db.part/db}

Schema def

{:db/id {:part :db.part/db, :idx -1000003}, :db/ident :user/friends, :db/valueType :db.type/ref, :db/cardinality :db.cardinality/many, :db/fulltext false, :db/doc "the friends of a user", :db.install/_attribute :db.part/db}

transactions

[:db/add entity-id attribute value]

[:db/retract entity-id attribute value]

{:db/id entity-id! attribute value! attribute value! ... }

[:db.fn/retractEntity entity-id]

identities

{:part :db.part/user, :idx -1000026}

{:db/id 17592186045420}

queries

[:find ?e :in $ ?email !:where ![?e :person/email ?email]]

[:find variables :where clauses]

[:find ?e ?x !:where ![?e :age 42] [?e :likes ?x]]

queries

• lots more

• pattern matching

• logic programming

• multiple data-sources

demonic

• datomic helper

• work at the data-structure level

• supports graphs: nested maps

• only writes dirty data

• “demarcations” as “batched datomic transactions”

• unit-testing support

Questions