OrigoDBBuild faster systems faster

Robert Friberg

• Design, build and maintain systems for clients at Devrex• Trainer – Microsoft NET, SQL Server, java, perl, python, linux• Machine learning, AI• Squash fanatic• @robertfriberg, robert@devrexlabs.com

Why?

ServiceLayer

DomainLayer

Data AccessLayer

RelationalModel

Views/SP’sXBuild faster systems faster

What is OrigoDB?

• In-memory database toolkit• Code and data in same process• Write-ahead command logging and snapshots• Open Source single DLL for NET/Mono• Commercial server with mirror replication

.NET Process Storage

Engine

2. Execute command

Handles queries and commands, guards model

1. AppendToLogPlaceOrderCommand

Snapshot

Client codepasses commandsand queries

PlaceOrderCommand

NewCustomerCommand

IncreaseInventoryLevelCommand

PlaceOrderCommand

PlaceOrderCommand

Snapshot

time

In-memoryModel

How does it work?

Demand drives change

• Performance• Data volume• Scalability• Availability• Modeling

• NoSQL

• Big data

• Graph

• Real time analytics

• In-memory computing

• Column stores

One size (RDBMS) no longer fits allPolyglot Persistence

B-trees and Transactions

LOG

DATA 64KB blocks w 8x8KB pagesLogical BTREE of 8kb data pagesIn the buffer pool (cache)

BufferManager

Transactions append inserted, deleted, original and modified pages to the LOG

• Fill factor• Page splits• Clustered index• Checkpoint

When?

• Whenever data fits in RAM• Alternative to general RDBMS OLAP/OLTP• Complex models and transactions• In-memory analytics• Traceability requirements (complete history of events)

Core framework types

Start your engines!

var engine = Engine.LoadOrCreate<SalesModel>();

// lambda queryvar customer = engine.Execute(db => db.Customers.GetById(42));

//Command objectvar cmd = new PlaceOrderCommand(customer.Id, newOrder);engine.Execute(cmd);engine.Close();

• 2 Core, 8GB RAM Cloud VM• IIS Web, SPA, ajax, ASP.NET MVC3• OrigoDB Server same host, currently 4GB process memory• 3k blogs, 500k posts, 500k keywords, 33 million links• Journal 1 GB, no snapshots, growth ~ 8MB/day

http://geekstream.devrexlabs.com

• Environmental news referral since 1997• 100 articles/day, fulltext search, summaries, categories, sources,

metadata• 5000 subscribers, clients, users, billing• Email history, weekly newsletters• < 4 GB Total RAM

Modeling

Creating a data model

• Domain specific vs. Generic vs. Hybrid• Rich vs. Anemic• Explicit vs. Implicit commands• References vs. Foreign keys

Computational model types

• Interpreter hosting – Javascript, Roslyn• Lucene index• Machine learning models (Accord.NET)

Object oriented domain modeling

• DDD?• Entities and collections• Choose collections wisely – space/time tradeoffs• Transaction script pattern or thin commands• Avoid CRUD

Silly Relational

[Serializable] public class RelationalModel : Model { private DataSet _dataset;

public RelationalModel() { _dataset = new DataSet(); } //... ExecuteQuery and ExecuteCommand omitted }

Generic Relational

[Serializable] public class RelationalModel : Model { Dictionary<string,SortedDictionary<object>> _tables; }

Domain specific relational

[Serializable] public class Model : Model { SortedDictionary<int,Customer> _customers; SortedDictionary<int,Order> _orders; SortedDictionary<int,Product> _products; SortedDictionary<string,Customer> _customersByName; }

[Serializable] public class Order { public int CustomerId; //foreign key vs. reference }

JS interpreter hosting (V8)

[Serializable] public class JurassicModel : Model { private ScriptEngine _scriptEngine;

public JurassicModel() { _scriptEngine = new ScriptEngine(); _scriptEngine.Execute("var model = {}"); } //... ExecuteQuery and ExecuteCommand omitted }

Commands

• Serial execution• Exclusive access to the model• Transition the model from one valid state to the next

s0 s1 s2t1 t2

Command guidelines

• No side effects or external actions• No external dependencies• Unhandled exceptions trigger rollback (full restore)• Call Command.Abort() to signal exception

The model is an object graph

TaskListTask

Task

Task

TaskList

Task

Task

Task

Task

Category

Category

Category

Category

TaskModel

Query alternatives

• Ad-hoc lambda: Engine.Execute(Func<M,R> query)• Derive from Query<M,R>• Compiled LINQ: Engine.Execute<R>(string, args)

Query guidelines

• Know thy object graphs• Don’t modify the model

Demo: HockeySkolan

• ASP.NET MVC 3 with backing OrigoDB• Domain specific model + CmsModel• Anemic model => fat commands, model is data

OrigoDB WorkshopModule 3 - Testing

Automated Test alternatives

• Transparent testing of domain behavior: entities, model• Test commands, queries, entities on model without engine• Test with in-memory storage• Full stack testing – slow, requires cleanup

In-memory storage

• Non persistent command journal and snapshots• Mimics FileStore using MemoryStreams• Tests serialization/identity issues

var config = EngineConfiguration.Create().ForIsolatedTest();

OR: config.SetCommandStoreFactory(cfg => new InMemoryCommandStore(cfg));

config.SetSnapshotStoreFactory(cfg => new InMemorySnapshotStore(cfg));

Demo: RedisModel

• Testing model behavior with NUnit

OrigoDB WorkshopModule 4 – Hosting

Direct in-process engine creation

• Static Engine methods• Create()• LoadOrCreate()• Load()

• Returns: Engine, Engine<M>

Engine<MyModel> engine = Engine.LoadOrCreate<MyModel>();

Engine.For<M>()

• Returns IEngine<M>() or derivative• Reuse based on EngineConfiguration.Location property• Remote or in-process• ILocalEngineClient<M>• IRemoteEngineClient<M>

• Running engines are tracked by Config.Engines

Db.For<M>()

• Returns a proxy for M• Remote or Local analogous to Engine.For<M>

x64 vs. x32

• Core Library compiled with AnyCPU• x32 = 32-bit pointers, max 4GB• x64 = 64-bit pointers• Server ships with x64 and x32 binaries

IIS Hosting

• Disable application pool recycling• Ensure single process, no farming or LB• Litter controllers with Db.For<M>() / Engine.For<M>()• Or put a static ref somewhere, eg Global.asax

Proxy

• Proxy has same interface as the Model• Method calls are intercepted• void methods interpreted as commands (and logged)• other methods interpreted as queries• Can be overriden with attributes• Local or remote, cluster

Demo: Let’s build a key/value store

• New project• Reference origodb.core• Define KeyValueStoreModel• void Put(key, value)• object Get(key,value)• bool Exists(key, value)• bool Remove(key,value)

• Add some unit tests

OrigoDB WorkshopModule 5 – Configuration

EngineConfiguration class – key properties• AsyncJournaling• EnsureSafeResults• Kernel• Location• PacketOptions• PersistenceMode• SnapshotBehavior

EngineConfiguration.Location property• File location for FileStorage• Connection string when SqlStorage• Defaults (when null)• Will look in app.config for connection string• Type name of model• Current working directory• App_Data in web context

• Magic• mode=remote;host=10.0.0.20;port=3001

Persistence modes

• Journaling (default)• SnapshotPerTransaction• ManualSnapshots

var config = EngineConfiguration.Create();

config.PersistenceMode = PersistenceMode.SnapshotPerTransaction;

var db = Engine.For<MyModel>(config);

Kernels

• OptimisticKernel (default)• RoyalFoodTaster

var config = EngineConfiguration.Create();

config.Kernel = Kernels.RoyalFoodTaster;

var db = Db.For<MyModel>(config);

Logging

• Logging disabled by default

//enable loggingConsoleLogger.MinimumLevel = LogLevel.Trace;

//Plugin custom loggerLogProvider.SetFactory(new Log4NetLoggerFactory());

Extensibility

• EngineConfiguration methods• SetAuthorizerFactory()• SetSynchronizerFactory()• SetCommandStoreFactory()• SetFormatterFactory()

ProtobufFormatter

• Protocol Buffers by Google• IFormatter wrapper around protobuf-net by @marcgravell• Contract based as opposed to embedded metadata• Compact, fast, loose coupling, cross platform• Configure with attributes or code • Use it!

Protobuf: Attribute based configuration [ProtoContract]    public class Company    {       [ProtoMember(1)]

       public string Name { get; set; }

       [ProtoMember(2)]

       public List<Employee> Employees { get; set; }    }

Protobuf: Code-based configuration

var typeModel = TypeModel.Create();

typeModel.Add(typeof (Company), false)

      .Add(1, ”Name")

      .Add(2, ”Employees");

Inject formatter factory

//use helper methods

ProtoBufFormatter.ConfigureSnapshots<MyModel>(config, typeModel);

ProtoBufFormatter.Configure(config, FormatterUsage.Results, typeModel);

//or do it yourself

config.SetFormatterFactory((cfg,fu)

=> new ProtoBufFormatter<MyModel>(typeModel),

FormatterUsage.Snapshot);

OrigoDB WorkshopModule 6 – Immutability

Immutability and blockingwriter

reader

States share immutable objects

s0

s3s1s2

(Animated slide)

Exampleimmutablemodel

Exampleimmutableentity

Immutable command example

Configuration

OrigoDB WorkshopModule 6 – Server

OrigoDB Server

• Console Application or Windows Service• Process hosting single Engine / Model• Ad-hoc Linq / Razor queries• Javascript API• Primitive web based UI• Commercial License• Multiserver replication

Transparent client migration

• Configuration strings:• mode=embedded• mode=remote;host=10.10.10.10;port=3001

Lab M6

• Follow the OrigoDB Server online tutorialhttp://origodb.com/

Thank you for listening!

• http://origodb.com• http://dev.origodb.com• http://github.com/devrexlabs• http://geekstream.devrexlabs.com• @robertfriberg, @devrexlabs