Deploying (micro)services with Disnix

Deploying (micro)services with Disnix

Sander van der Burg

Nov 14, 2015

Sander van der Burg Deploying (micro)services with Disnix

Microservice architectures

Microservice architectures have become quite popular:


Microservice architectures

There are even books available:


Microservice architectures: What are they?

In short, the microservice architectural style is anapproach to developing a single application as a suite ofsmall services, each running in its own process andcommunicating with lightweight mechanisms, often anHTTP resource API. These services are built aroundbusiness capabilities and independently deployable byfully automated deployment machinery. There is a bareminimum of centralized management of these services,which may be written in different programming languagesand use different data storage technologies.

– Martin Fowler


Microservice architectures: What are they?

In short, the microservice architectural style is anapproach to developing a single application as a suite ofsmall services, each running in its own process andcommunicating with lightweight mechanisms, often anHTTP resource API. These services are built aroundbusiness capabilities and independently deployable byfully automated deployment machinery. There is a bareminimum of centralized management of these services,which may be written in different programming languagesand use different data storage technologies.

– Martin Fowler


Microservice architectures: are they a revolution?




Service oriented architectures

Services are autonomous, platform-independententities that can be described, published, discov-ered, and loosely coupled in novel ways. They per-form functions that range from answering simplerequests to executing sophisticated business pro-cesses requiring peer-to-peer relationships amongmultiple layers of service consumers and providers.Any piece of code and any application componentdeployed on a system can be reused and trans-formed into a network-available service.


“A software component is a unit of composition withcontractually specified interfaces and explicit contextdependencies only. A software component can bedeployed independently and is subject to composition bythird parties.”


Let’s implement an example system!


Let’s implement an example system!


Stafftracker

Let’s maintain a collection of staff members and theirrooms

From the room number I can determine the zipcode

From the zipcode I can determine the street and city

Divide the system into processes communicatingthrough “light weight” network protocols (HTTP,TCP, ...)

Designing an architecture




Pros

Strong Module Boundaries:

You can build teams around each service

Independent Deployment:

Less likely that a failure crashes the whole systemA small component is typically easier to deploy (e.g.fewer dependencies)

Technology Diversity:

One service can be implemented in JavaScript,another in Python etc.Any storage means possible (e.g. databases)We can use many operating systems, hardwarearchitectures etc.

(source: http://martinfowler.com/microservices)

http://martinfowler.com/microservices



Cons

Distribution:

Network links are slow and subject to failure

Eventual Consistency:

Difficult to guarantee that data remains consistent

Operational Complexity:

...

(source: http://martinfowler.com/microservices)

http://martinfowler.com/microservices

Operational Complexity

All services can be deployed to one machine, but each service canalso run on a dedicated machine:

Worst case: 8 machines need to be managed



Diverse technology imposes many kinds of deployment procedures:

Deploying a Node.js package is different than deploying aPython package

Different operating systems, different dependencies, manyvariants



How to update the deployment frequently?

How not to break the system while upgrading?

How to minimize downtimes?

How to roll back in case of a failure?


The Nix project

Automated deployment using declarative specifications with thefollowing properties:

Generic. Can be used with many programming languages,component technologies, and operating systems.

Reproducible. (Almost) no impurities – if inputs are the same,result should be the same regardless of its location

Reliable. Dependency completeness, (almost) atomicupgrades and rollbacks.

Efficient. Only the required deployment activities areexecuted.


The Nix project

Automated deployment using declarative specifications with thefollowing properties:

Generic. Can be used with many programming languages,component technologies, and operating systems.

Reproducible. (Almost) no impurities – if inputs are the same,result should be the same regardless of its location

Reliable. Dependency completeness, (almost) atomicupgrades and rollbacks.

Efficient. Only the required deployment activities areexecuted.


The Nix project: Tools

Nix – declarative specification of package build procedures(including their dependencies)

NixOS – declarative specification of system aspects

Hydra – declarative specification of jobs

NixOps – declarative specification of a network of machines

Deployment activities are implicit – a user writes or adapts a deploy-ment specification, and the tools will carry out the required activities,such as building, transfering, activating, and deactivating.


Deploying (micro)services

How to deploy (micro)service(-oriented) systems the “Nix-way”?


Disnix

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix


Disnix: Modeling a service build

{stdenv, inetutils}:

let

makeFlags = "PREFIX=$out "+

"inetutils=${inetutils}";

in

stdenv.mkDerivation {

name = "hello-world-client";

src = ../../../services/hello-world-client;

buildPhase = "make ${makeFlags}";

installPhase = "make ${makeFlags} install";

}

Services are specified in a similar way as “ordinary” Nix packages:

Define a function taking local (intra-dependencies) as aparameters

Body describes how to build package from source and itsdependencies





{stdenv, inetutils}:

{hello world server}:

let

makeFlags = "PREFIX=$out "+

"helloWorldHostname=${hello world server.target.hostname} "+

"helloWorldPort=${toString (hello world server.port)} "+

"inetutils=${inetutils}";

in

stdenv.mkDerivation {

name = "hello-world-client";

src = ../../../services/hello-world-client;

buildPhase = "make ${makeFlags}";

installPhase = "make ${makeFlags} install";

}

Disnix expressions also take inter-dependencies into account


Composing services locally

{system, pkgs}:

let

callPackage = pkgs.lib.callPackageWith (pkgs // self);

self = {

hello_world_server = callPackage ../pkgs/hello-world-server { };

hello_world_client = callPackage ../pkgs/hello-world-client { };

};

in

self

As with ordinary Nix packages, we need to compose their intra-dependencies by providing them as function parameters.


Services model

{system, distribution, invDistribution, pkgs}:

let

# Imports the intra-dependency composition shown previously

customPkgs = import ../top-level/all-packages.nix { inherit system pkgs; };

in rec {

hello_world_server = rec {

name = "hello_world_server";

pkg = customPkgs.hello_world_server { inherit port; };

port = 3000;

type = "wrapper"; # Specifies how to activate and deactivate a service

};

hello_world_client = {

name = "hello_world_client";

# Refers to the intra-dependency closure of the client

pkg = customPkgs.hello_world_client;

dependsOn = { # Inter-dependency composition

inherit hello_world_server;

};

type = "echo";

};

}


Infrastructure model

{

test1 = { # x86 Linux machine (Kubuntu) reachable with SSH

hostname = "192.168.56.101";

system = "i686-linux";

targetProperty = "hostname";

clientInterface = "disnix-ssh-client";

};

test2 = { # x86-64 Linux machine (NixOS) reachable with SOAP/HTTP

hostname = "192.168.56.102";

system = "x86_64-linux";

targetEPR = http://192.168.56.102:8080/DisnixWebService/...;

targetProperty = "targetEPR";

clientInterface = "disnix-soap-client";

};

test3 = { # x86-64 Windows machine (Windows 7) reachable with SSH

hostname = "192.168.56.103";

system = "x86_64-cygwin";

targetProperty = "hostname";

clientInterface = "disnix-ssh-client";

};

}


Distribution model

{infrastructure}:

{

hello_world_server = [ infrastructure.test2 ];

hello_world_client = [ infrastructure.test1 ];

}

Maps services in the services model to target machines in the infras-tructure model


Demo


Demo

Initial deployment:


Adding a second client (by adding test2):


Moving the server from test2 to test3 and building on targets:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix \--build-on-targets


Disnix: Communication flow

disnix-env communicates with target machines through a customconnection client to remotely execute deployment activities:

SSH (the default)

SOAP/HTTP through DisnixWebService(http://github.com/svanderburg/DisnixWebService)

NixOps through DisnixOS(http://github.com/svanderburg/disnixos)


http://github.com/svanderburg/DisnixWebService

http://github.com/svanderburg/disnixos

Disnix: Communication flow

Two deployment tools are invoked to carry out remote deploymentactivities:

Nix (http://nixos.org/nix): package manager responsiblefor building, distributing

Dysnomia (http://github.com/svanderburg/dysnomia):plugin system responsible for activating, deactivating, locking,snapshotting, restoring


http://nixos.org/nix

http://github.com/svanderburg/dysnomia

Disnix: Process decomposition of disnix-env

Each deployment activity is implemented by a command-line tool.


Disnix: Process decomposition of disnix-env

Two additional tools are used when building on targets has beenenabled.


How can we deploy our example system?


How can we deploy our example system?


Implementation details

Each database: a MongoDB database instance

Each service: a process with REST API implementedin Node.js

Web application front-end: A web applicationimplemented in Node.js

nginx proxy

Infrastructure deployment

Before we can use Disnix, we need machines first! With thefollowing characteristics:

A Disnix service instance for executing deployment stepsremotely

MongoDB DBMS

Machines can be deployed with various kinds of (external)solutions (including: Disnix itself).


Infrastructure deployment

NixOps can be used to automatically instantiate and deploy anetwork of NixOS machines having the required characteristics:

{

test1 = {

{ pkgs, ...}:

{

networking.firewall.enable = false;

# Enable the Disnix service

services.disnix.enable = true;

# Enabling MongoDB also configures Disnix

# to use the Dysnomia plugin that can activate it

services.mongodb.enable = true;

services.mongodb.bind_ip = "0.0.0.0";

}

test2 = ...

test3 = ...

}


Integrating service and infrastructure deployment

Disnix’s service deployment and NixOps’ infrastructure deploymentfacilities can be combined with an extension toolset called:DisnixOS.

Major difference with the basic toolset: DisnixOS usesnetworked NixOS configuration files instead of aninfrastructure model


Demo


Demo

Deploying a network of NixOS machines with NixOps:

$ nixops create ./network.nix ./network-virtualbox.nix -d test$ nixops deploy -d test

Deploying services to machines with DisnixOS:

$ export NIXOPS DEPLOYMENT=test$ disnixos-env -s services.nix \-n network.nix -n network-virtualbox.nix \-d distribution.nix --use-nixops


A real world example: Conference Compass



Conference Compass provides a service to improve the waypeople experience events

Most visible part of the service: apps for conference attendeesEach customer basically gets “their own” app.

We have a product-line using a Nix-based build infrastructure,including Hydra



The app’s contents is customizable with a configurator serviceallowing organizers to create and update their content

Apps connect to a configurator to retrieve the data to bedisplayed and other configuration settings

Integration with third party information systems is alsopossible


Architectural decisions

Each app connects to its own dedicated configurator service:

No single point of failureMore scalable – slow configurator of one app does not affectanotherMore flexible – we can easily move configurators aroundMultiple version support – we can add new features for a newapp, without breaking old appsImplemented in Node.js

Each third party integration (called: channel) is a seperateprogram:

They may consume a lot of system resourcesImplemented in Python, because we find that more appropriatefor data conversion

Processes communicate through REST/HTTP APIs


Deploying services

We deploy four kinds of services with Disnix:

Configurators (Node.js programs)

Channels (Python programs)

Mongo databases (one for each configurator)

nginx proxies (one for each machine)

Responsible for forwarding requests to configurators andcaching expensive requests


Deploying services

A simple deployment scenario:

(Diagram generated with disnix-visualize)


Deploying services

We might want to move services to more powerful machines whenan event goes live (because more users of an app imply that moresystem resources are needed, e.g. network bandwidth):


Deploying services

For really big events, we may want to deploy redundant instancesconfigurators and channels to handle all the load:


Some statistics

We have been using Disnix + NixOps for deploying ourproduction environment since January 2015

Between 6-11 Amazon EC2 machines are managed by NixOps

±80 configurators have been deployed with Disnix includingtheir inter-dependencies

Translates to ±200 services managed by Disnix

Production environment is frequently updated. Sometimesthree times a day.

Transition phase duration is ±5 minutes in worst case. Mostsimple upgrades only bring a few seconds of downtime.


Conclusions

I have explained the relevance of componentized distributedsystem architectures (e.g. microservices)

I have explained how to do service deployment with Disnix

I have explained how to integrate infrastructure deploymentand service deployment

I have shown a real-life usage scenario


References

There is much more you can do with Disnix! This presentationonly covers a few basic aspects!

Disnix homepage: http://nixos.org/disnix

TCP proxy example: https://github.com/svanderburg/disnix-proxy-example

StaffTracker examples:

Java/WebServices/MySQL version: https://github.com/svanderburg/disnix-stafftracker-java-examplePHP/MySQL version: https://github.com/svanderburg/disnix-stafftracker-php-exampleNode.js/MongoDB version: https://github.com/svanderburg/disnix-stafftracker-nodejs-example

Disnix should be considered an advanced prototype tool!


http://nixos.org/disnix

https://github.com/svanderburg/disnix-proxy-example

https://github.com/svanderburg/disnix-proxy-example

https://github.com/svanderburg/disnix-stafftracker-java-example

https://github.com/svanderburg/disnix-stafftracker-java-example

https://github.com/svanderburg/disnix-stafftracker-php-example

https://github.com/svanderburg/disnix-stafftracker-php-example

https://github.com/svanderburg/disnix-stafftracker-nodejs-example

https://github.com/svanderburg/disnix-stafftracker-nodejs-example

References


Questions


Software

Deploying (micro)services with Disnix