Embed Size (px)
Citation preview
Habitat 301: Building Habitats
Chef Conf 2016
Jamie Winsor
@resetexistence
github.com/resetChef Conf 2016
NEW SLIDE HERE
Chef Conf 2016
Chef Conf 2016
Great for infrastructure, Good for applications
Chef Conf 2016
Application Automation Principles4 Automation Needs To Live With Your Application
4 Build Failures over Runtime Failures
4 Choreography over Orchestration
Chef Conf 2016
Atomic, Isolated, Repeatable & Auditable
Chef Conf 2016
Chef Conf 2016
BuilderSaaS To Build Your Packages
Chef Conf 2016
It's already live!4 Web - https://app.habitat.sh
4 API - https://app.habitat.sh/v1
Chef Conf 2016
Why build Builder?4 Public hosting for packages
4 Public builders for packages
4 History of where a package came from
Chef Conf 2016
Automatic Re-Build/Publish On Dependency
Update
Chef Conf 2016
pkg_name=hab-builder-apipkg_origin=corepkg_version=0.7.0pkg_maintainer="Jamie Winsor <[email protected]>"pkg_license=('apachev2')pkg_source=nosuchfile.tar.gzpkg_bin_dirs=(bin)pkg_deps=(core/glibc core/openssl core/coreutils core/gcc-libs core/zeromq core/libsodium core/libarchive)pkg_build_deps=(core/protobuf core/protobuf-rust core/coreutils core/cacerts core/rust core/gcc core/pkg-config core/node core/phantomjs)pkg_expose=(9636)srv_bin="bldr-api"pkg_svc_run="bin/$srv_bin start -c ${pkg_svc_path}/config.toml"
Chef Conf 2016
Building Builder4 Educate about Builder
4 For users
4 For collaborators
4 Show how Habitat helped to build Builder
Chef Conf 2016
It's about to get technical
Chef Conf 2016
Service Oriented Design4 Gateway nodes
4 Router nodes
4 Service nodes
Chef Conf 2016
Service Layout
Chef Conf 2016
Why Rust?
4 Systems programming language
4 Fast
4 Memory safety without a virtual machine
4 Thread safety guarantees
4 Great tooling
4 Basically a modern C
Chef Conf 2016
Walkthrough: Starting a new build
Chef Conf 2016
Hop 1/10: Client
4 Web Client
4 https://app.habitat.sh
4 Angular 2
4 HTTP Client
4 REST
4 application/json
Chef Conf 2016
Hop 2/10: Gateway
4 Public facing edge node
4 External authentication/identification by OAuth/Authorization header
4 Transform client-request into net-request & forward to router
4 Await reply if transactional and send reply to web-client
Chef Conf 2016
Authorization
4 Read value of Authorization header into protocol message
4 Forward protocol message to any RouteSrv
4 RouteSrv forward request to appropriate SessionSrv
Chef Conf 2016
Builder-Protocol/NetProtobuf & ZeroMQ Sockets
Chef Conf 2016
Protobuf4 Language agnostic DSL
4 Backwards compatibility between service versions
4 Composable, modular messages
message Person { required string name = 1; required int32 id = 2; optional string email = 3;}
Chef Conf 2016
ZeroMQ4 Socket library for building distributed services
4 Not a message queue/message bus
4 Makes no assumptions for your protocol
4 Guarantees for multi-part message delivery
Chef Conf 2016
components/builder-protocol/protocols/net.protomessage Msg { // hint for decoders required string message_id = 1;
// serialized msg body required bytes body = 2;
// used by RouteSrv to know where to send msg optional RouteInfo route_info = 3;}
Chef Conf 2016
Authorization (2)// components/builder-protocol/protocols/sessionsrv.protomessage SessionGet { required string token = 1;}
4 Create new SessionGet msg
4 Write value of Authorization header into token field
4 Forward SessionGet message to any RouteSrv
Chef Conf 2016
Hop 3/10: RouteSrv// components/builder-protocol/protocols/net.protomessage RouteInfo { // Determines destination service required Protocol protocol = 1;
// Determines destination shard optional uint64 hash = 2;}
enum Protocol { Net = 0; RouteSrv = 1; SessionSrv = 2; VaultSrv = 3; JobSrv = 4;}
4 Read RouteInfo and forward to appropriate service
Chef Conf 2016
Hop 4/10: SessionSrv1. Receive message from
RouteSrv
2. Dispatch SessionGet to handler
3. Handler reads token field of message and looks in datastore for session
Chef Conf 2016
Replying to Request
4 On Success: Session
4 On Failure: NetError
message Session { required uint64 id = 1; required string email = 2; required string name = 3; required string token = 4;}
Chef Conf 2016
NetError: Transaction Failuresmessage NetError { required ErrCode code = 1; required string msg = 2;}
enum ErrCode { // other codes... SESSION_EXPIRED = 8; // more codes...}
Chef Conf 2016
Dissecting The MessageCode: 8, Msg: rg:auth:1
4 Code: for the user
4 Msg: for the developer
4 service: rg
4 operation/function: auth
4 case: 1Chef Conf 2016
Hop 5/10: RouteSrv
4 Receive transaction reply from SessionSrv
4 Send response to appropriate HTTP-Gateway
Chef Conf 2016
Hop 6a/10: Gateway
1. Receive reply from RouteSrv
2. Forward reply to client thread
3. Client thread creates ProjectGet message from HTTP params
Chef Conf 2016
Hop 6b/10: Gateway
// protocols/vaultsrv.protomessage ProjectGet { required uint64 id = 1;}
4 Send ProjectGet through RouteSrv to VaultSrv
Chef Conf 2016
Actual Service Cluster
Chef Conf 2016
Chef Conf 2016
Builder Is Service Oriented & Sharded4 Service goes down: partial outage for all users
4 Shard goes down: partial outage of service for subset of users
4 Easily add new machines by re-balancing shards
Chef Conf 2016
Message Routing (cont.)// components/builder-protocol/protocols/net.protomessage RouteInfo { // Determines destination service required Protocol protocol = 1;
// Determines destination shard optional uint64 hash = 2;}
4 Populate protocol with message's protocol
4 Populate hash with a message's RouteKeyChef Conf 2016
Defining The RouteKey
Chef Conf 2016
Routable/// Defines a contract for protocol messages to be routed through `RouteSrv`.pub trait Routable: protobuf::Message + protobuf::MessageStatic { /// Type of the route key type H: RouteKey + fmt::Display;
/// Return a `RouteKey` for `RouteSrv` to know which key's /// value to route on. /// /// If `Some(T)`, the message will be routed by hashing the /// value of the route key and modding it against the shard /// count. This is known as "randomly deterministic routing". /// /// If `None`, the message will be randomly routed to an available node. fn route_key(&self) -> Option<Self::H>;}
Chef Conf 2016
Protobuf Definitionmessage ProjectGet { required uint64 id = 1;}
Trait Implementationimpl Routable for ProjectGet { type H = InstaId;
fn route_key(&self) -> Option<Self::H> { Some(InstaId(self.get_id())) }}
Chef Conf 2016
InstaId4 64-bit integer containing
4 create-time
4 sequence-id (generated by database, 0-1024)
4 shard (0-127)
4 Inspired by Instagram engineering1
1 http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
Chef Conf 2016
pub const EPOCH_MS: u64 = 1460499133628;pub const SHARD_COUNT: u32 = 128;
pub struct InstaId(pub u64);
impl InstaId { pub fn generate(auto_id: u64) -> Self { let time = Self::since_epoch(); let id = auto_id % 1024; let shard_id = id % SHARD_COUNT as u64; let mut iid: u64 = time << 23; iid |= id << 13; iid |= shard_id; InstaId(iid) }
pub fn since_epoch() -> u64 { let timespec = time::get_time(); let sec: u64 = timespec.sec as u64 * 1000; let nsec: u64 = timespec.nsec as u64 / 1000 / 1000; (sec + nsec) - EPOCH_MS }}
Chef Conf 2016
Hop 7/10: VaultSrv
1. Receive message from RouteSrv
2. Dispatch ProjectGet to handler
3. Read Project from database
4. Reply to HTTP-Gateway through RouteSrv
Chef Conf 2016
What is VaultSrv?
4 Persistent storage for
4 Origin memberships
4 Origin invitations
4 Projects
Chef Conf 2016
Persisting Protobufs Into The Database
Chef Conf 2016
Persistable/// Defines a contract for protocol messages to be persisted to a datastore.pub trait Persistable: protobuf::Message + protobuf::MessageStatic { /// Type of the primary key type Key: fmt::Display;
/// Returns the value of the primary key. fn primary_key(&self) -> Self::Key;
/// Sets the primary key to the given value. fn set_primary_key(&mut self, value: Self::Key) -> ();}
Chef Conf 2016
Protobuf Definitionmessage Project { required uint64 id = 1; required uint64 owner_id = 2; required uint64 origin_id = 3; required string plan_path = 4; oneof vcs { VCSGit git = 5; }}
Trait Implementationimpl Persistable for Project { type Key = u64;
fn primary_key(&self) -> Self::Key { self.get_id() }
fn set_primary_key(&mut self, value: Self::Key) { self.set_id(value); }}
Chef Conf 2016
Hop 8/10: REST-Gateway1. Receive reply from RouteSrv
2. Create new JobSpec from Session & Project
3. Send JobSpec through RouteSrv to JobSrv
message JobSpec { required uint64 owner_id = 1; required vault.Project project = 2;}
Chef Conf 2016
Hop 9/10: JobSrv
1. Receive message from RouteSrv
2. Dispatch JobSpec to job_create handler
3. Create Job from JobSpec
4. Persist Job to database/queue
Chef Conf 2016
What is JobSrv?
4 Store job history
4 Job is an entity created by a project
4 Active jobs are placed in a queue
4 Different qeueues for different OSes
4 Job create success/failure
Chef Conf 2016
Hop 9b/10: JobSrv
1. Send Job through RouteSrv to HTTP-Gateway
message Job { required uint64 id = 1; required uint64 owner_id = 2; required JobState state = 3; required vault.Project project = 4;}
enum JobState { Pending = 0; Processing = 1; Complete = 2; Rejected = 3; Failed = 4;}
Chef Conf 2016
Hop 10/10: REST-Gateway1. Receive reply from RouteSrv
2. Forward reply to client thread
3. Client thread serialize Job into JSON
4. Send 200 to client with encoded Job
Chef Conf 2016
JSON Trait Implementationimpl ToJson for Job { fn to_json(&self) -> Json { let mut m = BTreeMap::new(); m.insert("id".to_string(), self.get_id().to_json()); m.insert("state".to_string(), self.get_state().value().to_json()); Json::Object(m) }}
JSON Response{ "id": 62214184531664897, "state": 0,}
Chef Conf 2016
Worker Pool 11/10
Chef Conf 2016
Chef Conf 2016
Workers4 (N) workers connected to (X) JobSrv
4 OS specific
4 Pops a message off a JobSrv queue
4 Builds are done within a studio on the worker
4 Finished builds published to public depot as project's package ident
Chef Conf 2016
Habitat Builder Is Self Hosted4 Habitat supervises all processes
4 Supervisor watches public depot (self) for package updates
4 Builder builds it's own updates
4 Automatically updates when a build is published to public depot
Chef Conf 2016
Chef Conf 2016
Chef Conf 2016
Where Is config/default.toml?
Chef Conf 2016
Builder Uses Native Habitat Config[cfg]http_addr = "0.0.0.0:9636"
[cfg.github]client_id = "{clientid}"client_secret = "{clientsecret}"
[[bind.router.members]]alive = trueip = "10.0.0.190"port = "5562"service = "hab-builder-router"
Chef Conf 2016
Director
Chef Conf 2016
Director systemd[Unit]Description=Habitat Director
[Service]ExecStart=/bin/hab-director start -c /hab/etc/director/config.toml
[Install]WantedBy=default.target
Chef Conf 2016
Director Config[cfg.services.core.redis.${env}]start = "--permanent-peer --peer ${peer_ip}:9000"
[cfg.services.core.hab-builder-api.${env}]start = "--permanent-peer --bind database:redis.${env},router:hab-builder-router.${env} --peer ${peer_ip}:9000"
Chef Conf 2016
Bind Is Service Discovery
Chef Conf 2016
Shard AllocationChef Conf 2016
Electing A Master Router4 Automatically elected and configured by Habitat
through the --topology leader flag
4 Configured to expect (N) service nodes
4 Gives out shard assignments on service connect
Chef Conf 2016
Bring Your Own Worker
Chef Conf 2016
Habitat Hack DayThursday, 10AM-3PM
Chef Conf 2016
