Apache Mesos Distributed Computing Talk

Apache MesosA romantic stroll into Distributed Computing

Brandon Gulla - Praxis Engineering

●Beer.●Why distributed-computing is hard? ***●How Mesos helps.●Mesos in the wild.●DCOS-Demo.●Questions?●Repeat Step 1

This is a shortened version of the talk I normally give, feel free to contact me offline for a deeper dive.

Agenda

$ > whoamiBrandon GullaDistributed DevOps Engineerblgulla@praxiseng.com github/bgulla

Mesosphere

● Commercial company supporting Apache Mesos based in SF and Hamburg, Germany.

● Founded by the Apache Mesos Founder Ben Hindman, Former AirBnB CTO Tobi Knapp and previous Twitter Engineer Florian Leibert.

● Provides full stack Mesos solutions running Apache Mesos as well as their corporate product: DCOS (Datacenter Operating System).

● Keith McClellan will be presenting a demo as soon as you’re tired of me.

MesosFrameworks

●Servers, Memory and Disks have gotten cheaper so we buy more. These often go under-utilized.

●Traditionally, clusters have been provisioned statically.

●Puppet, SaltStack, Ansible, **pdsh** :(●Not Multi-tenant friendly.●DevOps Nightmare!!!●Doesn’t Scale :(

“Just Network them together!”

Personal Gateway To Mesos

●Customer had a 100+ node Hadoop cluster with legacy dependency requirements.

●Wanted to utilize current MapReduce libraries without breaking ingest requirements.

●More importantly: Customer wanted a way to manage their resources on demand...

Solution:

●Open-source two-level Resource Manager.

●Written in C++ wrapping around libprocess.

●Scalable - Runs on 10,000s of servers.●Fault-Tolerant! Multiple masters. Load

Balancing!●Solve Bigger Problems.●I/O Isolation via Linux cgroups.●Distributed Resource Manager:

●RAM●Disks●Ports●GPUs●Network (Yes- Software Defined Networking)

Welcome to Apache Mesos

Mesos: Turn Your Datacenter into One Large Computer● Where did we install MySQL? Who cares! Mesos +

Service Discovery = Dynamic Awesomeness. ● Separates dependencies between Ops and

Development. Ops wants to take a rack down, who cares, just move the processes from that rack somewhere else. Ops can do this as part of their work. Developers shouldn't know/care.

● Fault Tolerant, if something dies, Mesos will automatically spin another one up.

History

Shared Resources = Multi-Tenancy

Mesos Explained

Resource Offers and Launching A Task

Writing your own Mesos Framework

●Feature Complete API using:●JVM Languages (Java, Scala, Groovy)●C++●Python●Go

●Write a framework in minutes using Mesos-Compose.

●YML interface ●Similar to Docker-Compose

●Unit Testing with Mini-Mesos

Don’t want to write your own Mesos Framework? No problemo, Amigo.

0 to Mesos in seconds with Marathon.

Containers: VMs but awesomer.●Shares Kernel with Host (no hypervisor)●Boots in milliseconds.●Portable. docker pull python:27

●Run CentOS on Debian, Run Gentoo on Alpine.

●Dependency Friendly

●If you’re replacing a once statically installed, non-distributed application: Use Marathon.

●Apache, Nginx, Jenkins..etc

●If your application requires fine grain control and orchestration of tasks: Write your own Framework.

●Apache Accumulo●Hadoop Yarn (Shameless Apache Myriad Plug)●Spark

Framework Development: My Experiences

Mesos in the Wild●Netflix: Mantis+Fenzo●Apple: Jarvis (Siri)●Microsoft Azure Container Service.

Mesos: Full Stack Replacement

Current Mesos Equivalent

Systemd Marathon

Cron Chronos

DNS Mesos-DNS

Chroot / BSD Jail Docker

NFS Gluster, CEF, HDFS

Redefined ThinkingBefore:

After:

Processes

MesosFrameworks

Demo TimeTo Infinity and Beyond.