www.linuxfoundation.org

Guide to the Open Cloud Current trends and open source projects

November 2016

AUTHORS

Libby Clark and Mark HinkleThe Linux Foundation

Guide to the Open Cloud: Current trends and open source projects2

ContentsIntroduction 3

What is the Open Cloud Today, Why It’s Important, and How It’s Advancing 5

Emerging Trends 6

Cloud Native Applications 6

Containers 7

Unikernels 7

Why is the Open Cloud Important? 7

Profile Methodology 9

Profiles 11

Infrastructure as a Service (IaaS) 11

Platform as a Service (PaaS) 12

Virtualization, Containers and Cloud Operating Systems 13

Micro or Minimalist OSes 13

Virtualization 14

Management and Automation 15

Unikernels 17

DevOps CI/CD 19

Complete CI/CD cycle 19

Configuration management 20

Logging and Monitoring 21

Software-Defined Networking (SDN) 22

Networking for containers 23

Software-Defined Storage 24

Guide to the Open Cloud: Current trends and open source projects3

IntroductionThis third annual Guide to the Open Cloud aims to help companies stay informed on the latest open source cloud technologies and trends. The Linux Foundation’s cloud experts have curated a list of the most useful, influential, and promising open source projects with which IT managers and practitioners can build, manage, and monitor their current and future mission-critical cloud resources. In doing so, we’re also helping companies identify projects they can best leverage for external R&D through strategic open source participation. Contributing knowledge and code to open source projects not only helps companies meet their business objectives, but it creates thriving communities that keep projects strong and relevant over time, advances the technology, and benefits the entire open source cloud ecosystem.

Cloud computing is the cornerstone of the digital economy. Companies across industries now use the cloud—private, public or somewhere in between—to deliver their products and services. Forty-one percent of all enterprise workloads are currently running in some type of public or private cloud1. That number is expected to rise to 60 percent by mid-2018. And some 95 percent of companies are at least experimenting in the cloud2. Enterprises are continuing to shift workloads to the cloud as their expertise and experience with the technology increases. From banking and finance to automotive and healthcare, companies are facing the reality that they’re now in the technology business. In this new reality, cloud strategies can make or break an organization’s market success. And successful cloud strategies are built on Linux and open source software.

Some of the most successful public companies today are built around cloud-native applications—a fashionable term that simply means they’re designed to run in the cloud. Netflix, Facebook, LinkedIn, Twitter, and Amazon have all leveraged open source components within a distributed, microservices-based architecture to quickly deliver new products and services that are cost-effective and responsive to market demands and changes. By breaking applications up into microservices, or distinct, single-purpose services that are loosely coupled with dependencies and explicitly described through service endpoints, they have significantly increased the overall agility and maintainability of applications and used that to gain competitive advantage.

1 451 Research, “Voice of the Enterprise: Cloud Transformation - Workloads and Key Projects,” Sept. 1, 2016.

2 RightScale, “2016 State of the Cloud Report,” January, 2016.

Guide to the Open Cloud: Current trends and open source projects4

The rest of the market has scrambled to replicate this architecture and approach, cobbling together their own solutions using custom scripts and open source software—often using the open source versions of these web giants’ own infrastructure (ie Google’s borg, which became Kubernetes; Twitter’s Mesos project, VMware’s Cloud Foundry, etc.) This experimentation has set off a chain of innovation with four notable trends, still playing out today:

• Increasing consumption of public cloud resources • Adoption of container technologies like Docker and others (Fifty-three percent of

organizations are either investigating or using containers in development or in production3.) • The rise of DevOps as the most effective method for application delivery in the cloud• An explosion in available open source tooling as user companies like Walmart and Capital

One release their management software under open source licenses.

While deployment and management remain a challenge, microservices architecture is now becoming mainstream. In a recent Nginx survey4 of 1,800 IT professionals, 44 percent said they’re using microservices in development or in production. Adoption was highest among small and medium-sized businesses. Not coincidentally, the use of public cloud is also predominant among SMBs5, which are more nimble and faster to respond to market changes than large enterprises with legacy applications and significant on-premise infrastructure investments.

Many reports tout hybrid cloud as a fast-growing segment of the cloud. Demand is growing at a compound rate of 27 percent, “far outstripping growth of the overall IT market,” according to researcher MarketsandMarkets6. And IDC predicts that more than 80 percent of enterprise IT organizations will commit to hybrid cloud architectures by 20177. However, this growth is happening predominantly among large enterprises with legacy applications and the budget and staffing to build private clouds. They turn to cloud for storage and scale-out capabilities, but keep most critical workloads on premise. In the mid-market, hybrid cloud adoption stands at less than 10 percent, according to 451 Research. Hybrid cloud is, then, a good transition point for legacy workloads and experimenting with cloud implementation. But it suffers from several challenges with more advanced cloud implementations, including management complexity and cost.

“Most organizations are already using a combination of cloud services from different cloud providers. While public cloud usage will continue to increase, the use of private cloud and hosted private cloud services is also expected to increase at least through 2017. The increased use of multiple public cloud providers, plus growth in various types of private cloud services, will create a multi-cloud environment in most enterprises and a need to coordinate cloud usage using hybrid scenarios.

3 Cloud Foundry, ”Hope Vs. Reality: Containers in 2016,” June 2016.

4 Nginx, “The Future of App Development and Delivery,” March 2016.

5 RightScale, “2016 State of the Cloud Report,” January, 2016.

6 Wall Street Journal, “Special Report: CIOs Say Hybrid Cloud Takes Off,” Oct. 2015.

7 IDC, “IDC FutureScape: Worldwide Cloud 2016 Predictions — Mastering the Raw Material of Digital Transformation,”

2016.

Guide to the Open Cloud: Current trends and open source projects5

“Although hybrid cloud scenarios will dominate, there are many challenges that inhibit working hybrid cloud implementations. Organizations that are not planning to use hybrid cloud indicated a number of concerns, including: integration challenges, application incompatibilities, a lack of management tools, a lack of common APIs and a lack of vendor support,” according to Gartner’s 2016 Public Cloud Services worldwide forecast.8

Over the long term, workloads are shifting away from hybrid cloud to a public cloud market dominated by providers like AWS, Azure, and Google Compute. “The share of enterprise workloads moved to the public cloud is expected to triple over the next five years,” from 16 percent to 41.3 percent of workloads running in the public cloud, according to a recent JP Morgan survey of enterprise CIOs9. Among this group, 13 percent said they view AWS as “intrinsic to future growth.” AWS still very much dominates the cloud industry and has continued to grow at an incredible rate. It looks like hosted services are going to be a huge second line of business for the company. Based on its July 2016 earnings release, Amazon’s web business is on pace to be a $9 billion business this year.10

The shift to the public cloud will continue to accelerate as costs come down and security concerns are allayed. Security is still a top concern among companies considering moving workloads to the public cloud, according to Gartner, despite a strong track record of security and increased transparency from cloud providers. Rather, security is still an issue largely due to companies’ inexperience and improper use of cloud services. As companies gain more experience with public cloud use, security concerns will become less of a hindrance.

As cloud adoption grows, open source technologies will continue to be the source of innovation and the foundation for new markets and ecosystems. Each of the projects listed in this report is actively involved in creating the future of IT infrastructure on which companies will deliver their products and services, in the coming year and beyond. Let this report be your guide to become familiar with these projects, the categories of technology in which they are influential, and the ways in which they can help companies remain competitive in this age of digital transformation.

What is the open cloud today, why it’s important, and how it’s advancing Many companies are still building their own cloud stacks—assembling open source components to run a private or hybrid infrastructure. And you’ll find these components on our list, as they were in the past. But the drive is no longer to build enterprise alternatives to large, proprietary public clouds. Instead, more companies are turning to public cloud to run their applications and services, and as they do so the nature of the open cloud today is changing.

8 Gartner, “Forecast Analysis: Public Cloud Services, Worldwide, 2Q16 Update,” Sept., 2016.

9 Business Insider, “CIOs See Momentous Shift in Public Cloud Usage,” April 2016.

10 Amazon, “Q2 2016 Financial Results,” July 2016.

11 Gartner, “Gartner Says Worldwide Public Cloud Services Market to Grow 17 Percent in 2016,” Sept. 2016.

Guide to the Open Cloud: Current trends and open source projects6

As cloud computing continues to evolve, the infrastructure is becoming more flexible and programmable. There is a shift in the data center from finite hosted resources to an elastic compute fabric that is more readily consumed by developers.

By the end of 2016 the public cloud services market will reach $208.6 billion in revenue, growing by 172 percent from $178 billion in 201511. Cloud application services (software-as-a-service or SaaS) is one of the largest segments of that and is expected to grow by 21.7 percent in 2016 to reach $38.9 billion while Infrastructure-as-a-Service (IaaS) is projected to see the most growth at 42.8 percent in 2016.

The public cloud itself is largely built on open source software. Offerings including Amazon EC2, Google Compute Engine and OpenStack are all built on open source technologies. They provide APIs that are well documented. They also provide a framework that is consistent enough to allow users to duplicate their infrastructure from one cloud to another without a significant amount of customization.

This allows for application portability, or the ability to move from one system to another without significant effort. The less complex the application the more likely that it can remain portable across cloud providers. And so the development practice that seems to be most suited for this is to abstract things into their simplest parts.

A whole new class of open source cloud computing projects has now begun to leverage the elasticity of the public cloud and enable applications designed and built to run on it. You’ll find these new projects in our list as well.

EMERGING TRENDS Early on in cloud computing there was a migration of existing applications to Amazon Web Services, Google and Microsoft’s Azure. Virtually any app that ran on hardware in private data centers could be virtualized and deployed to the cloud. Now with a mature cloud market more applications are being written and deployed directly to the cloud and are often referred to as being cloud native.

CLOUD NATIVE APPLICATIONS While there is no textbook definition, the self-described cloud native in its simplest definition indicates applications that have been designed to run on modern distributed systems environments capable of scaling to tens of thousands of nodes. The old mantra, “No one ever got fired for buying IBM (or Microsoft),” has changed to the new slogan of “No one is going to get fired for moving to the cloud.” Rather than looking at hard and fast qualifiers for cloud-native we need to look at the design patterns that are being applied to this evolving breed of applications.

Guide to the Open Cloud: Current trends and open source projects7

In the pre-cloud days we saw virtualization take hold where entire operating systems were portable inside of virtual machines. That way a machine could move from server to server based on its compatibility with hypervisors like VMware, KVM or Xen Project. In recent years we have seen the level of abstraction at the application layer where applications are container-based and run in portable units that are easily moved from server to server regardless of hypervisor due to container technologies like Docker and CoreOS-sponsored rkt (pronounced rocket).

CONTAINERS A more recent addition in the cloud era is the rise of the container, most notably Docker and rkt. These application hosts are an evolution of previous innovations including Linux control groups (cgroups) and LXC, and an even further abstraction to make applications more portable. This allows them to be moved from development environments to production without the need for reconfiguration.

Applications are now deployed either from registries or through continuous deployment systems to containers that are orchestrated using tools like Ansible, Puppet or Chef.

Finally, to scale out these applications, the use of schedulers such as Kubernetes, Docker Swarm, Mesos, and Diego coordinate these containers across machines and nodes.

UNIKERNELS Another emerging technology that bears some similarity to containers are unikernels. Unikernels are a pared down operating system (the unikernel) which are combined with a single application into a unikernel application and are typically run within a virtual machine. They are sometimes called a library operating system because they include libraries that enable applications to use hardware and network protocols in combination with a set of policies for access control and isolation of the network layer. There were systems in the 1990s called Exokernel and Nemesis but today popular unikernels include MirageOS and OSv. Because unikernel applications can be used independently and deployed across diverse environments, unikernels can create highly specialized and isolated services and have become increasingly used for developing applications in a microservices architecture.

WHY IS THE OPEN CLOUD IMPORTANT?In previous years this report has distinguished between open source cloud and others. But as cloud technologies have evolved it’s evident that any cloud without open source would be the equivalent of an automobile without an engine. In 2006 we distinguished heavily between public and private cloud and open source and closed. Today the conversation has evolved into one cloud fabric of which open source has become an integral part.

Guide to the Open Cloud: Current trends and open source projects8

Perhaps what has most notably changed is that the initial cloud conversations about capex (capital expenditures) versus opex (operating expenditures) and the actual costs to deploy the cloud are now taking into account the advantages of improved agility and customization. Where open source has traditionally sparked interest because of its free nature (as in no acquisition cost) is now being lauded for the much harder to measure but much greater benefits of faster speed to value.

We also see an improved return on investment for those companies that participate in open source rather than only consume open source. They need to and are investing in the future direction of the open technology they rely upon actively rather than only being passive and opportunistic.

Besides open source software there is an overwhelming need for openness across cloud architectures. And while emerging technologies and trends such as containers have done a lot to improve interoperability among components and ensure application portability, much work remains to ensure the trend toward openness and standardization continues. To this end, foundations such as the Cloud Foundry Foundation, Cloud Native Computing Foundation (CNCF) and Open Container Initiative (OCI) are actively bringing in new open source projects and engaging member companies to create industry standards for new cloud-native technologies. The goal is to help improve interoperability and create a stable base for container operations on which companies can safely build commercial dependencies.

When work happens in the open, companies that participate are better able to compete in rapidly changing markets and the entire industry benefits from the increased innovation. That also means companies that do not use and participate in open source cloud projects will fall behind. By harnessing the power of shared R&D companies that participate in open source benefit from:

• Improved code quality • Increased security with the ability to find and fix vulnerabilities• Visibility into every layer of the infrastructure• Code access in order to add features and influence the direction of the technology• Insurance against lock-in through portability to other platforms• Lower cost through shared development• And more.

No single company could develop the technologies on this list on their own. Without open source collaboration, the open cloud we know today would not exist.

Guide to the Open Cloud: Current trends and open source projects9

Profile Methodology The Linux Foundation is creating the greatest shared technology asset in history, providing professional open source management to more than 50 industry-wide projects including Cloud Foundry, CNCF and OCI. By engaging with our members in our open source projects and organizing leading technical conferences in North America, Europe and Asia, we help guide many projects, technologies and companies that are driving the cloud. That puts us in a unique position to create this list.

With new projects forming at a steady pace to innovate and define the next generation of cloud-native infrastructure and web application deployment, it can be difficult to keep track of what’s happening in the open source community. To be most helpful, we’ve limited the list to the most important projects in each category including virtualization, containers, and cloud operating systems; Infrastructure as a Service; Platform as a Service; DevOps continuous integration/ continuous deployment tools (CI/CD); software-defined storage; and software-defined networking (SDN).

Projects were selected for the list by The Linux Foundation’s cloud experts based on their relevance, their relative maturity, and their relative visibility. More specifically, benchmarks for consideration included:

• the project’s origins• age of the project• number of contributors • number and frequency of commits• diversity of contributions• exposure• demonstrated enterprise use • expert opinions from IT practitioners

All profile data was collected from public sources, including project websites and source code repositories.

Please note that as the technology is moving at cloud speed, we can’t keep an inclusive directory of all open source cloud projects. But we can offer the notable ones. We have focused on projects with which our experts, and the communities they work with, are most familiar. As technology evolves, so does the list and we welcome input from the community.

This year, we’ve added a section on container management and automation tools, which are a hot area for development as companies race to fill the growing need to manage highly distributed, cloud-native applications. We’ve collected traditional DevOps CI/CD tools in a separate category, though functionality can overlap. While CI/CD tools are well tested in production and established for their purposes, most container management tools are still early days.

Guide to the Open Cloud: Current trends and open source projects10

Since our last report, the SDN category has continued to rapidly evolve and gain maturity. We are now beyond the conceptual phase of open source networking, and the companies that were assessing the potential of these projects two years ago have begun enterprise deployments.

The promise of lightweight cloud operating systems tailored to web applications, mobile and embedded systems and specialized use cases has now also become a reality. Such operating systems are ubiquitous in mobile computing and the Internet of Things, which has an entire cloud ecosystem of its own.

Guide to the Open Cloud: Current trends and open source projects11

ProfilesInfrastructure as a Service (IaaS)

APACHE CLOUDSTACKApache CloudStack, an Apache Software Foundation project, is software designed to deploy and manage large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform. It can be used to offer public cloud services, to provide an on-premises (private) cloud offering, or as part of a hybrid cloud solution. Users can manage their cloud with a Web interface, command line tools, and/or a full-featured RESTful API.

CloudStack on GitHub

HPE HELION EUCALYPTUSHPE Helion Eucalyptus is an open solution for building private clouds that are compatible with Amazon Web Services (AWS). It provides open source implementations of many Amazon Web Services to deploy AWS applications into a private cloud without changing tools, processes, or application code.

HPE Helion Eucalyptus on GitHub

OPENNEBULAOpenNebula is software to manage virtualized data centers for private, public, and hybrid IaaS clouds. Use OpenNebula to manage data center virtualization, consolidate servers, and integrate existing IT assets for computing, storage, and networking. Or provide a multi-tenant, cloud-like provisioning layer on top of an existing infrastructure management solution.

OpenNebula on GitHub

OPENSTACKOpenStack, an OpenStack Foundation project, is open source software for creating private and public clouds. The software controls large pools of compute, storage, and networking resources throughout a data center, and is managed through a dashboard or via the OpenStack API. OpenStack works with other enterprise and open source technologies making it ideal for heterogeneous infrastructure.

OpenStack on GitHub

Guide to the Open Cloud: Current trends and open source projects12

Platform as a Service (PaaS)

APACHE STRATOSApache Stratos, an Apache Software Foundation project, is a highly-extensible PaaS framework that helps run Apache Tomcat, PHP, and MySQL applications and can be extended to support many more environments on all major cloud infrastructures. For developers, Stratos provides a cloud-based environment for developing, testing, and running scalable applications. IT providers benefit from high utilization rates, automated resource management, and platform-wide insight including monitoring and billing.

Stratos on GitHub

CLOUD FOUNDRYCloud Foundry, a Cloud Foundry Foundation project at The Linux Foundation, is an open source cloud application platform that provides a choice of clouds, developer frameworks, and application services. It supports applications built in virtually any programming language, run as either Linux containers or Windows-based applications, and deployed across many infrastructure types: Amazon Web Services (AWS), Microsoft Azure, Google Compute Platform (GCP), OpenStack, VMware vSphere, VMware Photon Platform, IBM SoftLayer, and more.

Cloud Foundry on GitHub

DEIS WORKFLOWDeis Workflow, is Engine Yard’s version 2.0 (beta) of the open source Deis PaaS that makes it easy to deploy and manage applications on Kubernetes. It works in both public and private clouds, as well as on bare metal.

Deis on GitHub

FLYNNFlynn is an open source PaaS for running applications in production. In its 1.0 version, it’s designed to run anything that can run on Linux, and includes built-in service discovery as well as Postgres, MySQL, and MongoDB databases.

Flynn on GitHub

HEROKUHeroku is a cloud platform that lets companies build, deliver, monitor and scale applications. It’s based on a managed container system, with integrated data services and a powerful

Guide to the Open Cloud: Current trends and open source projects13

ecosystem, for deploying and running modern apps in multiple languages: Node, Ruby, Java, Scala, PHP, and more.

Heroku on GitHub

OPENSHIFTOpenShift is Red Hat’s PaaS that allows developers to quickly develop, host, and scale applications in a cloud environment. Based on top of Docker containers and the Kubernetes container cluster manager, OpenShift offers online, on-premise, and open source project options. OpenShift Origin is the open source upstream of OpenShift.

OpenShift on GitHub

Virtualization, Containers and Cloud Operating Systems

MICRO OR MINIMALIST OSES

PROJECT ATOMICProject Atomic is Red Hat’s umbrella for many open source infrastructure projects to deploy and scale containerized applications. It provides an operating system platform for a Linux Docker Kubernetes (LDK) application stack, based on Fedora, CentOS and Red Hat Enterprise Linux.

Project Atomic on GitHub

COREOSA lightweight Linux operating system designed for clustered deployments providing automation, security, and scalability for containerized applications. It runs on nearly any platform whether physical, virtual, or private/public cloud.

CoreOS on GitHub

PHOTON OSPhoton OS is a minimal Linux operating system for cloud-native apps optimized for VMware’s platforms. It runs distributed applications using containers in multiple formats including Docker, rkt, and Garden.

Photon on GitHub

Guide to the Open Cloud: Current trends and open source projects14

RANCHEROSRancherOS is a minimalist Linux distribution for running Docker containers. It runs Docker directly on top of the kernel, replacing the init system, and delivers Linux services as containers.

RancherOS on GitHub

VIRTUALIZATION

KVMKVM (Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko. It can run multiple virtual machines running unmodified Linux or Windows images.

KVM mailing lists

LXCLinux Containers (LXC) are lightweight virtual machines enabled by functions within the Linux kernel, including cgroups, namespaces and security modules. Userspace tools coordinate kernel features and manipulate container images to create and manage system or application containers.

LXC on GitHub

LXDLXD is Canonical’s container hypervisor and a new user experience for LXC. Developed in Go, it runs unmodified Linux operating systems and applications with VM-style operations.

LXD on GitHub

XEN PROJECTXen Project, a Linux Foundation project, develops virtualization technologies for a number of different commercial and open source applications including server virtualization, Infrastructure as a Service (IaaS), desktop virtualization, security applications, embedded and hardware appliances on x86 and ARM CPU architectures, and supports a wide range of guest operating systems.

Xen Project Git repositories

Guide to the Open Cloud: Current trends and open source projects15

MANAGEMENT AND AUTOMATION

APACHE AURORAApache Aurora is a Mesos framework that runs applications and services across a shared pool of machines, and is responsible for keeping them running, forever. When machines experience failure, Aurora intelligently reschedules those jobs onto healthy machines.

Aurora Git repository

APACHE MESOSApache Mesos, an Apache Software Foundation project, abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), to build and run fault-tolerant and elastic distributed systems.

Mesos Git repository

CLOUD FOUNDRY DIEGODiego is Cloud Foundry’s next-generation container management runtime system. Within CF, the Diego system distributes the hosted app load over all of the host VMs, and keeps it running and balanced through demand surges, outages, or other changes.

Diego on GitHub

DOCKER ENGINEDocker Engine is Docker’s open source containerization technology combined with a workflow for building and containerizing applications. Docker containers run on desktops, physical servers, virtual machines, data centers, and public and private clouds.

Docker on GitHub

DOCKER SWARM (MACHINE AND COMPOSE)Docker Swarm is native clustering for Docker. It turns a pool of Docker hosts into a single, virtual Docker host. Because Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to scale to multiple hosts. It supports multiple tools including Dokku, Docker Compose, Docker Machine, and Jenkins.

Docker Swarm on GitHub

Guide to the Open Cloud: Current trends and open source projects16

KONTENA Kontena is a platform to deploy, manage, scale, and monitor containerized applications and microservices. It works on any public cloud, private data center, hybrid environment, virtualized or bare metal.

Kontena on Github

KUBERNETESKubernetes, a Cloud-Native Computing Foundation project at The Linux Foundation, is a container cluster manager for automating deployment, scaling, and management of containerized applications, developed and open sourced by Google. It groups containers that make up an application into logical units for easy management and discovery.

Kubernetes on GitHub

MANAGEIQManageIQ is the open source project that powers Red Hat CloudForms. It automates, optimizes, and controls cloud and virtualization services throughout an application’s lifecycle.

ManageIQ on GitHub

OVIRToVirt is a virtualization management application. It can manage hardware nodes, storage and network resources, and deploy and monitor virtual machines running in a data center. It is the upstream project of Red Hat’s Enterprise Virtualization product.

oVirt on GitHub

SKIPPBOX (KMACHINE AND KOMPOSE)Kmachine is a Kubernetes tool from Skippbox that helps developers create a standalone, single-node Kubernetes instance in public clouds, similar to Docker machine. Kompose takes a Docker Compose file that describes a containerized application and automatically generates the corresponding Kubernetes objects.

Kmachine on GitHub | Kompose on GitHub

SYSDIG Sysdig is a distributed system monitoring and troubleshooting tool for Linux with native container support. It captures system state and activity from a running Linux instance, then saves, filters and analyzes. It’s scriptable in Lua and includes a command line interface and a UI.

Sysdig on GitHub

Guide to the Open Cloud: Current trends and open source projects17

WEAVEWORKS Weaveworks is a set of tools to cluster, view, and deploy microservices and cloud-native applications across local and external networks. Weave Net and Flux connect and secure applications across data centers and public clouds and work with any orchestrator. Weave Scope monitors applications in real time.

Weaveworks on Github

WERCKER Wercker is an automation platform to build and deploy containers for multi-tiered, cloud-native applications. It can build containers automatically and deploy them to multiple places including public and private Docker registries, and schedulers such as Marathon, DC/OS, and Kubernetes.

Wercker on GitHub

UNIKERNELS

CLICKOSClickOS is NEC’s high-performance, virtualized software middlebox platform for network function virtualization (NFV) built on top of MiniOS/ MirageOS.

ClickOS on GitHub

CLIVEClive is an operating system written in Go and designed to work in distributed and cloud computing environments.

HALVMThe Haskell Lightweight Virtual Machine (HaLVM) is a port of the Glasgow Haskell Compiler toolsuite that enables developers to write high-level, lightweight virtual machines that can run directly on the Xen hypervisor.

HaLVM on GitHub

INCLUDEOSIncludeOS is a unikernel operating system for C++ services running in the cloud. It provides a

Guide to the Open Cloud: Current trends and open source projects18

bootloader, standard libraries and the build- and deployment system on which to run services. Test in VirtualBox or QEMU, and deploy services on OpenStack.

IncludeOS on GitHub

LINGLing is an Erlang platform for building super-scalable clouds that runs directly on top of the Xen hypervisor. It runs on only three external libraries — no OpenSSL — and the filesystem is read-only to remove the majority of attack vectors.

Ling on GitHub

MIRAGEOSMirageOS is a library operating system incubating under the Xen Project at The Linux Foundation. It uses the OCaml language to construct unikernels for secure, high-performance network applications across a variety of cloud computing and mobile platforms. Code can be developed on a normal OS such as Linux or MacOS X, and then compiled into a fully-standalone, specialised unikernel that runs under the Xen hypervisor.

MirageOS on GitHub

OSVOSv is the open source operating system from Cloudius Systems designed for the cloud. It supports applications written in Java, Ruby (via JRuby), JavaScript (via Rhino and Nashorn), Scala, and others. And it runs on the VMware, VirtualBox, KVM, and Xen hypervisors.

OSv on GitHub

RUMPRUNRumprun is a production-ready unikernel that uses the drivers offered by rump kernels, adds a libc and an application environment on top, and provides a toolchain with which to build existing POSIX-y applications as Rumprun unikernels. It works on KVM and Xen hypervisors and on bare metal and supports applications written in C, C++, Erlang, Go, Java, Javascript (Node.js), Python, Ruby, Rust, and more.

Rumprun on GitHub

RUNTIME.JSRuntime.js is an open source library operating system (unikernel) for the cloud that runs JavaScript, can be bundled up with an application and deployed as a lightweight and immutable VM image. It’s built on V8 JavaScript engine and uses event-driven and non-

Guide to the Open Cloud: Current trends and open source projects19

blocking I/O model inspired by Node.js. KVM is the only supported hypervisor.

Runtime.js on GitHub

UNIKUnik is EMC’s tool for compiling application sources into unikernels (lightweight bootable disk images) rather than binaries. It allows applications to be deployed securely and with minimal footprint across a variety of cloud providers, embedded devices (IoT), as well as a developer laptop or workstation. It supports multiple unikernel types, processor architectures, hypervisors and orchestration tools including Cloud Foundry, Docker, and Kubernetes.

Unik on GitHub

DevOps CI/CD

COMPLETE CI/CD CYCLE

CONCOURSE Concourse is a scalable continuous integration pipeline tool sponsored by Pivotal. It was built for the needs of Cloud Foundry projects to test and deploy on different architectures and platforms and remain compatible with previous versions.

Concourse on GitHub

DRONEDrone is an open source continuous integration platform built on container technology for GitHub and Bitbucket. It monitors code for bugs and integrates with Amazon, Heroku, Google AppEngine, and more. It also works with SSH and shell scripts for custom deployments.

Drone on Github

HYGIEIA Hygieia is a configurable DevOps dashboard to visualize near real-time status of the entire software delivery pipeline, built and released as open source by Capital One.

Hygieia on GitHub

Guide to the Open Cloud: Current trends and open source projects20

JENKINSJenkins is an open source automation server. It provides hundreds of plugins to support building, deploying, and automating any project across multiple platforms. It can be used as a simple CI server or turned into a continuous delivery hub.

Jenkins on GitHub SHIPPABLEShippable is a continuous deployment platform built on Docker that provides end-to-end deployment pipelines for automation and integration. It integrates with multiple development tools and deploys to public clouds or in a hybrid fashion across public and private clouds. It also supports n-tier applications and cloud-native technologies such as Docker containers and microservices.

Shippable on GitHub

TRAVIS-CITravis-CI is a hosted continuous integration and deployment system that integrates with GitHub. It supports multiple languages, can test pull requests before they are merged, and can deploy tested code to Amazon S3 and Heroku.

Travis-CI on GitHub

CONFIGURATION MANAGEMENT

ANSIBLE Ansible is Red Hat’s open source IT automation engine for cloud provisioning, configuration management, application deployment, intra-service orchestration, and other IT needs on multi-tier architectures.

Ansible on GitHub

CHEFChef is a configuration management tool to automate infrastructure. It manages servers in the cloud, on-premises, or in a hybrid environment.

Chef on GitHub

Guide to the Open Cloud: Current trends and open source projects21

PUPPETPuppet is an open source server automation tool for configuration and management. It works on Linux, Unix, and Windows systems and performs administrative tasks (such as adding users, installing packages, and updating server configurations) based on a centralized specification.

Puppet on GitHub

SALT OPENSalt Open is orchestration and configuration management software to manage infrastructure and applications at scale. It’s the upstream open source project for SaltStack and runs on Linux and Windows.

Salt on GitHub

LOGGING AND MONITORING

FLUENTDFluentd is an open source data collector for unified logging layer, sponsored by Treasure Data. It structures data as JSON to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations.

Fluentd on GitHub

HEAPSTERHeapster is a container cluster monitoring and performance analysis tool in Kubernetes. It supports Kubernetes and CoreOS natively and can be adapted to run on OpenShift. It also supports a pluggable storage backend: InfluxDB with Grafana, Google Cloud Monitoring, Google Cloud Logging, Hawkular, Riemann and Kafka.

Heapster on GitHub

LOGSTASHLogstash is Elastic’s open source data pipeline to help process logs and other event data from a variety of systems. Its plugins can connect to a variety of sources and stream data at scale to a central analytics system.

LogStash on GitHub

Guide to the Open Cloud: Current trends and open source projects22

PROMETHEUSPrometheus is an open source systems monitoring and alerting toolkit, originally built at SoundCloud and now a Cloud-Native Computing Foundation project at The Linux Foundation. It fits both machine-centric and microservices architectures and supports multi-dimensional data collection and querying.

Prometheus on GitHub

WEAVE SCOPE Weave Scope is Weaveworks’ open source tool to monitor distributed applications and their containers in real time. It integrates with Kubernetes and AWS ECS.

Weave Scope on GitHub

Software-Defined Networking (SDN)

ONOSOpen Network Operating System (ONOS), a Linux Foundation project, is a software-defined networking OS for service providers that has scalability, high availability, high performance and abstractions to create apps and services.

ONOS on GitHub

OPENCONTRAILOpenContrail is Juniper Networks’ open source network virtualization platform for the cloud. It provides all the necessary components for network virtualization: SDN controller, virtual router, analytics engine, and published northbound APIs. Its REST API configures and gathers operational and analytics data from the system.

OpenContrail on GitHub

OPENDAYLIGHTOpenDaylight, an OpenDaylight Foundation project at The Linux Foundation, is a programmable, software-defined networking platform for service providers and enterprises. Based on a microservices architecture, it enables network services across a spectrum of hardware in multivendor environments.

OpenDaylight on GitHub

Guide to the Open Cloud: Current trends and open source projects23

OPEN VSWITCHOpen vSwitch, a Linux Foundation project, is a production-quality, multilayer virtual switch. It’s designed for massive network automation through programmatic extension, while still supporting standard management interfaces and protocols including NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, and 802.1ag. It supports distribution across multiple physical servers similar to VMware’s vNetwork distributed vswitch or Cisco’s Nexus 1000V.

OVS on GitHub

OPNFVOpen Platform for Network Functions Virtualization (OPNFV), a Linux Foundation project, is a reference NFV platform for enterprise and service provider networks. It brings together upstream components across compute, storage and network virtualization in order create an end-to-end platform for NFV applications.

OPNFV on Bitergia

NETWORKING FOR CONTAINERS

FLANNELFlannel is an etcd-backed overlay network for containers from CoreOS. It was designed for Kubernetes, but is a generic overlay network that can be used as an alternative to existing software-defined networking solutions.

Flannel on GitHub

LIBNETWORKLibnetwork is Docker’s project to create a multi-platform library for networking containers. It enables network plugins by providing a consistent programming interface and the required network abstractions for applications.

Libnetwork on GitHub

PROJECT CALICOCalico is Tigera’s data center networking software that uses a pure Layer 3 approach. It integrates with cloud orchestration systems (such as OpenStack) to enable secure IP communication between virtual machines, containers, or bare metal workloads.

Project Calico on GitHub

Guide to the Open Cloud: Current trends and open source projects24

WEAVE NET Weave Net is Weaveworks’ networking software for container applications across data centers and public clouds. It supports peer-to-peer encryption, traffic isolation, and segmentation for building secure control plane networks, running in production environments, or for joining container deployments in public clouds over the internet.

Weave on GitHub

Software-Defined Storage

APACHE CASSANDRAApache Cassandra, an Apache Software Foundation project, is a scalable, high-availability database for mission-critical applications. It runs on commodity hardware or cloud infrastructure and replicates across multiple data centers for lower latency and fault tolerance.

Cassandra on GitHub

CEPHCeph is Red Hat’s distributed, highly scalable block, object, and file storage platform for enterprises deploying public or private clouds. It’s commonly used with OpenStack.

Ceph on GitHub

COUCHDBApache CouchDB, an Apache Software Foundation project, is a single-node or clustered database management system. It provides a RESTful HTTP API for reading and updating database documents.

CouchDB on GitHub

DOCKER VOLUME PLUGINS Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems and enable data volumes to persist beyond the lifetime of a single Engine host. Volume plugins exist for multiple external storage systems including Azure File Storage, NetApp, VMware vSphere, and more.

See individual plugins on GitHub

Guide to the Open Cloud: Current trends and open source projects25

GLUSTERFSGluster is Red Hat’s scalable network filesystem and data management platform. It can deploy on-premise, in private, public, or hybrid clouds, and in Linux containers for media streaming, data analysis, and other data- and bandwidth-intensive tasks.

GlusterFS on GitHub

MONGODBMongoDB is a high performance document database designed for ease of development and scaling.

MongoDB on GitHub

NEXENTANexentaStor is a scalable, unified software-defined file and block storage service that includes data management functionality. It integrates with VMware and supports Docker and OpenStack.

Nexenta on GitHub

REDISRedis is an in-memory data structure store, used as database, cache and message broker. It supports multiple data structures and has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence.

Redis on GitHub

RIAK CSRiak CS (Cloud Storage) is object storage software built on top of Riak KV, Basho’s distributed database. It provides distributed cloud storage at any scale, and can be used to build public or private cloud architectures or as storage infrastructure for heavy-duty applications and services. Its API is Amazon S3 compatible and supports per-tenant reporting for use cases involving billing and metering.

Riak CS on GitHub

SWIFTSwift is OpenStack’s object storage system designed to store and retrieve unstructured data with a simple API. It’s built for scale and optimized for durability, availability, and concurrency across the entire data set.

Swift on GitHub

Guide to the Open Cloud: Current trends and open source projects26

Thank you for your interest in the 2016 Guide to the Open Cloud!

Looking for training and other resources to help you keep up on cloud technologies? Check out The Linux Foundation’s free Intro to Cloud Infrastructure Technologies course on edX, and our other e-learning courses on OpenStack Administration Fundamentals and Kubernetes Fundamentals.

Our cloud computing events are also a great place to meet the experts in the field and learn the latest technologies and trends. The Linux Foundation organizes industry-wide events including ContainerCon, CloudNativeCon, MesosCon, Cloud Foundry Summit, OpenDaylight Summit, and more. Visit events.linuxfoundation.org for a list of upcoming events.

The Linux Foundation promotes, protects and standardizes Linux by providing unified resources and services needed for open source to successfully compete with closed platforms.

To learn more about The Linux Foundation or our other initiatives please visit us at www.linuxfoundation.org