Building Service Platforms using OpenStack and CEPH€¦ · OpenStack and CEPH Malte Dreyer ....

Building Service Platforms using OpenStack and CEPH Malte Dreyer

Founded in 1810 Faculties: 9 Students: 35 000 (5 000 International) Degree Programs: 185 Professorships: 419(483)

Academic Staff: 2 000 Non-Academic: 1 500 3 Campuses, ~180 Buildings

Humboldt-Universität zu Berlin

Computer and Media Service

Broad Range of Services

Hardware Service

Identity Management

Image Editing

Mail Service

Mailing Lists

Map Service

Media Repository

Moodle

Printing Service

Public Computers

Publication Repository

Resource Management SW

Scanning Service

Shibboleth

Software Service

Statistics SW Service

Storage Area Network

Survey Service

Terminalserver Service

Repositories

Training

Trouble Ticket Systems

User Certificates

User Support

VPN

Video Conferences

Video Service

WLAN

Web Content Management

Web Files

Web Hosting

Website Management

Wiki and Blog Farms

Windows Domains

ePublishing

ADS

AFS

Accounts

Anti Virus Software

Audio / Video Equipment

Backup-Service

Calendar Service

Campus Management SW

Chat Service

Compute Clusters

Computer Networks

Computer Training Rooms

Database Service

Managed Desktops

File Exchange

File Service

Graphics Service

Customizations and

feature developments

Media Repository Farm

Joint projects and

software developments

Laudatio Corpus Repository

Projects in Close Cooperation with Scholars

"Products of research are not just publications.”

NSF senior policy specialist Beth Strausser.

Biographical Sketch(es), has been revised to rename the “Publications”

section to “Products” and amend terminology and instructions accordingly.

13 January 2013: "National Science Foundation’s Merit Review Criteria: Review and Revisions” Chapter II.C.2.f(i)(c),

1. Recommended open access to scholarly papers of

publicly funded research

2. Recommended open access to all digital outputs of

publicly funded research

3. Mandated open access to scholarly papers of publicly

funded research

4. Mandated open access to all digital outputs of

publicly funded research

5. Enforced, mandated open access to scholarly papers

of publicly funded research

6. Enforced, mandated open access to all digital

outputs of publicly funded research

80 individual projects

80 setups

80 databases

80 configs

Custom software

Fedora Repository

CakePHP

Elastic Search

PID-Server (Handle)

jQuery / Bootstrap

Apache Xerces

+RedMine for Tickets

From a Data Center Point of View

Cloud as e-Infrastructure

Supporting Research

Infrastructure Servers–Storage-

Network

Infrastructure Servers–Storage-

Network

Infrastructure Servers–Storage-

Network

Platform OS & Basic Services

Platform OS & Basic Services

Applications Applications and Data

IaaS

PaaS

SaaS

XaaS/EaaS: Software-led Infrastructure

(Software Defined Data Center)

Backup

Database

Identity

Monitoring

Network

Security

Storage

…………………. as a Service

Media Repository

Seafile

Repository xyz

…………………. as a Service

Flexible e-Infrastructure…

reduce technology stack for project staff

concentrate on what you are really interested in

keep services alive

beyond project lifetime

easy testing

environments

We Took a Look At…

(Being aware that these are different in type)

• amazon web services

• Apache Cloudstack

• Docker / LXC

• Eucalyptus

• Microsoft Azure

• OpenNebula

• OpenStack

• VMware

And still use Proxmox for VM management

for many services

… and Chose OpenStack

still under heavy development

Macro Scripting

„Manage VMs like a Linux Process“

Pet or Cattle

Images: Sheep: Public Domain; Toy Dog: By Toyloverz (Own work) [CC BY-SA

3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

OpenStack Component Architecture

• Preise [TEUR]

2016 2017 2018

WDM (Licht) 200 200

Linecards 10 Gbps x20

200

MLX (Router) Chassis 8x

175 175

Linecards (26) 10Gbps x8

286 286

Summe 661 661 200

CEPH Server SAN

Figure to the left: By Inktank, CC-BY-SA

http://ceph.com/docs/master/architecture/

0

200

400

600

800

1000

1200

0

20

40

60

80

100

120

140

160

1 2 3 4 5 6 7 8

Aggregated MB/s

MB/s

Number of instances

Read from disk

Read from cache

Aggregated readfrom disk

Aggregated readfrom cache

Seafile: Deployment in OpenStack

Using Ansible and Heat

HEAT

my_port01:

type: OS::Neutron::Port

properties:

admin_state_up: true

name: Port01

network_id: { get_resource: my_first_network }

my_server07:

type: OS::Nova::Server

properties:

name: seaf07

key_name: { get_param: keypair_name }

flavor: { get_param: machine_flavor }

image: { get_param: image_id }

networks: [{ "port": { get_resource: my_port07 }, …}]

user_data: |

#!/bin/bash -v

….

cinder_volume07:

type: OS::Cinder::Volume

properties:

size: { get_param: volume_size }

volume_attachment07:

type: OS::Cinder::VolumeAttachment

properties:

volume_id: { get_resource: cinder_volume07 }

instance_uuid: { get_resource: my_server07 }

mountpoint: /dev/vdb

ANSIBLE Playbook

# update packet cache

- hosts: all

tasks:

- apt:

- name: apply common configuration to all nodes

hosts: all

roles:

- common

- name: deploy and configure memcached server

hosts: memcached

roles:

- memcached

- name: deploy MySQL and configure the databases

hosts: db

roles:

- db

- name: deploy seafile and apply configuration

hosts: nodes

roles:

- nodes

- name: deploy loadbalancer

hosts: loadbalancer

roles:

- loadbalancer

tags:

- loadbalancer

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Organizational Responsibilities

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Organizational Responsibilities /1

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Organizational Responsibilities /2

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Organizational Responsibilities /3

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Organizational Responsibilities /4

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Organizational Responsibilities /5

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Organizational Responsibilities /6

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Organizational Responsibilities: New Distributed Competency

Build new competencies

• OpenStack is complex and additional technologies are needed

• An advanced training program

Changing system administration styles

• Develop vs. configure manually step by step

Re-Use what others built already (and share again)

• Read scripts from other people

Be patient when OpenStack is still immature, or Neutron fails again

• So far it develops/improves very fast

• Dogfooding builds trust

Organizational Challenges and Conclusions

..…Thanks…..

malte.dreyer@cms.hu-berlin.de

Questions?

malte.dreyer@cms.hu-berlin.de