Hostvn ceph in production v1.1 dungtq

What is Ceph?How we deploy in production

VietOpenStack Meetup

Hanoi July 1, 2017

Agenda

• What is Ceph?

• Key features

• Why we choose Ceph

• How we deploy in production

• Lessons learned

• What next?

• Questions

About me

Trinh Quang Dung• Chief Technology Officer at Hostvn.net

• Been with Hostvn since 2009

• Start deploy OnApp public cloud since 2011

• System lead for all Hostvn services

• Excited about storage, distributed systems, networking, linux, automation and monitoring

• Interested in new technology

What is Ceph?

Ceph is a free & open distributed storage platform that provides unified object, block, and file storage

Key Features

• Self managing/healing

• Self balancing (uniform distribution)

• Painless scaling

• Reliable multiple Replicas

• Distributed Data/ replicas spread across nodes/racks/DCs

• Data placement with CRUSH• Pseudo-random placement algorithm• Rule-based configuration

Why we choose Ceph

• Free & Open Source software

• Unified distributed storage system

• Horizontal scaling

• No Vendor “lock-in”

• Doesn’t require specified hardware

• Customize as needed

#We call this is a “Heart of system”, sound great?

Everything free? Not really!

• Need one guy that really know what’s need to do

• Need a dedicated team to operation

• Need a lot of times to build, test, troubleshoot, solving issues

• Need consider everything that could happen

• Need support from else where when you’re over-control

…and finally it’s need to find out a “Golden ratio” for configuration

Hey! Look like we should stop right here? Hold on. Move to the next slide please!

How we deploy in production

Few things/ideas• We want to handle ceph cluster as much as possible (command line only)

• Troubleshoot ceph cluster is not easy task so it must be take note everything we did

• Start with small cluster first

• Using all SSD drive to achieve better read/write performance

• Journal on the same osd-data drive

How we deploy in production - cont

Few things/ideas• Keep all default settings

• LVM with RBD backend

• Using x2 replication pool

• Deployment automation

• Monitoring all metrics, logs of cluster


Software• Ceph Jewel 10.2

• CentOS 7.3 x64 minimal

• ansible

• collectd, graphite, grafana, beacon

• filebeat, logstash, elasticsearch, kibana


Hardware• 1x Admin node

• 2x Intel Xeon 2.9Ghz

• 32GB memory

• 2x 120GB OS

• 2x 1GbE, 2x 10GbE

• 3x MON nodes• 2x Intel Xeon 2.9Ghz

• 32GB memory

• 2x 120GB OS



Hardware• 3x OSD nodes

• 2x Intel Xeon 2.9Ghz

• 32GB memory

• 2x 120GB OS

• 4x 2TB SSD

• LSI HBA


• 1x 10GbE ceph-public switch

• 1x 10GbE ceph-cluster switch

• 1x 1GbE management switch


• Prepare• Install ntp, configure and start it on all node (important)

• Set selinux to permissive mode and disable iptables

• Create hostname for each MON, OSD node in /etc/hosts (or using local dns)

• Create a user, add ssh public_key for that user on MON, OSD node (i.e: hostvn)

• Create yum repo for ceph install at /etc/yum.repos.d/ceph-deploy.repo

• Update OS, install epel-release, ceph-deploy

[ceph-noarch]

name=Ceph noarch packages

baseurl=https://download.ceph.com/rpm-jewel/el7/noarch

enabled=1

gpgcheck=1

type=rpm-md

gpgkey=https://download.ceph.com/keys/release.asc


• Deploy• Create a directory on admin node for maintaining the

configuration files and keys

• Initial monitor nodes:

• Create config file /etc/ceph/ceph.conf

• Install ceph, add the initial monitor and gather the keys:

mkdir /etc/ceph && cd /etc/ceph

osd pool default size = 2

osd pool default min size = 1

public network = 10.10.0.0/24

cluster network = 172.16.10.0/24

ceph-deploy install admin-node mon0 mon1 mon2 osd0 osd1 osd2

ceph-deploy mon create-initial

ceph-deploy new mon0 mon1 mon2


• Add disk to cluster (on each OSD node)

• Create a pool and rbd

ceph-deploy osd prepare {node-name}:{data-disk}[:{journal-disk}]

ceph-deploy osd prepare osd0:sda:/dev/sda

ceph-deploy osd activate {node-name}:{data-disk-partition}[:{journal-disk-partition}]

ceph-deploy osd activate osd0:/dev/sda1:/dev/sda2

ceph osd pool create {pool-name} {pg-num}

ceph osd pool set {pool-name} size {size}

ceph osd pool create onapp-vps 1024

ceph osd pool set onapp-vps size 2

rbd create vps0 -p onapp-vps --size 2000000

[root@mon0 ~]# rbd info vps0 -p onapp-vps

rbd image 'vps0':

size 1953 GB in 500000 objects

order 22 (4096 kB objects)

block_name_prefix: rb.0.10a7.238e1f29

format: 1


• Cluster status

[root@mon0 ~]# ceph df

GLOBAL:

SIZE AVAIL RAW USED %RAW USED

22285G 21848G 436G 1.96

POOLS:

NAME ID USED %USED MAX AVAIL OBJECTS

onapp-vps 1 221G 1.99 10904G 57977

[root@mon0 ~]# ceph status

cluster a10afbea-f727-430b-93d8-c59d214f0bef

health HEALTH_OK

monmap e3: 3 mons at {mon0=10.10.0.198:6789/0,mon1=10.10.0.199:6789/0,mon2=10.10.0.202:6789/0}

election epoch 8, quorum 0,1,2 mon0,mon1,mon2

osdmap e82: 12 osds: 12 up, 12 in

pgmap v74854: 1024 pgs, 1 pools, 221 GB data, 57977 objects

436 GB used, 21848 GB / 22285 GB avail

1024 active+clean


• OSD tree[root@mon0 ~]# ceph osd tree

ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY

-1 21.71997 root default

-2 7.23999 host osd0

0 1.81000 osd.0 up 1.00000 1.00000

1 1.81000 osd.1 up 1.00000 1.00000

2 1.81000 osd.2 up 1.00000 1.00000

3 1.81000 osd.3 up 1.00000 1.00000

-3 7.23999 host osd1

4 1.81000 osd.4 up 1.00000 1.00000

5 1.81000 osd.5 up 1.00000 1.00000

6 1.81000 osd.6 up 1.00000 1.00000

7 1.81000 osd.7 up 1.00000 1.00000

-4 7.23999 host osd2

8 1.81000 osd.8 up 1.00000 1.00000

9 1.81000 osd.9 up 1.00000 1.00000

10 1.81000 osd.10 up 1.00000 1.00000

11 1.81000 osd.11 up 1.00000 1.00000

Diagram

Lessons learned

• Designing simple as much as possible

• Always use enterprise SSD for journal if using SATA drive

• Take sizing PGs carefully

• Choose the right hardware

• IOPS’s isn’t everything, latency potential bottleneck

• Keep an eye on back-fill, re-balance process

• No one size fits all model so make sure you’re understanding what you need when deploying ceph

What next?

• Extending current SSD ceph cluster with more OSD nodes

• Deploy a new ceph cluster with SATA drive

• Journal on enterprise SSD

• 3x replication pool

• Upgrade 40-60 Gbps networking for ceph-cluster

• Planning to replace current storage

• Integrated with OpenStack, seriously? :D

Thanks For Your Time!Questions?

CÔNG TY CỔ PHẦN CÔNG NGHỆ ÁNH SÁNG VIỆT

Trụ sở: Số 1, ngõ 73 Hoàng Cầu, Đống Đa, Hà Nội, Việt NamVP Hồ Chí Minh: 166/3 Hồ Bá Kiện, Q10, TP. Hồ Chí Minh, Việt NamWebsite: www.hostvn.net - www.hostvn.comDịch vụ: Domain | Hosting | Cloud VPS | Email | Server | License | Online Solutions

http://hostvn.net/

http://hostvn.com/

Devices & Hardware

Hostvn ceph in production v1.1 dungtq