OpenCms Days 2014 - OpenCms cloud setup with the FI-TS

Interactive Thinking

OpenCms Cloud setup with the FI-TS

November 4, 2014

OpenCms Days 2014

Talking about

» Introduction

» A brief look back

» Todays cluster setup

» Cloud-readiness

» Dynamic Up- & Down-Scaling

OEV Interactive Thinking10.11.20142

IntroductionPreconditions + Architecture

Preconditions

» High Availability (24x7, 99.5%)

» 60 Sites (40 https), 250.000 Resources, 25GB

» 8.000.000+ Requests/day during business hours

» User- & Group-Management via LDAP (2000 Users, 250 Groups)

» ~25 Webapps beside OpenCms

» Connection to 20 customer backends via private network

»Workplace only accessible via private network


Architecture (detailed)


Inte

rnet

Inte

rnet

Load

bal

ance

r+

SSL

Load

bal

ance

r+

SSL

ContentDatabaseContent

Database

OpenCmsOpenCms

ApacheApache

AppDataAppDataSe

rvic

eSe

rvic

e

ESBESB

Backend GatewayBackend Gateway

LDAPLDAP

StaticexportsStatic

exports

Serv

ice

Serv

ice

Service Fast LaneService Fast Lane Serv

ice

Serv

ice

Backend Fast LaneBackend Fast Lane

Architecture (Bird‘s-Eye-View)


InternetInternet

Loadbalancer + SSLLoadbalancer + SSL

MySQLMySQL

OpenCmsOpenCms

ApacheApache

MySQLMySQL

OpenCmsOpenCms

ApacheApache


CustomersCustomers

MySQLMySQL

OpenCmsOpenCms

ApacheApache

publishpublish …

Wo

rkp

lace

acc

ess

LDAPLDAPServicesServices ServicesServices

Star of the Show


InternetInternet


MySQLMySQL

OpenCmsOpenCms

ApacheApache

MySQLMySQL

OpenCmsOpenCms

ApacheApache


CustomersCustomers

MySQLMySQL

OpenCmsOpenCms

ApacheApache

publishpublish …

Wo

rkp

lace

acc

ess


OCEEOCEE

OpenCms Enterprise Extensions

» Accelerator: Reduces database calls

» VFS Doctor: Simplifies database maintaining for witnessed or duplicate resources

» LDAP Connector: self-explanatory

» Cluster Manager: manage Workplace- and Non-Workplace-Servers

» Replication: publish over server boundaries


Combat tested and approvedSince 2006

The Past-2013

The former provider

» Apache and JBoss on different servers

» OpenCms 7.5 and Services within same JVM

» Dedicated hardware

» Contract abandoned at the end of 2013


InternetInternet


MySQLMySQL

JBossOpenCmsServices


ApacheApache


MySQLMySQL



ApacheApache

In search for a new provider…

… we found:

» State-of-Art Datacenter

» Connectivity to all our customers

» Cloud-System (IaaS)

» Detailed OpenCms-knowledge

» Long time developer collaboration

» Cloud experience


Combining the best of two Worlds

The Content-Delivery-TripletThe Content-Delivery-Triplet


Who is who in the Cloud-Ecosystem

Hardware

IaaS

PaaS

SaaS

BPaaS

Today2014

After the migration

» Apache and JBoss on the same servers

» OpenCms 7.5 and Services within same JVM

» Virtualized Environment

» Network storage

» Self-Service-Management

» Application- and System-Monitoring

» Enhanced Alerting


InternetInternet


MySQLMySQL



ApacheApache


MySQLMySQL



ApacheApache

Man

agem

en

t &

Mo

nit

ori

ng

Man

agem

en

t &

Mo

nit

ori

ng


Nimbus


Zabbix


AppDynamics


Splunk

Experiences after 10 Month (PaaS)

» Setting up a 3rd cluster member within 24h

» Needs reconfiguring the workplace-server

» DB-cloning is the bootleneck

» +1 OCEE-License required

» Running perfect


InternetInternet


MySQLMySQL



ApacheApache


MySQLMySQL



ApacheApache

Man

agem

en

t &

Mo

nit

ori

ng

Man

agem

en

t &

Mo

nit

ori

ng

MySQLMySQL



ApacheApache

Experiences after 10 Month (IaaS)

» Elastic Loadbalancer based on nginx & haproxy

» Fast

» Setup shared with other customers

» SSL-Cipher-Ordering

» Header crippling

» no access to logfiles

» Self-Service-GUI can only handle1 certificate per HTTPS => No SNI

» Limited HTTP/1.1 support

» POST chunkend encoding


InternetInternet


nginxSSL-Termination

nginxSSL-Termination

haproxyLoadbalancing

haproxyLoadbalancing

Experiences after 10 Month (IaaS)

» Server sizing only as fixed packages

» S = 1CPU/4GB RAM

» M = 2CPUs/8GB RAM

» L = 4CPUs/16GB RAM

» XL = 8CPUs/32GB RAM

» Resizing requires Backup and Restore

» Only for Operating System (~16GB)

» Data-Storage untouched


The FutureNovember 3, 2014 12:00 -

… are we ready for it?

» Adapt to workload changes using performance indicators

» Anomaly-detection

» Tracking & tracing microservices


Now that we have a cloud…

from to

Where is it?

Using automation tools like puppet, chef or ansible for

» Networking

» OS

» Apache

» Tomcat

»MySQL


Setting up the base system

Architecture (Bird‘s-Eye-View)


InternetInternet


MySQLMySQL

OpenCmsOpenCms

ApacheApache

MySQLMySQL

OpenCmsOpenCms

ApacheApache


CustomersCustomers

MySQLMySQL

OpenCmsOpenCms

ApacheApache

publishpublish …

Wo

rkp

lace

acc

ess


With automation tools you can

» Deploy OpenCms

» Deploy OCEE


Setting up OpenCms + OCEE

But you can‘t

» Aquire a license key to activate OCEE

» Depends on server.ethernet.address

» server.ethernet.address must be unique

Possible workarounds

» Crack the license manager => evil

» Stockpile licenses => expensive

» Call Alkacon, install license after assignment => slow

»@Alkacon: provide a licensing webservice, please => nice to have

Site Management

Workplace only accessible via private network:

» Using a DNS-Scheme likeDevelopment http://d.cloud.privateIntegration http://i.cloud.privateProduction http://r.cloud.private (r = Redaktion, german for editors staff)

Each customer is provided with a shorthand symbolic name:

» OEV Online Dienste = oevVersicherungskammer Bayern = vkb…

Symbolic name = OpenCms Sitename = Servername = Secure Servername:

» /sites/oev http://oev.d.cloud.private https://oev.d.cloud.private/sites/vkb http://vkb.d.cloud.private https://oev.d.cloud.private…


Site Management

Symbolic name = OpenCms Sitename = prefix for Servername

» /sites/oev http://oev.d.cloud.private https://oev.d.cloud.private/sites/vkb http://vkb.d.cloud.private https://oev.d.cloud.private

» Easy to setup with Nameserver-Delegation


InternetDNS

InternetDNS

CloudDNS

CloudDNS

Clientnslookup

Clientnslookup

*.cloud.private Everything else

Site Management

Does not work for production system, because

» /sites/oev https://www.oev-online.de ≠ https://oev.r.cloud.private/sites/vkb https://www.vkb.de ≠ https://vkb.r.cloud.private

No secure Server-Aliases for HTTPS

» /sites/oev https://www.oev-online.de ≠ https://www.oevonline.de/sites/vkb https://www.vkb.de ≠ https://www.versicherungskammer.de

» Site Manager in OpenCms 9 has no option to distinguish betweenworkplace- and non-workplace-servername

» There is only one secure server allowed => no SNI

» Site-Management within Cluster Manager can only remove sites from non-workplace-servers or clone (via compare) to the non-workplace-servers

» Changing sites requires reconfiguration of frontside webservers


Site Management (Workaround)

All changes in Site-Management are written to opencms-system.xml immediately

»Watch the file for changes

» using Linux inotify-interface

» use inotifywait within shellscripts


#!/bin/shwhile inotifywait -e modify opencms-system.xml; do

# do something usefuldone

Site Management (Workaround)

do something useful (examples)

» Create opencms-system.xml for non-workplace-servers and distribute them

» Use a servername-lookup-table with sitename as key

» Create fragments for Apaches httpd.conf

» Using xsl to transform xml

» Populating map-Files for Apache RewriteMaps does not require restarting

» Apache 2.4 mod_macro is your friend

» Check-in changes to configuration management

» Analyse changes

» automagicaly populate your configuration templates

» Distribute the needed configs

» Restart services as needed (regarding availability requirements)


Up- & Down-Scaling

Metrics retrieved from

» Zabbix

» Apache mod_status

»MySQL stats

» JMX

» JVM

» Tomcat

» OpenCms (extended by own JMX-lib)

» AppDynamics

Thresholds => WIPTrigger scaling, but don‘t build servers


Conlusion

Conclusion

»Mass-hosting is possible with OpenCms

» Much easier with it‘s little helper: OCEE

» Unified naming for clarity

» External toolset needed

» Experienced knowledge of the inner workings


Cloudready with minor drawbacks

OEV Online Dienste GmbHHansaallee 18340549 Düsseldorf

oev-online.de

Tel.: 0211 / [email protected]

ThanksFor Your Attention.

Michael Linkenheil

Software

OpenCms Days 2014 - OpenCms cloud setup with the FI-TS