HAProxy scale out using open source

Scale out using open source

HAProxy - Scale out using open source | by Ingo Walz

2

Why HAProxy?

High availability

Powerful loadbalancer for websites due to its proxy nature

Open Source

Enterprise ready


3

Enterprise options

ALOHA HAProxy Loadbalancer Appliance

HAProxy Enterprise Edition - HAPEE

http://www.haproxy.com/

http://www.haproxy.com/


4

Who's using it?

http://www.haproxy.org/they-use-it.html

http://www.haproxy.org/they-use-it.html


5

Featureset

Content switching / filtering Asymetric load balancing Priority activation SSL offloading HTTP compression TCP buffering Priority queue / rate shaping Direct server return (DSR)

http://en.wikipedia.org/wiki/Load_balancing_(computing)#Load_balancer_features

http://en.wikipedia.org/wiki/Load_balancing_(computing)#Load_balancer_features


6

Looks familiar?


7

Fix your Single-Server Environment

Congratulation, your whole environment is one Single Point Of Failure!


8

Fix your Single-Server Environment

Always try to follow the principle:

One function per component

Not anymore because it scales the best, more because it's the cleanest way to manage them.

Configure Services, not Servers


9

Make your application cluster capable

You need to deploy to a various number of different machines

Do not use something like NFS to workaround

A CI will help you

Session clustering

Avoid to work on the filesystem to save data / user input

Use central technologies to save your data (e.g. databases)

HAProxy - Scale out using open source | by Ingo Walz 10

Installation


11

Known procedure

$ wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.6.tar.gz

$ tar xvzf haproxy-1.5.6.tar.gz

$ cd haproxy-1.5.6

$ ./configure USE_OPENSSL=1 USE_PCRE=1

$ make

$ sudo make install

http://www.haproxy.org/#down

http://www.haproxy.org/download/1.5/src/haproxy-1.5.6.tar.gz

http://www.haproxy.org/#down


Configuration


13

Configuration sections

global# process wide and eventually OS specific

# some have CLI equivalents

[ .. ]

defaults# set default parameters for all following sections

[ .. ]

frontend# describes a set of listening sockets accepting client connections

[ .. ]

backend# describes a set of servers to which the proxy will connect

# to forward incoming connections

[ .. ]

listen# defines a complete proxy with its frontend and backend parts combined in one section.

# It is generally useful for TCP-only traffic

[ .. ]

/usr/local/etc/haproxy/haproxy.cfg


14

TCP vs HTTP loadbalancing

defaults

mode tcp # Can balance everything, the default

defaults

mode http # But you want that!

Layer 7 loadbalancing advantages● Request inspection● Content switching● Header manipulation● Cookie persistence● Advanced health checks



15

Loadbalance your nginx


16

Loadbalance your nginx

global

maxconn 4096

daemon

defaults

mode http

timeout connect 5000

timeout client 50000

timeout server 50000

frontend www_fe

bind :80

# Close connection to server but keep open for client

option http-server-close

default_backend www

backend www_be

server nginx1 10.0.0.10:80 check




17

Still a single point of failure


18

HA with HAProxy & keepalived


19

HA with HAProxy & keepalived

vrrp_script chk_haproxy {

script "killall -0 haproxy"

interval 2

weight 2

}

vrrp_instance VIRTUAL {

interface eth0

virtual_router_id 10

state MASTER #state BACKUP

priority 100 #priority 101

advert_int 1

virtual_ipaddress {

10.0.0.30

}

track_script {

chk_haproxy

}

}

/etc/keepalived/keepalived.conf


20

HA with public IPs


21

HA with public IPs - failover


The final step


23

Loadbalance MySQL - TCP


24


frontend mysql_fe

bind :3306

mode tcp

default_backend mysql_be

backend mysql_be

mode tcp

option mysql-check user haproxy

server mysql1 10.0.0.40:3306 check

server mysql2 10.0.0.45:3306 check backup

mysql~> INSERT INTO mysql.user (Host,USER) VALUES ('10.0.0.20','haproxy'); FLUSH PRIVILEGES;

mysql~> INSERT INTO mysql.user (Host,USER) VALUES ('10.0.0.25','haproxy'); FLUSH PRIVILEGES;



25



26



All about SSL


28

Setup SSL Pass-Through

frontend www_fe

bind :80

bind :443

mode tcp

default_backend www_be

backend www_be

mode tcp



No HTTP mode possible – how to inspect encrypted headers?



29

SSL Termination – why you should offload

Single configuration point for all certificates

Certificates not widely spread across the infrastructure

Offload the decryption load

Typically, your HAProxy will have a bit of CPU left

You need to decrypt to inspect the request information


30

Setup SSL offloading

frontend www_fe

bind :80

bind :443 ssl crt /etc/haproxy/sample.pem

# Close connection to server but keep open for client



backend www_be



$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout \

ssl/sample.key -out ssl/sample.crt

$ cat ssl/sample.key ssl/sample.crt > ssl/sample.pem



31

SNI – How it works

● Multiple certificates per IP / frontend profile● Client and server need to support it


32

Setup SNI

frontend www_fe

bind :80

# sample.pem as default, other pem's based on host header

bind :443 ssl crt /etc/haproxy/sample.pem crt /etc/haproxy/certs/

# Content switch based on certificate (and based on host)

use_backend sample1 if { ssl_fc_sni sample1 }

use_backend sample2 if { ssl_fc_sni sample2 }


backend sample1


backend sample2


backend www_be





Secure your entry point


34

Protect against syn flooding

# Consider this amount of clients as valid

$ sysctl -w net.ipv4.tcp_max_syn_backlog=”4096”

# Once net.ipv4.tcp_max_syn_backlog is reached, enable syn cookies

$ sysctl -w net.ipv4.tcp_syncookies=1

# Enable reverse path filtering, is the source routable through the incoming interface?

$ sysctl -w net.ipv4.conf.all.rp_filter=1


35

Basic iptables

$ cat iptables.sh

#!/bin/bash

iptables -F # Drop current table

# Drop incoming traffic (eth0 is the public available interface)

iptables -i eth0 -P INPUT DROP

iptables -i eth0 -P FORWARD DROP

# Allow outgoing traffic

iptables -P OUTPUT ACCEPT

# Allow ping

iptables -i eth0 -A INPUT -p ICMP -j ACCEPT

# Allow SSH (this should be avoided, SSH to haproxy via internal interface / through VPN)

iptables -i eth0 -A INPUT -j ACCEPT -p tcp --dport 22

# Allow HTTP


# Allow HTTPS


# Allow connections from localhost on every port

iptables -i eth0 -A INPUT -j ACCEPT -s 127.0.0.1

# Already opened connections are accepted on every port (required for some daemons)

iptables -i eth0 -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT


36

Use HAProxy to secure your environment

frontend www_fe

bind :80

bind :443 ssl crt /usr/local/etc/haproxy/sample.pem


#detect and reject shellshock requests

reqdeny ^[^:]+:\s*$\s*$\s+\{

reqdeny ^[^:]+:\s+.*?(<<[^<;]+){5,}

#This rule to display SSLv3 error message

acl sslv3 ssl_fc_protocol SSLv3

http-request allow if sslv3

use_backend backend_sslv3 if sslv3


backend backend_sslv3

mode http

errorfile 503 /usr/local/etc/haproxy/pages/poodle.http



37

Check SSLv3 error message

$ openssl s_client -connect 10.0.0.30:443 -ssl3

[ … ]

SSL-Session:

Protocol : SSLv3

---

GET /

[ … ]

<html>

<head>

<title>SSLv3 detected</title>

</head>

[ … ]

</html>


38

Persistent HTTP loadbalancing

backend www_be

cookie PHPSESSID prefix

server nginx1 10.0.0.10:80 cookie nginx1 minconn 10 maxconn 20 check


# Set-Cookie:PHPSESSID=nginx1~7cmjd41klupaderap0q7tve357; path=/

Persistence only if PHPSESSID cookie is set!

backend www_be

cookie server insert indirect nocache



# Set-Cookie:server=nginx1; path=/



39

ACL

Extract data from request / response stream

Perform content switching

Conditional request handling

Can help you to secure your environment

E.g. display an error message for SSLv3


40

Loadbalancing algorithms (most useful)

roundrobin

leastconn

Suggested if you have very long sessions

source

Only useful in TCP environments

Other methods availablehttp://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-balance

http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-balance


41

“Reverse proxy” usage

frontend www_fe

bind :80

bind :443

[ … ]

use_backend nginx1_backend if { path_beg /nginx1 }

use_backend nginx2_backend if { path_beg /nginx2 }

backend nginx1_backend

reqrep ^([^\ :]*)\ /nginx1(/.*) \1\ /\2

server nginx1 10.0.0.10:80 cookie nginx1 check

backend nginx2_backend

reqrep ^([^\ :]*)\ /nginx2(/.*) \1\ /\2

server nginx2 10.0.0.15:80 cookie nginx1 check

You need to cut nginx1/2 from the request



42

Statistics


Run the example


44

Reminder


45

Used technologies

Vagrant >= 1.5.2

ChefDK >= 0.2.0

Berkshelf


https://github.com/iwalz/zendcon-haproxy



47

Project structure


48

Vagrant

$ vagrant plugin install vagrant-omnibus

$ vagrant plugin install vagrant-berkshelf

Omnibus for chef solo support

Berkshelf to manage cookbook dependencies


49

Vagrant up

$ git clone https://github.com/iwalz/zendcon-haproxy

$ cd zendcon-haproxy

$ vagrant up haproxy1

$ vagrant up haproxy2

$ vagrant up nginx1

$ vagrant up nginx2

$ vagrant up mysql1

$ vagrant up mysql2

Don't simply use `vagrant up`, the Berkshelf

Dependencies will be messed up



50

How to continue

Architecture documentation (outdated, but still useful)

http://www.haproxy.org/download/1.3/doc/architecture.txt

Official documentation

http://cbonte.github.io/haproxy-dconv/configuration-1.5.html

Haproxy.com Blog

http://blog.haproxy.com/

http://www.haproxy.org/download/1.3/doc/architecture.txt

http://cbonte.github.io/haproxy-dconv/configuration-1.5.html

http://blog.haproxy.com/


Questions?

Internet

HAProxy scale out using open source