High Availability for Insta CertifierInsta Certifier HA can be implemented with DRBD and Linux-HA. Since DRBD does not require directdependencies to Certifier software, it can be used with any version of Certifier. Linux-HA requiressupport from control and monitoring scripts and is supported by Certifier versions starting from 4.1.0.

DRBD is a block device which is designed to build high availability clusters. This is doneby mirroring a whole block device via (a dedicated) network. ( http://www.drbd.org/)

The Linux-HA project is a widely used and important component in many interestingHigh Availability solutions, and ranks as among the best HA software packages for anyplatform. ( http://www.linux-ha.org/)

The HA solution with these tools is a two-node setup with primary and secondary (backup) configuration.DRBD is used for data synchronization between the nodes. Linux-HA is used for monitoring the state ofthe nodes and controlling the switch-over procedure.

If the HA solution does not require automatic switch-over, the Linux-HA is not a necessity. The steps forswitching from primary node to secondary can be done manually.

Resources that are under HA control include:

DRBD• File system mount required by DRBD• Cluster IP address used by clients to connect Certifier• Certifier•

Resources are active on primary machine and inactive on secondary. DRBD replicates all Certifier data,i.e. the contents of /usr/local/certifier, to the secondary machine. When responsibility movesfrom primary to secondary, all resources on the secondary are activated and deactivated on primary. Thesequence is the following:

Certifier services are stopped on primary node.1. IP address is given up on primary node.2. /usr/local/certifier is unmounted on primary node.3. DRBD certifier resource's primary status is given up, i.e. replication is stopped.4. DRBD certifier resource is given primary status on secondary node.5. /dev/drbdN is mounted to /usr/local/certifier.6. IP address is assigned to the secondary node.7. Certifier services are started on secondary node.8.

Node Requirements

Both nodes should be identical at least for relevant parts:

Disks and partitions used for replication.• Network interface cards.•

Both nodes should have at least two network interfaces:

One dedicated for DRBD and Linux-HA communication. Crossover cable is recommended forDRBD replication. Linux-HA can communicate also via serial line.

•

1

1

One for outside communication.•

There should also be some outside node to ping in order to monitor network availability. When settingHA up, make sure the nodes can ping each other, and that firewall allows required traffic.

Setting up DRBD with Certifier

For DRBD with Certifier, the following procedure must be carried out on both nodes:

Create two dedicated disk partitions:One for Certifier data, i.e. the contents of /usr/local/certifier. The sizerequirement depends on the expected size of the Certifier database.

♦

One for DRBD's meta-disk. Roughly calculated size requirement is < ((<data partitionsize> / 32768) + 1) MB.

♦

1.

Setup network interfaces.Make sure that the nodes can see each other via dedicated interface.♦ It is convenient to add peer node name to /etc/hosts file.♦ DRBD uses TCP ports from 7788 upwards; one port is enough for Certifier resource.Connections are made for both directions between primary and secondary node. Makesure the firewall does not block traffic in the chosen port.

♦

2.

Install DRBD, e.g. yum install drbd and yum install kmod-drbd.3. Create configuration into the file /etc/drbd.conf.4. Create meta-disk: drbdadm create-md certifier5. Start DRBD: run modprobe drbd and drbdadm up certifier.6.

These steps should be carried out on primary node only:

Make initial synchronization: drbdadm -- --overwrite-data-of-peer primarycertifier

1.

Wait for synchronization to finish; check it with command cat /proc/drbd.2. Create file-system: mke2fs /dev/drbd1 (-j option can be used for ext3).3. mkdir /usr/local/certifier4. mount -o rw /dev/drbd1 /usr/local/certifier5. Install Certifier

rpm -i certifier-x.y.z.i386.rpm1. /usr/local/certifier/ssh-ca-setup2.

6.

These steps should be carried out on secondary node only:

Install Certifierrpm -i certifier-x.y.z.i386.rpm1. /usr/local/certifier/ssh-ca-setup2.

1.

Run rm -fr /usr/local/certifier/* to remove the contents of/usr/local/certifier; this will be mounted from /dev/drbd1 when the node isactivated.

2.

Note that the /dev/drbd1 cannot be mounted on secondary node.

Example configuration for DRBD

Below is a sample /etc/drbd.conf file. It must be the same for both nodes.

2

2

global { usage-count no;}common { protocol C;}resource certifier { device /dev/drbd1; disk /dev/sdb1; meta-disk /dev/sdb2[0]; syncer { rate 40M; } on certifier-ha-1.my-domain.net { address 10.10.10.101:7789; } on certifier-ha-2.my-domain.net { address 10.10.10.102:7789; }}

Notes:

Protocol C means synchronized writing where primary waits secondary to complete writingoperation. If network latency is too big for this, protocol can be A (asynchronous). It has a risk ofdata loss however.

•

The syncer rate is recommended to be approximately 1/3 of the available bandwidth.•

Linux HA

Linux HA, http://www.linux-ha.org/, can be used to manage resources of HA cluster. With Certifier, theresources are:

DRBD resource; mainly primary/secondary status.• File system mount; /dev/drbd1 device mount to /usr/local/certifier.• Cluster IP address used by clients.• Certifier starting and stopping.•

The core of the Linux-HA is the heartbeat software. It supports simple resource management, but can alsobe used with more advanced resource managers like Pacemaker ( http://clusterlabs.org/wiki/Main_Page).These instructions are dealing with the simple manager that comes with the heartbeat.

The same node setup applies as specified in the DRBD setup instructions above. In addition, heartbeatcommunicates using UDP port 694, so that must be enabled in the firewall.

Do the following steps to setup Linux-HA:

Install heartbeat, e.g. yum install heartbeat. (Note: CentOS 5.1 required this commandto be run twice because of some problem in the rpm installation script.)

1.

Create heartbeat configuration files to /etc/ha.d/:ha.cf: cluster communication parameters.♦ authkeys: authentication keys for joining the cluster.

After creating authkeys, run chmod 600 /etc/ha.d/authkeys,otherwise heartbeat will not start.

◊ ♦

haresources: resources managed by heartbeat.♦

2.

Run /etc/init.d/heartbeat start to start heartbeat.3.

3

3

DRBD should be running before heartbeat starts, but the primary should NOT be set, it iscontrolled by heartbeat.

♦

After heartbeat is started, it takes a while to the primary node to claim the resources.♦

Certifier is managed by hearbeat using /etc/init.d/certifier script.

The simple resources manager in heartbeat does not automatically monitor resources, it only managesthem when node fails. To monitor Certifier, an external script must be used. Certifier versions startingfrom 4.1 provide a perl script for simple server availability checking by polling the HTTP services. Thescript file is /usr/local/certifier/bin/certifier-ha-monitor. It may be necessary tomodify it to suit the Certifier setup. The script notifies heartbeat to do a switch-over if the polling fails.Before using the script, it should be copied outside the /usr/local/certifier path. A good placeis /usr/lib/heartbeat/.

After a switch-over to the secondary node and repairing the primary node, the primary node can be madeactive again by stopping the heartbeat on the secondary node /etc/init.d/heartbeat stop andthen starting it over when the primary node has reclaimed the resources /etc/init.d/heartbeatstart.

Example Configurations

Configuration file ha.cf:

use_logd yesautojoin noneudpport 694auto_failback offucast eth0 192.168.1.101ucast eth1 10.10.10.101ucast eth0 192.168.1.102ucast eth1 10.10.10.102initdead 30warntime 10deadtime 20deadping 30keepalive 2node certifier-ha-1.my-domain.netnode certifier-ha-2.my-domain.netping 192.168.1.1respawn hacluster /usr/lib/heartbeat/ipfailrespawn hacluster /usr/lib/heartbeat/certifier-ha-monitor

This configuration specifies two nodes which communicate using unicast method via both interfaces.Notes:

auto_failback is recommended to be set off. This means that the failed primary node cannotanymore take over after switching to secondary node without manually setting it up again.

•

ping IP specifies an outside IP that is checked for network availability. There can be multiple IPs.• respawning the ipfail is doing the actual monitoring of the ping IP.• respawning certifier-ha-monitor MUST be set only to the primary node, because it can only workon active node.

•

Configuration file authkeys:

auth 11 crc

If used in insecure network, you can use stronger authentication keys:

4

4

auth 11 sha1 <key-for-sha1-any-text-you-want>

Configuration file haresources:

certifier-ha-1.my-domain.net \ drbddisk::certifier \ Filesystem::/dev/drbd1::/usr/local/certifier::ext2 \ 192.168.1.10 \ certifier

The host name specifies the primary node. The resources are in order of start-up. Shutdown is done inreverse order. The resources are:

DRBD replication• File system mount /dev/drbd1 -> /usr/local/certifier• IP address assigned to the active node• Certifier; the resource name maps to init script /etc/init.d/certifier.•

5

5