DRPTechnicalDocumentation

ICT Continuity Program Management and DR Plan: BS ISO/IEC 27031:2011 Code of Practice

Authors: Azaam Alfi

Title: ICT Continuity Program Management and DR Plan:

BS ISO/IEC 25777 Code of Practice Page 1 of 40

Version: 2.1

Date: February 24, 2015

Status: Published

Copyright © Plan B disaster Recovery Plan 2010. Confidential

Please note this DR Plan template is the Copyright of Plan B Disaster Recovery Plan. It may be used to create a customised DR plan for your own organisation or passed on to other organisations in its original format with this copyright notice. It may not be sold or otherwise used for commercial gain or as part of a commercial offering.



Version: 2.1


Status: Published


Contents Introduction 4

Appendix E – Technical Documentation 4

Summary 4

MySQL native data replication 5

“Unison” 2-way file synchronization 7

Monitoring script 13

“Namecheap” dynamic IP web service 17

Version control for project files and server configuration files 36

Version control for databases 37

Appendix G – Infrastructure Diagram 39



Version: 2.1


Status: Published


Version History

V1.0 Azaam Alfi - October 14, 2014

V2.0 Azaam Alfi - February 11, 2015

V2.1 Azaam Alfi - February 24, 2015

Introduction ICT continuity management supports the overall business continuity management (BCM) process of an organization. BCM seeks to ensure that the eaa processes are protected from disruption and that eaa is able to respond positively and effectively when disruption occurs. eaa sets out its BCM priorities and it is within this context that ICT continuity management activities take place. ICT continuity management ensures that the required information and communications technology and services are resilient and can be recovered to predetermined levels within timescales required by and agreed with directors of eaa. Thus, effective BCM depends on ICT continuity management to ensure that eaa can meet its objectives at all times, particularly during times of disruption. To be successful, both BCM and ICT continuity management have to be embedded within the culture of eaa.

Appendix E – Technical Documentation

Summary



Version: 2.1


Status: Published


Our goal here is to synchronize all files which include code and other files such as uploads and images necessary to run to all projects which are web applications and MySQL data in real time between the live server and the backup server. This is required incase our live server goes down or loses data. Our backup server will need to be hot all the time which means that anytime we need to close the live server and use the backup server it need to be up and running with the most current version of data and files.

There are 4 major components I have worked on that you MUST understand:

MySQL native data replication “Unison” 2way file synchronization Monitoring script “GoDaddy” A records Domain configuration

MySQL native data replication

Data replication here is for the 2 identical databases in the live and backup server. Basically replication here is applying any change made to 1 database or database server instantly to the other database or the database server. For example, one record is added to table “A” in the live server. That same record is instantly added to table “A” in the backup server through the MySQL replication.

Replication has a “Master” database and a “Slave” database the changes made on the “Master” are applied on the “Slave” but the changes made on the “Slave” are not applied to the “Master”. Having 1 database as “Master” and the other as “Slave” is called a “Master to Slave” relationship.



Version: 2.1


Status: Published


You can learn more by clicking these links:

http://en.wikipedia.org/wiki/Replication_%28computing%29#Database_replication

http://dev.mysql.com/doc/refman/5.0/en/replication.html

I have went a step forward and created a “Master to Master” relationship in which 1 database is both the master and the slave and the other is also the master and the slave. This means if a change is made to on the live it will instantly be applied to the backup server. Also if a change is made on the backup server then it will be replicated instantly on the live so its a “2way” replication.

setting up a master to master relationship

I have this article to set up the master to master relationship:

https://www.digitalocean.com/community/tutorials/howtosetupmysqlmastermasterreplication

If you are working Centos Linux distribution use “server_id” instead of “serverid” and make changes under the [mysqld] line. Also the configuration file in Centos is usually in /etc/my.cnf if not enter command locate i my.cnf to find it.

In the my.cnf file it is recommended to remove or comment out the binlog_do_db to use replicatedodb to filter out what gets read by the slave. This always for the use of multiple databases. the “binlog_do_db” seems to support 1 database. Please note that this will filter databases only read from the master. Look here for more information on the “replicatedodb”:



Version: 2.1


Status: Published








https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication




http://dev.mysql.com/doc/refman/5.0/en/replicationoptionsslave.html#option_mysqld_replicatedodb

Troubleshooting MySQL replication

Steps 2 and 3 in the article above may need to be reapplied if servers lose

connection. make sure the you can enter as “replicator” from live to backup and from

backup to live

For example, in the live server type this command # mysql u [replicator username] p[replicator password] h[backup server IP/domain] it should log you in..

make sure firewall on backup server does not block incoming connections on

tcp port 3306 (mysql port) from the live server and vice versa. login to mysql and type “show slave status” and read any error messages

carefully and maybe try googling it. Do not forget to test replication changes on both servers after resolving

issues.

“Unison” 2-way file synchronization

I have installed and configured “Unison” as the file synchronization program. This is a 2way file synchronization program and works similarly to the “Master to Master” relationship described above except that is related to files and changing their content not databases. Here is the article I used to set both servers on:

http://xmodulo.com/synchronizefilesbetweentwoservers.html



Version: 2.1


Status: Published


http://dev.mysql.com/doc/refman/5.0/en/replication-options-slave.html#option_mysqld_replicate-do-db




http://xmodulo.com/synchronize-files-between-two-servers.html



I use the ssh to make the the transfers as it is more secure but needs to keep connecting and disconnecting to server also ssh encrypts the files between transfers unlike using a direct socket connection to the server so keep it this way.

Please make sure that unison is installed on both the backup and live servers with the SAME version as I have used different versions and files could not be transferred. Even if files do transfer we don’t want what issues we will be facing if we different versions are used so don’t risk it. After installing unison on BOTH servers check the version by entering this command:

# unison version

At the time of this writing the most up to date stable version I am using is:

unison version 2.40.102

You can follow the answer in this post to install it on the centos: https://www.digitalocean.com/community/questions/installunisonincentos7

You can download version 2.40.102 from here: http://pkgs.fedoraproject.org/repo/pkgs/unison240/unison2.40.102.tar.gz/42caf869f40440ed0e75eef07a605b2e/unison2.40.102.tar.gz

Configuration



Version: 2.1


Status: Published




https://www.digitalocean.com/community/questions/install-unison-in-centos-7



http://pkgs.fedoraproject.org/repo/pkgs/unison240/unison-2.40.102.tar.gz/42caf869f40440ed0e75eef07a605b2e/unison-2.40.102.tar.gz




Unison will be running ONLY from the backup server so configuration is at ONE server only. I have placed the configuration for one profile (you can create more profiles if you need them) while logged in as root here:

~/.unison/sync_live.prf

Here is the profile file:

https://drive.google.com/a/eaa.com.sa/file/d/0B_VFtr2nWTnuaktfdDhocHdMRHc/view?usp=sharing

The first line the local root path and the second is server root path which should be in this format ssh://[username]@[host]/[absolute path to the root]. You need to have keybased passwordless ssh access to the server (which means access the server without entering its password) read this article here if you did not do this already or just need to understand it:

https://www.centosblog.com/passwordlesssshkeybasedauthentication60secondscentoslinux/

path = test_project1/uploads/ path = test_project2/uploads/ path = test_project3/uploads/

These are the sub directories under the local root that will be checked which mostlikely will be the uploaded files.



Version: 2.1


Status: Published








https://www.centosblog.com/passwordless-ssh-key-based-authentication-60-seconds-centos-linux/




#show text interface instead of graphic

ui = text

Unison has a Graphical User Interface (GUI) and we don’t want to show it as file synchronization should automatically be handled by the server. Even though I place this option in the file I don’t know why it still shows the GUI so I still have to include this option when running the sync by including:

ui text

Running/testing

I run the unison ONLY on the BACKUP server since I want to try to reduce load on the live server. So the backup server will be checking changes from within itself and from within the live server but this still means that synchronization is 2way or bidirectional. To manually run the Unison program I enter this command:

# unison ui text sync_live

You can add a file on 1 server run the program and see it present on the other server. You can also delete files and change their content to see the changes applied at the other server.

Please note that when the last file in the folder is removed on 1 side it will not be removed on the other side unless another file is added on the side it was removed then it will delete the file deleted on the second side and add the new file to be synchronized.



Version: 2.1


Status: Published


Automating the process

Ok now we understand what unison does and how to synchronize the files, so now its time to make the server handle it all by itself. At a very basic level, unison requires 2 things to run, a connection from the backup to the live server and access to ssh port (tcp port 22) so we need to check that both are open before scheduling a run to the unison which will be made every minute. Here is the bash script used to run unison every minute ONLY IF server is up and port 22 is open:

https://drive.google.com/open?id=0B_VFtr2nWTnuMHZhV0JqVHlBajg&authuser=0

One note to mention…

server_status_file="./server_status.txt"; ssh_service_file="./server_ssh_service.txt";

The script basically checks if one these files has “off” as their content and if it does not then it will schedule to run unison the next 1 minute and places a timestamp in a text file when the last run was schedule which is 1 minute ahead of the time stamp.

You can easily stop scheduling by changing the content of 1 of the 2 files mentioned above to “off” and the next minute will be unison will not run. You can then start it after stopping it by just running by first making sure script is executable by running this command:

chmod +x ./unison_sync.bash



Version: 2.1


Status: Published


Then making sure that BOTH files mentioned above are set to anything BUT “off” or do not exist (as they will created with “on” as their content if they do not exist.

Also a program called “atd” must be running or else an error like this will show: job 3 at 20150127 07:30

Can't open /var/run/atd.pid to signal atd. No atd running?

job 4 at 20150127 07:30

Can't open /var/run/atd.pid to signal atd. No atd running?

The resolution can be found here:

http://ubuntuforums.org/showthread.php?t=1115274

Just run the “atd”:

sudo atd

Then running the script:

./unison_sync.bash

These files are controlled by the monitoring script as well which is mention in the next section..

Monitoring script

We need to be vigilant on issues that occur and notified immediately if anything happens. The backup server will have a constant eye on the live server. I have made



Version: 2.1


Status: Published





a monitoring scripts that runs every minute and checks 4 things the “nmap” (which must must installed before hand) command:

If the server is running or if connection to the server from the backup could

not be made if the the apache web server is running (by checking tcp port 80) if the MySQL database server is running (tcp port 3306) if the SSH server is running (tcp port 22) which is important for unison to be

running

Requirements before running

Make sure that the backup server has access to all ports mentioned above as well as the server itself you can use this command to check the status:

# nmap p 22,80,3306 [server_address]

If you do not have access to the server or to 1 or more of the ports mentioned above please check the incoming connections on the firewall on the live server.

The script will use the mail command so make sure you can send an email from the BACKUP server by entering this command:

# echo “the email body here..” | mail s “just a subject” $USER

You can then check the local mailbox by entering this command:

# mail



Version: 2.1


Status: Published


In the home directory (~/) of the user that will run the script there should be a .forward file which contains the email addresses that will receive emails sent by the monitoring script. The emails will be seperated by “; “ the .forward should like this:

[email protected]; [email protected]; [email protected]

Please test the mail program and check if emails are sent to all emails specified in the .forward file. They may enter as spam and if they do Gmail can filter them back to inbox.

Finally, make sure that this bash script file (which the monitoring script) is executable by entering this command:

# chmod +x ./check_nmap.bash

Responding to email alerts

If any fault is detected an email will be sent. Lets say for instance an email was sent that the live server is down because the internet connection for the live server was disconnected. I would connect the internet back on for the live server make a quick ping from the live server to google by entering this command:

# ping google.com

Then I would go to the backup server and make sure it can see the live server by also pinging to it and if ping was successful I would cd into the monitoring script path and change all these files content to “on”

./control.txt file (this file stops the monitoring script)



Version: 2.1


Status: Published


./server_status.txt (stops the unison synchronization)

./server_ssh_service.txt (stops the unison synchronization)

Run the monitoring bash script then run the unison bash script which BOTH need to be in the SAME path.

Here is the monitoring script:

check_nmap.bash



Version: 2.1


Status: Published


https://drive.google.com/a/eaa.com.sa/file/d/0B_VFtr2nWTnucS1Ga0tkdmVxOEU/view?usp=sharing











Version: 2.1


Status: Published



“Namecheap” dynamic IP web service

After conducting some tests with 2 “A records” pointing to 2 different IPs in one DNS Zone file on Godaddy, there were some technical failures with this scenario as some locations pointed to the backup server even though it was down and some pointed the primary server as expected and this presented a performance issue for some local networks as the backup server is much slower than the primary. Also there were times where the site would show “DNS error”. After some careful testing and research, the resolution for this was to setup a “Dynamic IP” web service which basically changes the IP automatically to the backup server and back to the live through a web service run by script. I have found and used a free dynamic IP web service called “Namecheap FreeDNS” (which is apparently free as the title states) located here:

https://www.namecheap.com/domains/freedns.aspx

Configuring Godaddy to use namecheap nameservers and service



Version: 2.1


Status: Published





The first step I did was signed up with the FreeDNS service and followed the instructions of the tutorial located here (with some adjustments in the sequence):

https://www.namecheap.com/support/knowledgebase/article.aspx/536/51/howdoisetmydomaintousenamecheapsfreednsservers

Regardless of the sequence shown above, the first step would be to go to

Godaddy and sign in to the account containing the domains. On the home

page click “My Account” at the top.



Version: 2.1


Status: Published


https://www.namecheap.com/support/knowledgebase/article.aspx/536/51/how-do-i-set-my-domain-to-use-namecheaps-freedns-servers






Version: 2.1


Status: Published


A top bar drops down, from within it click the “Visit My Account” button.



Version: 2.1


Status: Published


In the account page click the “+” button in the “Domains” tree item.



Version: 2.1


Status: Published


The tree item will expand. Next to the domain you need to configure, click the

“Launch” button.



Version: 2.1


Status: Published




Version: 2.1


Status: Published


In the “Settings” tab click manage in the “Nameservers” row.



Version: 2.1


Status: Published


In the following popup choose the “Custom” radio button option.



Version: 2.1


Status: Published


Click “Enter custom nameservers” to show a new popup with 2 text fields.



Version: 2.1


Status: Published


Input the first 2 text fields:

freedns1.registrarservers.com


Then click “ADD NAMESERVER” 3 times to have 3 more text fields appended.



Version: 2.1


Status: Published




Version: 2.1


Status: Published


Fill in the the last 3 text fields to complete the sequence of nameservers from 1 to 5:




Then click OK to save them to the first popup.



Version: 2.1


Status: Published




Version: 2.1


Status: Published


Confirm your input then click “Save” to update the nameservers.



Version: 2.1


Status: Published


On the top left, a confirmation box should inform that the update has been been submitted.

PLEASE NOTE: In the nameservers row the previous records will show. Just wait 5 minutes then refresh to see if they were updated.



Version: 2.1


Status: Published


The refreshed page should then show the updated Nameservers.

Configuring “Namecheap” for each domain

Now configure namecheap to correctly as instructed here:



Version: 2.1


Status: Published


https://www.namecheap.com/support/knowledgebase/article.aspx/536/51/howdoisetmydomaintousenamecheapsfreednsservers

Also enable the Dynamic DNS for the domain as instructed here:

https://www.namecheap.com/support/knowledgebase/article.aspx/595/11/howdoienabledynamicdnsforadomain

Installing “DDclient” and configuring to use with “Namecheap”

Install the “DDclient” on the Linux Centos as this will change the IP to backup and back to live appropriately when called. Here is the instructions to install the “DDclient”.

https://techjourney.net/installddclientdynamicdnsddnsupdateclientincentos/

Now we need to configure the ddclient on Linux Centos and this here is the specific configuration for namecheap:

https://www.namecheap.com/support/knowledgebase/article.aspx/583/11/howdoiconfigureddclient



Version: 2.1


Status: Published





https://www.namecheap.com/support/knowledgebase/article.aspx/595/11/how-do-i-enable-dynamic-dns-for-a-domain



https://techjourney.net/install-ddclient-dynamic-dns-ddns-update-client-in-centos/



https://www.namecheap.com/support/knowledgebase/article.aspx/583/11/how-do-i-configure-ddclient



This is one configuration file I am using to change the DNS records for the domain and subdomains for myrubrics.com: myrubrics.com.conf

Here are the 2 scripts that configure DNS to the live server and to the backup server for all the domains at once:

run_live_server_online.bash

run_live_server_offline.bash

Testing automatic redirection

You can test that the secondary IP will be instantly pointed to by correctly setting up the apache web server on both servers for a domain. Then you can simply stop the apache web server on the live server or simply turn it off to see the site redirect. Please note that if you don’t see it redirect empty your local browser’s cache and refresh or open it from another browser.



Version: 2.1


Status: Published


https://drive.google.com/a/eaa.com.sa/file/d/0B_VFtr2nWTnucVhvRndXYUozdGM/view?usp=sharing

https://drive.google.com/a/eaa.com.sa/file/d/0B_VFtr2nWTnucVhvRndXYUozdGM/view?usp=sharing

https://drive.google.com/open?id=0B_VFtr2nWTnubmY1aUIwN1UwSmM&authuser=0

https://drive.google.com/open?id=0B_VFtr2nWTnubmY1aUIwN1UwSmM&authuser=0

https://drive.google.com/open?id=0B_VFtr2nWTnuX0RDX3g0bUtrb1E&authuser=0




Version control for project files and server configuration files

While the server is running on a 2 way real time synchronization any file changes made on any server will affect the other. There must be an additional control measure outside of the synchronization. There is a rsync synchronization script that will rotate the last 7 daily versions for a set files within a path:

Documentation for the script: http://www.linuxmagazine.com/Online/Blogs/ProductivitySauce/RotatingBackupwithrsync

http://www.noah.org/wiki/Rsync_backup

The original script: http://www.noah.org/engineering/src/shell/rsync_backup

I have made some changes which includes compressing and archiving the script in a tar archive compressed with gz (.tar.gz): https://drive.google.com/open?id=0B_VFtr2nWTnuaURkeC10NndEREU&authuser=0

I have also made a paths_script.bash with in the directory for backup and rotation to define all paths to be backed up: https://drive.google.com/a/eaa.com.sa/file/d/0B_VFtr2nWTnubnQzYVVkOUJFQkE/view?usp=sharing



Version: 2.1


Status: Published


http://www.linux-magazine.com/Online/Blogs/Productivity-Sauce/Rotating-Backup-with-rsync






http://www.noah.org/engineering/src/shell/rsync_backup



https://drive.google.com/open?id=0B_VFtr2nWTnuaURkeC10NndEREU&authuser=0




https://drive.google.com/a/eaa.com.sa/file/d/0B_VFtr2nWTnubnQzYVVkOUJFQkE/view?usp=sharing




A cron job (preferably nightly cron) is needed to run the paths_script.bash file: # crontab e

then add this line and save the file: 30 15 * * * /var/backup/files/paths_script.bash > /dev/null 2>&1

The trailing > /dev/null 2>&1 will prevent the cron from sending outputs as emails.

Version control for databases

While the server is running on a 2 way real time replication any database changes made on any server will affect the other. There must be an additional control measure outside of the replication. There is a database backup rotation script called automysqlbackup that has all needed functionality for backing up databases and rotating them.

I have used “automysqlbackup” which is a very popular mysql backup and rotation with many useful and sufficiently tested features.

Here is the script: http://sourceforge.net/projects/automysqlbackup/

Here is the documentation for the script: http://www.linux.com/learn/tutorials/659107howtodopainlessmysqlserverbackupswithautomysqlbackup

https://drive.google.com/open?id=0B_VFtr2nWTnuOGJialE3SVlKOFE&authuser=0



Version: 2.1


Status: Published


http://sourceforge.net/projects/automysqlbackup/




http://www.linux.com/learn/tutorials/659107-how-to-do-painless-mysql-server-backups-with-automysqlbackup






The configuration file that I have will give you more information about the script: https://drive.google.com/open?id=0B_VFtr2nWTnuUDRnTVRUVGNKbnM&authuser=0



Version: 2.1


Status: Published



https://drive.google.com/open?id=0B_VFtr2nWTnuUDRnTVRUVGNKbnM&authuser=0





Appendix G – Infrastructure Diagram



Version: 2.1


Status: Published





Version: 2.1


Status: Published


Documents

DRPTechnicalDocumentation