Gabriella Davis Paul Mooney - Turtle Partnership Blogblog.turtleweb.com/turtleblog.nsf/dx/Build A Domino Cluster.pdf... · Gabriella Davis Paul Mooney 1 ... ─ e.g. Neverfail

© 2011 IBM Corporation

How to build a better cluster - from a standing start

● Gabriella Davis● Paul Mooney

1


About us● Gabriella Davis

─ The Turtle Partnership─ Technical Director─ Security, System Design, Deployment of Lotus software since 1989─ Yes, I’m much older than Paul but feel free to be surprised by that

● Paul Mooney─ Bluewave Technology─ Senior Architect─ 14 years experience with Domino─ Geek / Biker / Jogger

2


Let’s get legal!● This slide presentation may contain the following copyrighted, trademarked, and/or restricted terms:● IBM® Lotus® Domino®, IBM® Lotus® Notes®, IBM Lotus Symphony®, LotusScript®● Microsoft® Windows®, Microsoft Excel®, Microsoft Office®● Linux®, Java®, Adobe® Acrobat®, Adobe Flash®

3


Simple Math...● We have 250 slides● We have 105 minutes● 250 /105 = 2.3 SPM (slides per minute)● Oh. S*&t.

4


What are you going to learn today● What is a cluster and why you want it● How does it work in the Domino world● Other types of clustering that aren’t as good● How to set up Domino clustering correctly● How to monitor and tune your new cluster environment● How clustering behaves for clients● Some examples of cluster topologies● Things to watch out for● If we have time

─ How to juggle─ How to attend a session at lotusphere that does not include the word “strategic”─ How the cloud is evil─ How your mother never loved you─ How to win the lottery

5


How are we going to teach you?● A Bit Of “Tell” before the “Show”● Then the “Show”

─ Starting with a single domino server (one we built earlier)─ Register an additional server─ Install the additional server─ Setup clustering─ Test clustering from a client, let’s take a server down live!─ How to tune your cluster to do what you want not what it wants

● Terrible Terrible things that can happen if you don’t heed our advice

6


Our Dilemma● We want to step you through all our tricks for setting up a cluster LIVE● We want to make sure you have all the information you need to complete the

work yourself when you get back to the office

● To get all this content over we’re going to be covering a lot of ground very quickly

● That’s exciting, isn’t it !

7


What Is A Cluster And Why Do You Want It?

8


So What’s This All About?● Wouldn’t it be great when a server gets very busy or goes offline

─ if your users don’t notice─ If mail doesn’t backup queuing for delivery on an offline mail server─ If out of office still works─ If mail still routes─ if everything doesn’t get r-e-a-l s-l-o-w─ If your Blackberry or Traveler server automatically knows to look elsewhere

● Altogether now... Yes it would!

Follow us into this magical world called clustering

9


What Is Clustering?● Grouping together of resources so they appear as one resource instead of

many to the user

● Creating a single virtual server experience

● In Domino Terms─ Separate Domino servers that operate individually─ Each server is ‘cluster aware’ i.e. it knows it belongs to a cluster and what the other

servers are─ Each cluster aware server shares its data in real time with the rest of the cluster─ The configuration of each server is separate and independent (everything from

hardware and OS to Domino Tasks)

10


What Other Types Of Clustering Are There?● The sharing of disk and storage

─ Single point of failure being the disk─ RAID isn’t clustering and neither are SANs

– despite what hardware companies will tell you● MS Clustering Active/Active – Active/Passive

─ Can’t do load balancing only failover─ Must be identical OS, software and versions─ All or nothing – everything must be clustered

● Bit-level clustering─ e.g. Neverfail─ Active/Passive─ Data transferred beneath OS level to passive box

11


Why Use Domino Clustering vs OS Level● More flexible

─ Cluster different hardware, Win32, Linux, AIX together─ Partially cluster selected data─ Locate cluster servers anywhere with sufficient bandwidth connecting them

● More stable─ Load balancing means that users failover within the cluster before a server is

pushed to its limits─ If an entire server fails there is an entire duplicate of data and services waiting

12


How Does It Work In The Domino World?

13


Clustering In Domino “Just Works”● Yes it does, it’s smart, it learns and dynamically adjusts itself to the current load● It’s incredibly useful and self maintaining ● Temptation is to not work with it until you need it but that would be a mistake● One of the single best features of Domino

─ Now available for Express Customers too (you lucky, lucky people)─ Often implemented poorly and ignored or not implemented at all

● The beauty of Domino clustering lies in being able to selectively choose which databases you want to cluster across which servers, you don’t have to create mirror copies of each server

14


What is clustering?● Replication’ish

─ It is NOT replication and doesn’t use the same processes or threads─ It is event driven and in memory

– not scheduled─ It’s designed to be very fast so it doesn’t waste time and just sends everything

● Single, one-way push of data writes from one server to another─ Since 6.5.3 reading a document can be a “write”

– So unread marks can cluster correctly● Occurs almost immediately

─ Depending on server load but should be less than 10 secs● Data entered in server A goes to Server B

─ Update is sent by Server A’s cluster replicator● Data entered in Server B goes to Server A

─ Update is sent by Server B’s cluster replicator● Been around since R4x days and constantly enhanced and improved

15


What is clustering?● Version independent

─ Clustering an R6.x server with R8.x server works● Platform independent

─ You can cluster from and between Win2k, Win2k3, Win2k8, Linux, AIX, OS400, Solaris, etc etc

● Stunning feature of domino─ No other product Gab and I have ever come across does clustering so well

16


Components of clustering● Cluster Manager● Cluster Replicator (clrepl)● Cluster Database Directory (cldbdir)● Cluster Database Directory Manager ● Cluster Administrator

17


Cluster Manager● Not a visible task● What servers are in the cluster● How busy are those servers● Activating workload balancing● Managing and logging failover● Directing client connections

18


Cluster Replicator● Cluster Replicator task

─ CLREPL● It is not a listed task in SERVERTASKS line by default

─ Since R7 days– It can be there though (especially if it is older or customised)

─ Cluster Manager will load it automatically if the server is a cluster● You can have multiple cluster replicator threads

─ Typically equal to total members in a cluster less one─ Monitor cluster statistics to see if you need to do this (more later)

● Lotus supports up to 6 servers in a cluster─ You can have more─ Do you really need more?

– Let us answer this for you. No.

● Changes to be sent to cluster mates are held in memory by the CLRepl Task ─ Tell CLRepl Quit loses in-memory changes─ Tell CLRepl Pause / Resume retains and accumulates in memory changes

19


More About Cluster Replication● 8.0.2 and onwards

─ Streaming cluster replication─ Data transfers from buffer of domino server to cluster mates─ Doesn't wait to write to disk, then write to other servers─ Faster─ Broken (supposedly fixed in 8.0.2 FP1 and disabled in 8.0.2)

● DAOS and clustering─ One or both servers can be using DAOS─ If both are DAOSified

– Both have their own .NLO store─ If you have DAOSified cluster servers over 8.5.1

– Cluster replication is DAOS aware, it won’t resend a file if it already exists on the destination server

– CLREPL will check to see if the attachment already exists before sending– MASSIVE performance gain in clustering and reduction in network traffic

20


Cluster Database Directory● Cluster directory task

─ cldbdir● Creates and maintains the cluster database directory

─ cldbdir.nsf● One document per database per server

─ Each document contains the replica id of the database, the server where it is located and its cluster “state”

● CLDBDir.Nsf is replicated throughout the cluster ─ Maintains a list of all replicas on each cluster server─ If a server replica is not listed there... it WILL NOT cluster as the Cluster Manager

won’t know to failover a user requesting that database to that server─ The ClRepl tasks uses the Cldbdir.nsf db to verify which databases are to be

clustered and where to failover users requesting that database─ Can be used to mark a database as “out of service”

– Force a failover

21


Cluster Administrator● Not a visible task● Adds CLRepl, CLDBDir to the servertasks when a server is added to a cluster● Sets up calendaring for the cluster● Starts the clustering tasks on the server● Starts Adminp if it isn’t already● Reverses all those changes when a server is removed from a cluster

22


Components Work Together● Cluster Database Directory keeps a record of all databases and their state● This verifies if a db is in place on a server and if it’s in an available state for user

access ● The Cluster Database Directory Manager keeps the ClDDDir.nsf up to date● The Cluster Replicator ensures all data changes on any server are pushed to

the other clustermates immediately

23


Calendaring In A Cluster● Usually Domino maintains freetime information for users on their home server in

the file busytime.nsf● If you want to check someone’s schedule the server uses busytime.nsf to verify

availability● Since users can be on varying servers within a cluster and don’t have a single

‘home’ the busytime.nsf can’t be used● Busytime.nsf on individual servers becomes Clubusy.nsf replicated across all

servers in a cluster● Busytime.nsf is deleted from the server when it is added to a cluster by the

Cluster Administrator● Clubusy replicates using the cluster replicator tasks but should also be included

in standard replication schedule as backup

24


What “isn’t” clustering● It isn’t replication● Replication formula is ignored by clustering

─ Acting as designed● Premise is that database cluster replicas should ALWAYS be identical

─ Replication formulae could prevent that─ Scheduled replication will honour replication formulae

● It isn’t good at managing deletion stubs● It’s in memory changes only

─ If you quit the clrepl task then those in memory changes are lost─ Always have a backup replication document between server mates in a cluster

– but no more often than every 60 mins● It works for NRPC calls only

─ Notes clients─ Domino servers─ It is NOT web clustering

– We get to that later

25


How To Set Up Domino Clustering Correctly The “Show” Bit

26


In the beginning● Single Domino server - atlantic/svr/demo● DNS name - atlantic.snt-track.com● Version of Domino - 8.5.2 FP1● OS - Windows 2003● Five mail files registered● No implementation of anything - DAOS etc● Default configuration document created

● Additional machine that will be the cluster - pacific/svr/demo─ pacific.snt-track.com─ Windows 2008

27


Registering a new server in the domain

28

● Launch the administration client● Go to the Configuration tab

● Click on Registration● Select Server



29

● Choose the atlantic as the registration server● Select Server● Navigate to the atlantic/svr/demo server

● Do NOT use local● Select the Certifier id file

● Click Certifier id● Navigate to the servers OU id file● Click on OK● Click on OK

● Enter the password



30

● The register server dialog box appears● Registration server● Should still be atlantic● Certifier● Should still be /svr/demo● Click on Continue



31

● The “Register new server” dialog box opens● Enter Pacific as the server name● Enter title if desired● Domino domain = Demo● Click on the icon● To select administrators● Choose your administrator

group● Select OK



32

● IN PRODUCTION● Enter a password for the server id file● Server id files SHOULD have passwords● Keep password blank● Storage of your id file

● Uncheck “In Domino Directory”● Check “in file”● Select the desktop

● Select the icon ● Click on REGISTER ALL● Status bar updates



33

● The server should now be visible in the domain


Let’s Start By Installing Our 2nd Server● We’re going to install Domino 8.5.2 32bit in this instance

34


Run the installer file ● If you are running Windows 2008, run the installer as administrator

35


Security Warning● The installer is an executable which will trigger an OS alert

36


Extract install files to a directory● If you select default you will end up with 550+MB of install files in your server

temp directory - you may want to clean those up!─ Paul’s tip - I typically have a “software” folder created on every server that contains

all software installed on that server (including rollback software)– Yes it uses up space, but it gives Paul a warm fuzzy feeling

37


The Installer Will Now Run● This should take only a few minutes (<10)

─ If it takes longer, unless your hardware is very old, there may be something blocking the install

– such as evil virus protection. ● If you are upgrading and the install fails, existing files may be still in use

─ Do a clean reboot and try again

38


The Installer GUI● There is also a console installer for Windows

─ On Linux there is also an optional GUI installer ● Click on NEXT

39


License Agreement● Accept the license● Click on NEXT

40


Installation Directory For Program Files● When installing a 32bit application (as we’ve chosen to do here) on a 64bit OS

the default is \program files (x86)─ We don’t want that - spaces, brackets, long directories. bad!

● Click on Browse

41


Better Installation Directory● Change the install path to ‘c:\IBM\Lotus\Domino’● Click on NEXT

42


Data Directory● Domino assumes the data directory will be a subdirectory called “data”

underneath the program directory● You can accept that or choose another directory or even drive● Click on NEXT

43


Server Type - Enterprise● If you want to cluster your server then you must have an enterprise server

license or CEO entitlement● You can choose “Customize” and select specific Domino features to install,

including clustering● Click on NEXT

44


Confirmation Screen● Read this carefully to confirm you have made the right selections● Once installation is complete these settings are written to the registry● Click on NEXT

45


Installer Running

46


Installer Complete!

47


Before Going Further - Let’s Install A Fixpack● It’s tempting to jump right in, but a few minutes now to install the latest fixpack

could save you hours later

48


Running the Fixpack Installer● Again it will extract to a temporary directory● Click on RUN

49


License Agreement● Select ACCEPT● Select NEXT

50


Confirming Install Directories● If you have partitioned servers there will

be multiple data directories to choose from─ You will have to run the updater for

each data directory / partition you want to update

● Select the DATA directory and click on NEXT

51


First The Installer Verifies Source Version● You can only use the incremental installer to update from a specific version

─ If you aren’t on 8.5.2 you can’t use the 8.5.2 update installer● Click on NEXT

52


Installer Complete● Click on CLOSE

53


Starting Domino With Java Controller● On Windows 2008 there is no visible Domino console● Modify the properties of the desktop icon created by the installer● Right click on the shortcut and select

PROPERTIES

54


To Use Java Controller● Select the SHORTCUT tab● Add -jc to the program line eg c:\ibm

\lotus\domino\nserver.exe -jc● Select APPLY● Select OK

55


Starting Domino The First Time● We will need to configure it so we

need on screen dialogs● Right Click on the Shortcut and

select RUN AS ADMINISTRATOR

56


It begins

57


First Setup Screen● Click on NEXT

58


Choose “Set Up An Additional Server”● You already have one server in your organisation● It will expect you to already have a server ID● It will expect to be able to connect to another server to get the configuration

59


Select A Server ID File● Click on BROWSE

60


Select A Server ID File● Copied onto the server from the

workstation where we generated it

● Select the pacific server ID● Click on SELECT

61


Confirmation Of Server Name● The installer reads the ID file you gave it to find the server name

─ Note it’s not editable at this point as the ID file already exists● Click on NEXT

62


What Services Do We Want The Server To Run● LDAP will be selected by default, you can deselect it

─ and should if you don’t need it● Don’t choose options you don’t need immediately

─ you can always enable them later● Click on NEXT

63


Configuring TCPIP● These settings are written to the server document via the Adminp task● There must be a fully qualified hostname and if there is an OS configured one, it

will be selected by the installer● Click on CUSTOMIZE

64


Modifying TCPIP Settings● Both the hostname and fqhn must resolve on the server● The server will use the hostname (or ipaddress if you enter one) to find itself● Use a local hosts file to ensure it does resolve correctly● Enter the correct hostname in the HOST NAME field● Click on OK

65


Confirmation Of TCPIP Settings● Click on NEXT

66


Connect To Another Server For Configuration● This can be the server you

intend to cluster with but doesn’t have to be

● Usually we select the Admin server for the Domain to minimise the risk of replication conflicts in the server document

● Enter ATLANTIC/SVR/DEMO in server name

● Enter ATLANTIC in optional network address

● Click on NEXT

67


Directory Information● Choose the same configuration

as on the server you want to cluster with

● 99% of the time, PRIMARY DOMINO DIRECTORY

● Click on NEXT

68


Security Setup● Leave as is● Anonymous accounted will be added with No Access to all databases/templates● LocalDomainAdmins will be added with Manager access to all databases/

templates● Select NEXT

69


Confirmation Screen - Verify Details● Select SETUP

70


Files Installing● The installer will fail if

─ it can’t find the “Atlantic” to find its configuration─ you already have files in place from a previous failed install

71


Ta-da!

● Select FINISH

72


A Brief Diversion Into Dangerous Territory● When installing under Windows 2008 we want to alter the install Domino

service to run Domino as java controller ─ We have to modify the registry by using regedit (be v. careful!)

● Click on Windows Start button● Type in REGEDIT● Press Enter

73


Search For The Service Name● Click on Edit/Find● Type in Lotus Domino Server● Click on Find next

─ It will bring you to the service details

74


When You Find It - Double Click On ImagePath● The registry reference IMAGEPATH will be under the Lotus Domino server

registry directory.● Double click on it

75


Add -jc To The End Of Value Data● Add -jc to the end of the Value field● Click on OK

76


Now Start The Domino Service● Double click on the Lotus Domino server icon● Start as a service● Console will appear

77


If It Goes Wrong● You can always delete , modify or add a new Domino service using the file

ntsvinst in the Domino program directory

78


Testing Connectivity Between Servers● We assume nothing

─ Always test that:– Clients can connect to servers– Servers can connect to servers

● Both server names from resolve in the DNS from each server so─ Typing “Pacific” from the OS of Adriatic/Demo should return the correct ip for Pacific─ Typing “Adriatic” from the OS of Pacific/Demo should return the correct server ip for

Adriatic● But each server should also be able to resolve itself.

─ Often the server needs to connect to itself, most notably when – Delivering mail– Running agents

─ It first tries the setting in TCPIP address on the server document, then tries the FQHN on the basics tab

– Typing “Pacific” from the OS of Pacific/Demo should return the correct server ip for Pacific (itself)

– Typing “Adriatic” from the OS of Adriatic/Demo should return the correct ip for Adriatic (itself)

79


Curse you Windows!● There have been persistent problems with Windows where it will simply “forget”

its own hostname─ If you use a hostname in the TCPIP settings in the server document as the address

of the server─ Even if the hostname still resolves outside of Domino─ Side effects include agent based mail not sending or even attempting to send─ If you suspect this is your problem you can confirm it by typing

– Trace xxx/xxx where xxx/xxx is the server name itself– If it fails to connect to itself you’ll need to restart Domino

● A regular Domino restart would usually avoid this happening (once a month)─ This was very common in older Windows OS versions

80


Ensuring Your Admin Client Can See The Server● Your server is registered, but the admin client will not have it in the list of

servers● The admin client will not pick up the new server automatically● Although you can always do File - Open Server and open the server itself, we

don’t want to do that─ You haven’t switched to that server fully in the admin client ─ If you run multiple domains you haven’t switched to the domain fully

81


Ensuring Your Admin Client Can See The Server● Select Administration/Refresh Server List / Current Domain (or All Domains if

you run multiple Domains)

● The bookmarks should update and the new servers will now be visible

82


Confirm Connectivity From The Client● From your admin client

─ Select File / Preferences / User Preferences

● Select the Ports tab in Preferences

83


Trace From The Client● Click on Trace

─ Enter Pacific/svr/demo in the server field● Click on the Trace button

● Trace runs successfully and reports the route take to connect to the server

─ make sure the servernameis resolvable and the connectionsisn’t via passthru

─ Passthru connections don’t workwith Full Access Administration which you’ll need later

84


Tricky Client Connection Documents● You can automatically create a connection document to a server by doing

File - Application - Open then typing the ip address of the server in the server address field ─ This is good!

85


Tricky Client Connection Documents● However the connection document that is created will have a priority set as

“low” so will be ignored ─ this is bad!

● You will need to modify the connection document created in your local names.nsf and modify the priority on the “Advanced” tab of the document to say “normal”

86


Server Trace - Can The Servers See Each Other● Select your Atlantic/Svr/Demo server

─ Choose the server from the bookmarks bar● Select the Server tab on the admin client● Select the Server Console option in the sub-menu● Choose the Live option

87


Server Trace - Can The Servers See Each Other● You now have a live console● In the console command bar, type in

─ ‘Trace pacific/svr/demo’● Click on SEND or Press Enter

88


Server Trace - Can The Servers See Each Other● Trace will run on server● You should see “connected to server”● Success!

89


Trace From Other Cluster Mate● Select your Pacific/Svr/Demo server

─ Choose the server from the bookmarks● Select the Server tab on the admin client● Select the Server Console option in the sub-menu● Choose the Live option

90


Trace From Other Cluster Mate● You now have a live console● In the console command bar, type in

─ ‘Trace atlantic/svr/demo’● Click on Send or Press Enter

91


Trace From Other Cluster Mate● Trace will run on server● You should see “connected to server”● Success!

92


Creating Your Backup Replication Document● Always always always

─ Have a connection document replicating between server cluster mates● Very good practice

─ Helps if cluster fails─ Replicates stubs better─ Cluster replication is in-memory only so if one server is down the replication will

never catch up● Helps in the setup

─ The creation of a cluster request has to replicate to the cluster servers● It.makes.sense.anyway

─ All servers in a domain should be replicating the system databases at the very least– unless a proven, documented, business reason says otherwise

● but then you don’t have a working cluster, you have something else

93


Create A Connection Document● Select the atlantic server in the admin client● Go to the Configuration tab● Under Server, select Connections● Connections may already exist from the setup

─ Delete them

94


Create A Connection Document● Click on ADD CONNECTION

95


Basics tab● Connection type - Local Area Network● Source server - leave as Atlantic/Svr/Demo● Click on the button beside Destination Server - Select Pacific/Svr/Demo● Click on OK● Make sure both the Source and Destination Domains are correct, in this case

“Demo”● Select the button

─ Select TCPIP (in most circumstances, more on ports later)─ Select OK

96


Replication/Routing tab● Mostly left as is● Choose the button beside Mail Routing and select -None-

─ Click on OK● You don’t want your cluster replication backup connection to route mail, it’s not

necessary● You may not want to replicate templates in which case at *.NTF in the Files To

Exclude field

97


Schedule tab● Leave Schedule as Enabled● Change Connect at Times to 08:00 - 07:55

─ You don’t want to replicate only during daylight hours─ This instructs replication to happen 24 hours a day (oh alright 23hrs 55mins a day)

● Change Repeat Interval Of to 60 minutes─ For this session we’re going to use 5 minutes but cluster replication is in memory, if

it’s working you shouldn’t need to replicate more than every 60 minutes and doing so more often can break your cluster replication tasks

● Leave Days of Week as Sun-Sat

98


Comments tab● Enter details on the connection

─ So you know why it is there─ What the purpose of it is

● Enter author details─ Who created it─ When

99


Checking Replication● Click on● Connection now visible in view

100


Checking Replication● Select your Atlantic server


101


Checking Replication● You now have a live console● In the console command bar, type in

─ ‘sh sch” or “sh sched” if you like a bit more typing● Click on Send or Press Enter

102


Checking Replication● You should get this output

● TIP - always good to wait and watch scheduled replication take place─ just to be sure─ Alternatively, check the replication log in log.nsf

103


Creating your first cluster● Now we are ready to cluster the two servers

─ And 103 slides already... we are doing well ;)● In the admin client, select the Atlantic/Svr/Demo server● Click on the Configuration tab● Select the All Server Documents view / menu option on the left● Click on the servers Atlantic/Svr/Demo and Pacific/Svr/Demo to select them

104


Now The Difficult Bit - Creating A Cluster● Click on “Add To Cluster” button

●

● but that’s not all.......

105


Now The Difficult Bit - Creating A Cluster

● Click YES on the verification dialog box to confirm that you did mean to click on the “Add To Cluster” button and that you do mean to apply this setting to every server you have selected─ Now would be a good time to look at your screen and triple check what servers you

have selected

106


Creating Your First Cluster● Select *Create New Cluster and click on OK

● Enter a name for your cluster─ the cluster name will be visible to administrators and in configuration documents but

not to clients─ choose a name that makes sense and you don’t mind keeping eg “MailCluster” ─ Stay away from special characters─ For this SNT we’re going to call our cluster “OCEANS”

● Click on OK

107


Using Adminp● You will now get a dialog asking if you want to create the cluster immediately

─ That’s fine if you have access from your client to both servers and the connectivity is good and you have no issues making server side changes whilst the server is live

● However, choosing to create the cluster via the Administration Process will mean that all tasks are created in admin4.nsf and we can track and review them from there

108


Using Adminp● For today’s demo we want to track activity so we’re choosing “No” and forcing

the server to create requests that Adminp will process

Click on OK to the next dialog which is confirming the requests have been generated

109


Using Adminp● The requests you just generated will be processed by the Administration server

for your domain─ That server is running the adminp task─ All your servers should be running adminp but if the administration server or any of

the servers you want to cluster aren’t, your cluster request won’t work● On your admin client select the Administration Server for your Domino

domain, in our case that’s Atlantic/Svr/DemoFiles─ The Administration server will be the first one listed in your domain bookmark

– With the key!

110


Using Adminp● In your admin client select the Files tab● Choose the admin4.nsf database (Administration Requests) which should

appear near the top of the files list● Double click on the database to open it

If you see this screen, press Esc or close the tab to close it

111


Using Adminp● Select the All Requests By Action view from the left hand menu

─ Or All Requests By Time Initiated● You will see your request “Add Server To Cluster” created as two entries, one

for each server

112


Using Adminp● Taking a look at one of the requests we can see that the

─ Server To Perform The Action is the Administration Server for the Domino Directory

– this may not be (and often isn’t) one of the servers you are clustering– the admin4.nsf database must replicate the request to that server before

anything will happen

113


Using Adminp─ Name to perform the action on is atlantic/svr/demo in this instance (there is also a

document for pacific/svr/demo)– This means the adminp task on your Administration server will update the

server documents for those 2 servers when it receives this request

─ If your new server documents aren’t replicated to the Administration server, nothing will happen

114


It’s All About The Replication● Admin4 HAS to be replicating between your servers to be clustered and the

Administration server for the domain or these requests you’re generating will never be processed─ Only the Administration Server for the domain will process the request to add a

server to the cluster

● Names.nsf HAS to be replicating between your servers to be clustered and the Administration server for the domain or these requests you’re generating will never be processed─ If the server documents we just created aren’t present in the directory the

Administration server is using, the “add to cluster” request will fail─ If the updated server documents which are now assigned to a cluster are not

replicated back to the original servers (atlantic/svr/demo and pacific/svr/demo) then those servers will never complete their cluster setup

115


It’s All About The Replication● If for some reason your request doesn’t appear to work, don’t just do it again,

review your replication between servers

● You should always have a connection document that is able to replicate from your administration server to every other server, even indirectly, names.nsf and admin4.nsf (and events4.nsf but that’s for another session)

116


Using Adminp● Once the adminp task has processed the request the server documents in the

directory on the Administration server will now have a cluster name

● Process is marked as completed● SUCCESS!

117


Using Adminp● Request in admin4.nsf is automatically marked “Complete” (with a tick)

● SUCCESS!

118


Using Adminp● Now we just have to wait for those updated server documents to arrive into the

directories on Atlantic/Svr/Demo and Pacific/Svr/Demo─ In this case there’s no wait time for Atlantic since that is the Administration Server

● The server document update will trigger the cluster manager task on atlantic/svr/demo and pacific/svr/demo─ Without you doing anything further, the servers will complete their own setup

119


On The Consoles● On Atlantic you will see this

● The Cluster Database Directory task starts● Cluster database directory created● Attempts to replicate cldbdir.nsf with Pacific/Svr/Demo

─ Fails─ The cluster request has not yet completed on Pacific/Svr/Demo yet so the cldbdir.nsf

database doesn’t yet exist● The Cluster Replicator task starts

120


On The Consoles● On Pacific you will see this

● Cluster task start● Now cldbdir.nsf replicates correctly!● You will see reference to busytime and clubusy - we will talk about calendaring

in a bit

121


Cluster Database Directory● Task cldbdir runs on any Domino server that is clustered● The cldbdir tasks is responsible for keeping the cldbdir.nsf database up to date

─ The cldbdir.nsf database contains the filename, replica id and status of every database and template on a server

─ The cldbdir.nsf replicates between all servers in a single cluster● When trying to open a database on a server, the cldbdir.nsf database can tell

the cluster task if another server has a replica of the requested database for the user to be diverted to─ A user will never to sent to a cluster mate unless a replica of their requested

database is available on the destination server

● Cldbdir.nsf is maintained via the cluster replicator task● If cldbdir.nsf is not common across all cluster mates, and up to date, your

cluster won’t work properly

122


Cluster Directory Database● Select the Files tab● The cluster directory database is now visible

● Open the database by double clicking on it

123


Cluster Directory Database● Select Databases By Server● Both servers should be visible

● Select Databases by File Name● Two documents per database, one representing each database replica on each

of the two servers─ Not all databases have to be on all cluster mates. More on this later

124


Cluster Directory Database● Open one of the documents - these aren’t editable and are maintained by

cldbdir tasks─ databases must share a replica id to cluster, but not a filename─ databases can be marked Out Of Service if you don’t want them to cluster replicate

125


Check Tasks On The Server● Select your Atlantic server


126


Check Tasks On The Server● In the console command bar, type in

─ ‘sh ta”● Click on Send or press Enter

127


Check The Tasks On The Server● You should see these tasks running

128


Ask The Cluster If It Is Running● On the console, enter “sh cl” or “show cluster”

129


Cluster Security And Standardisation● Consider what you want a cluster to do● Both servers could be working with the applications

─ Running agents─ Permitting user access─ Replicating

● All servers in a cluster should have identical security to─ Applications─ Code execution

● There are easy ways to do this

130


Server Groups● Create a cluster group for your servers● Navitage to the People and Groups tab on your admin client● Click on the Groups view● Click on Add Group

131


Server Groups● Group Name

─ Give it a meaningful name. We’re using CLSVROceans (cluster servers oceans)● Select the button beside Group Type and choose Server Group

─ You don’t want to use the server group you’re creating here for mailing● Select the button beside Category and choose Administration

─ Or another category if you prefer● Complete the Description field

132


Server Groups● Select the icon beside Members● Choose the servers you want from the list of all directory entries

─ In this case atlantic/svr/demo and pacific/svr/demo─ Click on Add

133


Server Groups● We’re going to explicitly add this group to ACLs as a “Server Group”● Don’t use wildcards in the member fields● Don’t be tempted to use a client id such as an administrator’s name in the

member field─ People or wildcards added into a group containing server names will invalidate the

group as a ‘server group’ so its ACL rights won’t apply

134


Adding The Group To All ACLs● This is one of the best things we’ll show you today● First turn on the magic “Full Access Administration”

─ Full Access Administration gives you “Manager” access to every database on the server but no roles

● In the Administrator client choose the “Administration” menu and the “Full Access Administration”

135


Adding The Group To All ACLs● If you don’t have rights to use Full Access Administration on that server you’ll

get a dialog box telling you● Rights to use Full Access Administration are controlled on a server by server

basis● Configuration Tab - Current Server Document - Security Page

136


Adding The Group To All ACLs● In the admin client, select the Files tab● Click on the top level directory● On the right “Tools” pane

─ Expand Database─ Select Manage ACL

137


Adding The Group To All ACLs● We are going to replace the existing server name in any database ACL with the

new group name we just createdserver with the Group name

● So we’re going to “Modify” ACL entries

138


Adding The Group To All ACLs● Modify Dialog - first we choose the name we want to search for

● Then we select from the address dialog (or type, if you’re sure you’ll get it right) the name of the group we just created

139


Adding The Group To All ACLs● After clicking OK you are brought back to the modify ACL dialog● Click on the drop down beside User and change the type to Server Group● Leave other settings as they are

─ Click on OK─ You don’t want access or roles to change

140


Adding The Group To All ACLs● Each database on the server will now be scanned to see if “Atlantic/Svr/Demo”

is in the ACL─ it isn’t case sensitive but it won’t match “Atlantic” only, the syntax must be correct

● When completed the following dialog or similar appears● Click on OK

141


Adding The Group To All ACLs● If your dialog shows x database processed / x errors the reasons could be

─ If you don’t have Full Access Administration turned on then you can only change databases that your ID already has Manager access to

– All other databases will throw an error and fail to update─ Any databases that don’t have “Atlantic/Svr/Demo” in them will fail to update

– This isn’t a problem. If the server name wasn’t there then you don’t need it replaced

– If you dialog shows some failures, click on the status bar at the bottom of your admin client to pull up a list of recent log messages

142


Standardising Your Server Configuration● For Clusters to work well

─ Settings on each server document should be identical where possible● Good for user experience

─ Both servers behave the same● Good for applications

─ Code executes correctly● Good for clustering

─ Servers can automagically create replicas on each other

● If a cluster is working properly and providing load balancing the user’s should be unaware of its existence. Two users accessing the same application on different servers should have an identical experience

143


Server Level Rights● For both server documents● In the admin client go to the Configuration tab● Go to the All Server Documents view or menu item

─ Open the server document that you want to update using the button

144


Server Level Rights● Go to the Security tab● Scroll down to Server Access section (on the bottom half of the page)● Select the field Create Databases and Templates● Enter the values “CLSVROceans” and “LocalDomainAdmins”

─ this grants both servers atlantic/svr/demo and pacific/svr/demo as well as any administrators, rights to create databases on both servers

145


Server Level Rights● Repeat previous step for

─ Create New Replicas, so each server can create replicas of their dbs on the other─ Create Master Templates─ Allowed to use Monitors

● For Trusted Servers─ Add the cluster group, so each server can run agents on the other

● You must grant “Create new replicas” rights as a minimum─ You don’t need to do all of the other fields

– Its just good practice● Click on SAVE AND CLOSE

146


Code Execution Rights● For both server documents● In the admin client go to the Configuration tab● Go to the All Server Documents view or menu item


147


Code Execution Rights● Go to the Security tab● Scroll down to Programability

Restrictions section (on the right side of the form)─ Ensure that these fields match on both

servers

● For this demo─ Enter LocalDomainAdmins in the following

fields– Sign or run unrestricted methods and

operations– Sign agents to run on behalf of

someone else– Sign agents or XPages to run on

behalf of the invoker– Sign or run restricted LotusScript/Java

agents─ Enter */demo

– in Run Simple and Formula agents● Click on SAVE AND CLOSE when complete

148


Putting Replicas Onto The Cluster Server● There are many ways to put replicas onto the other server● Manual Create Replica

─ Enter each database and create a replica– Slow (created from your client connection to the server, not server to server)

● Automated Create Replica─ Using Administration Process to create the replicas

– Faster (you can select lots of databases/directories in one go)● Accelerated Create Replica

─ Significantly faster version of Automated Create Replica, uses backup level APIs● Physical database copy

─ OS level copy– Still a replica– Replica ID– Will need to be consistency checked (which can take a lot of time)– Risk of corruption– Cluster Database Directory won’t know about its existence immediately

149


Replicating Multiple Database Over To New Cluster ● We want to use Accelerated Create Replica

─ This isn’t a specific option, if you send “New Replica” requests via your Admin client they will be added to admin4.nsf to be processed by adminp

─ The request added to admin4.nsf is actually for “Accelerated Create Replica” so the adminp task will try to do that first

– If Accelerated Create Replica fails, adminp will generate a new “Create Replica” request to happen instead

─ You don’t choose “Accelerated Create Replica” vs “Create Replica” - the server does that for you

150


Replicating Multiple Database Over To New Cluster ● In Domino Administrator go to Files tab● Select the “Mail” directory (or whichever directory you’d like to replicate)

─ Or manually select individual databases by holding down the CTRL key● Click on Create Replica under the Tools - Database menu

151


● On the “Create Replica” dialog check the box for show only cluster members for the cluster OCEANS─ You don’t have to do this but if doesn’t correctly show your cluster servers, then your

cluster is not yet set up correctly and you need to stop and review● All the requests being created are shown on the right hand side of the dialog

Replicating Multiple Database Over To New Cluster

152


Replicating Multiple Database Over To New Cluster ● Creating these replicas correctly and quickly is key to our successful build of a

cluster● Ignoring errors or alerts that appear during replica creation will come back to

haunt you later● There are some tricks to minimise errors during replica creation but even when

you think you have everything configured correctly● TEST. TEST.TEST

─ Select one single database and try a replica request against that─ Only when you get that working, try a few others─ Don’t try running against every database on the server straightaway

● This isn’t hard, but we’ve both built countless clusters and would still take the time to test using single databases─ 30 minutes spent on testing could save you countless hours in deployment

153


Accelerated Create Replica Failing● Accelerated Create Replica (ACR) requires a specific set of circumstances for it

to work ─ but it will save you at least 60% of initial replication time so ensuring your

configuration meets these criteria will pay off in hours of saved time

154


What’s Needed For ACR To Work ● Servers must be clustered

– Test this using the server command we discussed earlier– Enter “Sh Cl” or “Show Cluster” on the console of each of the 2 servers you

should see similar to the following● note: output shows 2 cluster members and their names

sh cl Cluster information: Cluster name: OCEANS, Server name: atlantic/svr/demo Server cluster probe timeout: 1 minute(s) Server cluster probe count: 17906 Server cluster default port: * Server cluster auxiliary ports: Server availability threshold: 0 Server availability index: 100 (state: AVAILABLE) Server availability default minimum transaction time: 3000 Cluster members (2): Server: atlantic/svr/demo, availability index: 100 Server: pacific/svr/demo, availability index: 100

155


What’s Needed For ACR To Work ● Servers must both be v6 or higher● ACR was introduced in v6 so it isn’t supported if either the source or destination

server is pre v6● If either server is pre v6, the admin process will use standard replica creation to

complete the task

156


What’s Needed For ACR To Work ● Destination server must be 6.03 or higher

─ A new database property was introduced in 6.0.3 entitled– “support reply/forward indicator”

─ Since that setting is a default for any databases created post 6.0.3, a source server that is 6.0.3 or newer will most likely be home to databases with that property set

─ A destination server that is earlier than 6.0.3 will not be able to handle ACR creation of databases with a property it doesn’t recognise

157


What’s Needed For ACR To Work ● Source and Destination databases must share Full Text Index settings● When creating replica requests you have a checkbox to create a matching Full

Text Index. ACR will only work if the request matches the status of the source database─ Databases that are FT Indexed, must have replicas created with FT Indexes─ Default is that the box is unchecked which means “do not create Full Text Index”

158


What’s Needed For ACR To Work ● DAOS must not be enabled for a source database

─ Standard replicas can be created from databases that have file attachments stored in DAOS NLO’s, but not Accelerated Create Replicas

● If you want to enable DAOS and you know you’re building a cluster, build the cluster first then enable DAOS on both servers─ This is purely to make the initial replication of data from source to destination server,

faster. With DAOS enabled the replication will still happen, just not using Accelerated Create Replica

159


What’s Needed For ACR To Work ● Some slightly more obscure database settings that would prevent Accelerated

Create Replica from working─ Source database must not use single copy object store (unlikely)─ Source database must not use a single copy template that doesn’t exist on destination

server

● So that’s your ACR checklist. Spend 30 minutes verifying this and save yourself many many hours

160


Tip! A Nice Notes.ini Setting● ADMINP_ACCELERATED_REPLICA_OVERRIDES

─ is a notes.ini setting put in place on the source server from which you want to generate replicas

● ADMINP_ACCELERATED_REPLICA_OVERRIDES=4 ─ Ignore checking that both servers are in a cluster. This very useful if you want to

replicate large numbers of databases between 2 servers that aren’t clustered● ADMINP_ACCELERATED_REPLICA_OVERRIDES=1

─ Ignore checking that reply/forward indicators are supported. Useful if your destination or source server is prior to 6.0.3.

– However we have seen the reply/forward error in later releases too so this is useful notes.ini setting to apply regardless

● ADMINP_ACCELERATED_REPLICA_OVERRIDES=5─ Ignore both checking that source and destination servers are clustered and checking

that reply/forward indicators are enabled

● All other ACR requirements are fixed and unavoidable

161


There’s Always A But.....● Accelerated Create Replica will create a full replica of a database even if it has

replication disabled● To you it will look like everything has replicated perfectly however, that database

will not accept any further updates, either from the cluster replicator task or from the scheduled replication

● Run the cluster analysis tool (more on this later on) to identify which databases on your source server have replication disabled─ Or ask the Domino Administrator team to add that as a column in the Files view,

we’ll be right behind you!– (and whilst you’re at it, can you ask them to add a column showing if there’s a

replication selection formula too - thanks)

162


Some Tips Prior To Creating Replicas● Make sure you have enabled unread marks to replicate for all databases at

least within the cluster, if not across all servers● Remove Full Text Indexes from databases before you start and re-create them

after replication is finished─ Rebuilding Full Text Indexes after a major upgrade is best practice anyway─ Create Full Text Indexes without indexing file attachments for better performance,

stability and less disk usage

163


Enabling / Disabling Unread Mark Cluster Replication● Unread mark architecture has changed a number of times in Domino

─ Different releases/different solutions● Standard fixed from 6.02/6.5.3 timeframe● Unread marks are stored in a table

─ When you read a document, it is “written” to the table─ It’s a database overhead

● Unread marks can therefore replicate ─ Consider the extra load on the cluster─ Do you need unread marks on certain databases?

164


Enabling / disabling unread mark cluster replication● Navigate to the Files tab in the Administrator client● Go to the Mail directory● Right click on a mail file● Select Properties

165


Enabling / Disabling Unread Mark Cluster Replication● Select the Advanced tab● Under the item Replicate Unread Marks

─ Select Clustered servers only● Click on the X to close the properties dialog box

166


Setting Unread Marks For Many Databases● In the Administrator client choose the Administration menu and then Full

Access Administration─ To ensure you have access to all the databases you are about to change

● Navigate to the Files tab in the Administrator client● Go to the Mail directory● Right Mouse Click - choose Advanced Properties● Now select Replicate Unread Marks and check Clustered Servers Only

167


Removing Or Creating Full Text Indexes For Multiple Databases● In the Administrator client choose the Administration menu and then Full

Access Administration─ To ensure you have access to all the databases you are about to change

● Navigate to the Files tab in the Administrator client● Go to the Mail directory● Right Mouse Click - choose Full Text Index● Choose to Create or Delete the Full Text Indexes

168


How Clustering Behaves For Clients

169


Clients And Clustering● The Notes client is “Cluster Aware”● When it communicates with a Domino server that is in a cluster, that server

updates a file on the client with its cluster name and its cluster mates─ this makes the client aware that the server it wants is in a cluster and means even if

the server is down, the client has built in information about where to go next─ if you add a domino server to a cluster and rapidly shut down the original server, the

notes clients will never get a chance to “learn” about the cluster● If a primary server fails, the Notes client will search the other cluster mates it

knows about for a replica of the database it wants─ Searches cluster mates alphabetically not by availability

– this is very important in your planning─ Opens database on the first available cluster mate it finds─ Of course the Notes client has to be able to find these cluster mates

– So you need a connection document from the client to every cluster mate that could be used

– Or have the sever names registered in DNS

170


Cluster.NCF● The cluster.ncf is a file in the data directory on the client● It contains details of all clustered servers you have connected to from that

client, along with their cluster mates─ there can be several clusters in your cluster.ncf

● If you delete the cluster.ncf the Notes client will forget all about any clusters it was aware of, but will relearn the first time you reconnect to a clustered server

● Time=21/12/2010 00:57:53 (802577A4:0083A4BA)● Oceans● CN=Atlantic/OU=Svr/O=Demo● CN=Pacific/OU=Svr/O=Demo

171


Notes 8.5.2 - Improved Client Clustering/Repilcas● Client Transparency● In the 8.5.2 client, the ability to failover to a cluster mate without the user being

notified or stopped in their work has been improved● During the following activities, if the primary server goes down, the user will be

invisibly transferred to one of the cluster mates─ Creating a new entry for PIM views (Mail message, Calendar entry, or To Do)─ Opening a document in an application─ Opening a different folder or view─ Replying or forwarding with attachment─ Creating a new mail message while multiple documents are selected─ Deleting a single or multiple document─ Sending a message─ Switching back to Primary server when it becomes available

172


Notes 8.5.2 - Managed Replicas● Managed replicas are local replicas of mail files that behave as though they are

server replicas● Created by using a Mail Policy, a user assigned a managed replica will always

be directed to that local replica even if they are on the network and the server is available

● If the managed replica can be smaller in size than the server replica, restricted by age of documents or simply not contain attachments─ attempts by the client to access older documents or retrieve attachments result in

that information being transparently pulled from a server replica● If a local managed replica becomes corrupt in some way, the client is

transparently failed over to a server replica─ In this way Managed Replicas force the client to behave like a cluster mate for mail

● Managed replicas are intended for server based clients who have poor network connections to their home server or to reduce the number of concurrent clients connected to a home server.

173


Client Behaviour● When clicking on a bookmark for mail the users will be sent to their home

server─ if the home server is down or at its threshold level and there is another cluster mate

available the client will be redirected there─ If the user explicitly clicks on a workspace icon pointing to a server, the database will

open on that server unless the server is down completely ● No point building a cluster and leaving all your users with the same home

server─ All users will be forced to connect to that server initially until it’s so busy it has to

redirect requests─ You are not getting good value for your hardware investment─ Not effective load balancing

– Move some users so their home server is a different cluster mate

174


Move Many Users At Once To New Server● From the Administration server● Navigate to the People and Groups tab on your admin client● Select the People you want to move

● Choose Move To Another Server from the Tools menu on the right─ Move user ensures a mail file replica is on the new home server─ Updates the person document with the new home server─ Updates the client location settings when the user next logs in

175


Servers And Clusters● Servers operate slightly differently to clients when deciding where to failover● Domino server C will be aware that Server A and B are cluster mates

─ As with the clients, the server has a cluster.ncf in its data directory with a maintained list of servers and clusters

● If Server A goes down.. Domino Server C will try to find a cluster mate to work with ─ Alphabetically─ By availability (Server Availability Index - more on this later)─ Mail routing

176


Servers And Clusters● When replicating/routing to a server in a cluster

─ Create the connection TO the cluster name● Suppose you want to replicate from ARTIC server to the cluster● Let’s create a connection to the cluster from ARTIC

─ We create it on the administration server─ NOT on ARTIC

● Why?─ Centralised management of the directory is always a good thing

177


Servers And Clusters● You can and should also utilise cluster awareness for replication

─ Create connection documents to the cluster name, not server names─ By doing this you are telling atlantic/svr/demo in the connection below, to replicate

with any server in the OCEANS cluster– Since cluster replication should route all changes immediately, you should only

need to replicate with one server in the cluster– Using the cluster name as a destination means any not all!

178


Create A Connection Document● Select the Administration server in the admin client, in this case that is Arctic/

Svr/Demo● Go to the Configuration tab● Under Server, select Connections● Click on

179


Basics Tab● Connection type - Local Area Network● Source server - Change to Arctic/Svr/Demo● Click in the field Destination Server - type in the Cluster name OCEANS● Click on OK● Select the Choose Ports button - Select TCPIP● Confirm that both source and destination domain fields are set to “demo”● Do not set a destination address

180


Replication/Routing Tab● Mostly left as is● Set Routing Task to be -None-

181


Schedule Tab● Leave Schedule as Enabled● Change Connect at Times to 08:00 - 07:55

─ You don’t want to replicate only during daylight hours─ This instructs replication to happen 24 hours a day (oh alright 23hrs 55mins a day)

● Change Repeat Interval Of to 60 minutes─ For this session we’re going to use 5 minutes but cluster replication is in memory, if

it’s working you shouldn’t need to replicate more than every 60 minutes and doing so more often can break your cluster replication tasks

● Leave Days of Week as Sun-Sat

182


Comments Tab● Enter details on the connection

─ So you know why it is there─ What the purpose of it is

● Enter author details─ Who created it─ When

183


Checking Replication● Click on SAVE AND CLOSE● Connection now visible in view

184


Domino Mail Failover In Routing● The Router tasks in Domino is cluster aware● You can configure your servers to route mail to a cluster mate if the destination

server isn’t available● This is done in the server configuration document● Select the Administration server in the admin client, in this case that is Arctic/

Svr/Demo● Go to the Configuration tab● Under Server, select Configurations

185


Domino Mail Failover In Routing● If there is a server configuration already for your server (in this case atlantic/svr/

demo) then we will edit it● Alternatively click to add a new configuration document for

your server● We have an existing configuration for * which means [All Servers] so we’re

going to edit that

186


Domino Mail Failover In Routing● Click on the Router/SMTP tab● Click on the Advanced tab● Click on the Controls tab

187


Domino Mail Failover In Routing● Here we have a field called Cluster Failover● Choices are:

─ Enabled for last hop only─ Enabled for all transfers in domain─ Disabled

188


Domino mail failover in routing● Enabled for last hop only

─ If mail is sent from Arctic/Svr/Demo to Paul Mooney whose mail server is Atlantic/Svr/Demo and Atlantic/Svr/Demo is not available, the router task will send the mail to Pacific/Svr/Demo instead

─ However mail that would usually route through Atlantic/Svr/Demo for a user on a remote server, will not route through Pacific/Svr/Demo instead

189


Domino Mail Failover In Routing● Enabled for all transfers in domain

─ If mail is sent from Arctic/Svr/Demo to Paul Mooney whose mail server is Atlantic/Svr/Demo and Atlantic/Svr/Demo is not available, the router task will send the mail to Pacific/Svr/Demo instead

─ Mail that would usually route through Atlantic/Svr/Demo for a user on a remote server, will route through Pacific/Svr/Demo instead

190


Domino Mail Failover In Routing● Enabled for all transfers in domain

─ This setting makes your mail routing topology less controlled / structured because the router task will create its own “best fit” route from Server A to Server G if some of the intermediate servers are down.

– unexpected routing results– difficult to troubleshoot

191


Domino Mail Failover In Routing● Disabled

─ If a server is down, even if it’s in a cluster, mail for that server will queue up waiting on the sending server

192


So Let’s Test It !● Fail server● Test failover - try and open a database on the server that is now down● Test home mail - click on mail bookmark to open mail● Bring server back up ● Test - - click on mail bookmark to open mail

193


Drop The Atlantic Server● RDP/VNC to the ATLANTIC server● Type in Q on the console● Server shuts down

194


Launch Your Mail File● Launch the Lotus Notes client

─ If you don’t know how to do that, you are in trouble!● Click on the Mail bookmark● The server will try to connect

─ Fails● Then will open replica on cluster mate● The Status Bar on the client indicates the steps

● Send yourself an eMail for testing

195


On The Server Console● On pacific server, you will see

● Record of failover from a notes client

● When you send the email─ Server records failure but delivers to replica

196


Check the pacific server● On the console of the PACIFIC server● Type in SH CL

─ Note pacific knows that Atlantic isunavailable

● Check the cluster queue─ Type in Sh stat replica.cluster*

● List of databases waiting to cluster replicate

197


Final Checklist - And One You Should Repeat Regularly

198

© 2011 IBM Corporation 199

Using the Cluster Analysis Tool● What is it for?

─ Cluster analysis will assist in finding issues with your cluster environment– mismatched ACLS– missing databases– mismatched server settings

● Have Full Access Administration enabled● In the admin client go to the Server tab then the Analysis tab● Go to the Cluster menu option on the right


Using the Cluster Analysis Tool● Check the items to scan for

─ Server details─ ACLs not matching in server replicas─ Databases with replication disabled. ─ Replication selection formulas not matching─ Databases that don't’ exist on all servers in a cluster


Choose typical settings● Select everything

─ Except Web Navigator– Not really used anymore

● Note results database─ This will create a cluster analysis database locally─ Based off cluster analysis template

– Which is local to admin client● Choose Overwrite this database● Click on OK● You will see status bar animation

─ NOTE - on large sites, this can take hours


Cluster analysis database opens● Select the By Test view● Collapse the view by clicking on

or hold Shift key and press “-”● Review the results

─ Many are irrelevant─ Some are vital

– e.g. Replicas exist within Cluster– Are any applications/mail files failing?

● Resolve the issues and re-run the test● Make the cluster analysis part of your

scheduled maintenance routine● Close the database when finished


How To Monitor And Tune Your New Cluster Environment

203


Work Queues And Clustering● Cluster tasks can get very busy

─ Heavily used mail servers─ Heavily used application servers

● On extremely busy servers Domino Clustering can lag behind● Cluster replication occurs one database at a time per cluster replicator task

─ all changes to that database are accrued and applied in a batch● Increasing number of clrepl tasks will reduce queues

─ at the expense of memory and disk writes─ it’s an option but may not be necessary

204


Work Queues And Clustering● Work queue depth

– how many databases are queuing to be replicated by the cluster task

sh st replica.cluster.workqueuedepth* - Server Console Command

Replica.Cluster.WorkQueueDepth = 0 - how many databases are queuing now Replica.Cluster.WorkQueueDepth.Avg = 2 - how many queue in average Replica.Cluster.WorkQueueDepth.Max = 7 - how many have been in the queue at max 3 statistics found

Your work queue depth should be stable, not Increasing

205


Work Queues And Clustering● Seconds On Queue

– how long a database sits on the cluster replicator queue before its data is transmitted off and out of memory

sh st replica.cluster.secondsonque* - Server Console Command

Replica.Cluster.SecondsOnQueue = 6 - Current wait time for data to be replicated Replica.Cluster.SecondsOnQueue.Avg = 7 - average wait time for data to be replicated Replica.Cluster.SecondsOnQueue.Max = 15 - maximum wait time for data to be replicated 3 statistics found

Your seconds on queue should remain in single digits if you want your cluster to be transparent to the users. This means that additions / deletions / updates would take only a few seconds to appear on cluster mates.

Seconds on queue will occasionally peak, this isn’t a big concern but the current wait time and average wait time should be fairly stable

206


Work Queues And Clustering● If your cluster statistics are showing poor performance or cluster delays then

you can consider increasing the number of cluster replicators (CLRepl) to manage the load

● To add multiple cluster replicators you need to modify the notes.ini file● From the server console type the command

– Set Config Cluster_Replicators=X where X represents the number of replicators you need

● Each additional cluster replicator is an additional Domino task that consumes processor and memory─ Too many tasks can also cause conflicts in the queue

● Adding additional cluster replicators should be carefully considered and applied in stages

207


Displaying cluster work queues● Select your Atlantic/Svr/Demo server


208


Getting Work Queue Depth Console Stats● Enter sh st replica.cluster.workqueuedepth* in console box● Click on send

● Output

209


Getting Seconds On Queue Console Stats● Enter sh st replica.cluster.secondsonqueue* in console box● Click on send

● Output

210


Adding Cluster Stats To The Monitor● In the Administration Client

─ Select the Administrative server─ Click on the Server Tab─ Click on Monitoring─ Click on the Start button

─ Monitoring connections are from your client, make sure you have a connection or route to all servers you want to monitor or the server won’t report

211


Adding Cluster Stats To The Monitor● Right click on the Statistics

─ Select Monitor New Statistic● The following dialog box appears

● Scroll down to Replica, then expand Cluster─ Expand WorkQueueDepth

– Select Avg– Click on OK

212


Adding Cluster Stats To The Monitor● The statistic is now visible on Monitor

─ Repeat previous steps for every stat you want visible

213


Cluster Traffic● Cluster traffic is significant● By default, cluster traffic will use your public network connection.

─ This leads to replication data between servers fighting for bandwidth with client access activity

● One solution which separates cluster traffic from client traffic─ Have multiple network cards on cluster servers─ With different IP address on each network cards─ Create additional ports for domino servers on each server / cluster mate─ Map a port to a particular IP address

– Therefore you are mapping a port to a network card─ Force cluster replication over the new port/network card

– If the servers are co-located you could have a high speed fibre connection handling replication

214


Setting Up Private Traffic Cluster Replication● Edit the notes.ini file to add new values● We can do this from the server console● If we want to create a new port for handling cluster traffic called “ClusterTCPIP”● We can add the new port settings but they won’t take effect until we restart the

server

215


Setting Up Private Traffic Cluster Replication● Ensure both network cards are enabled are have ip addresses bound to them● Add entries into local hosts file on the server so that each ip address has its

own hostname● You should be able to ping both hostnames from the server you are working on● You should be able to ping both ip addresses from the cluster mate server

216


Setting Up Private Traffic Cluster Replication● We have 2 network cards on our server now

─ ip address: 192.168.100.1─ ip address: 10.0.0.10

● By default Domino will grab all ip addresses for port 1352 and will bind tasks such as HTTP to the first network card

● We can change that● From the server console type

─ Set Config TCPIP_TCPIPAddress=0,192.168.100.1:1352─ Set Config ClusterTCPIP_TCPIPAddress=0,10.0.0.10:1352

● Now we have told Domino that there are two network cards and that it should bind each to its own port “TCPIP” and “ClusterTCPIP”

217


Setting Up Private Traffic Cluster Replication● Now we have to tell Domino that both ports should be used● From the server console we update notes.ini

─ Set Config Ports=TCPIP,ClusterTCPIP

● We now have to tell Domino that the ClusterTCPIP port should be used for Cluster traffic

● From the server console we update notes.ini─ Set Config Server_Cluster_Default_Port=ClusterTCPIP

218


Setting Up Private Traffic Cluster Replication● Now we have to update the server document● In the admin client go to the Configuration tab● Go to the All Server Documents view or menu item


219


Setting Up Private Traffic Cluster Replication● Go to the Ports page● In Notes Network Ports we need to add the new port details● Port order is very important here because Domino tasks will always use the first

port listed by default● Port - ClusterTCPIP which is the name we just added into notes.ini

220


Setting Up Private Traffic Cluster Replication● Protocol is non editable and will be kept updated by the server● Notes Network - should be different for this port and shared across cluster

mates● Net Addresses - could also be the hostname that resolves to this 2nd ip ● Enabled - make sure you enable the port

221


Setting Up Private Traffic Cluster Replication● Now we just need to restart the server● From the server console type

─ Restart Server● Once the server comes back up we can confirm the port is active by issuing the

following server console command─ Sh Port ClusterTCPIP

● We can also confirm that it is being used by the cluster replicator by issuing the following server console command─ Sh Cl or Show Cluster

Cluster name: OCEANS, Server name: atlantic/svr/demoServer cluster probe timeout: 1 minute(s)Server cluster probe count: 18834Server cluster default port: ClusterTCPIP

222


Setting Up Private Traffic Cluster Replication● When we set a private network to route cluster replication traffic over we are

telling Domino not to send cluster traffic over any other port● If the cluster port fails on the server, the cluster replication activity will back up

as it can’t route through any other port● Often no-one will notice that the port has failed because no users are active on

it● Best Practice is to include an auxiliary port which tells Domino what to try if the

cluster port fails● From the Domino Server console issue the following command

─ Set Config Server_Cluster_Auxiliary_Ports=TCPIP─ this tells Domino to try the TCPIP port if the CLUSTERTCPIP port isn’t responding

– You can also use Server_Cluster_Auxiliary_Ports=* to mean “use any port”

223


Resilient Port Configuration● Single points of failure in the private port concept

─ the network cards─ the private network itself─ we can use auxiliary ports to work around this and it does give us the advantage of

separating the traffic and sending the cluster traffic over a faster network

● Alternatively for simpler configuration use teamed NICs on newer servers

● Ports aren’t everything● Use resilient switches● Use resilient networks

224


Fun With Server Clustering● You can force all users to fail to another cluster server

─ Useful for Disaster Recovery testing ─ Useful for server maintenance work─ Useful for migrations

● On the server console type the following command─ Set Config Server_Restricted=

– 1 Prevent any new sessions opened on this server until restart● Administrators bypass this restriction if using the Domino

Administrator client– 2 Prevent any new sessions opened on this server until setting changed

● Administrators bypass this restriction if using the Domino Administrator client

● Setting is persistent even through server reboots– 0 Feature disabled

225


Fun With Server Clustering● On the server console type

– Drop All– this disconnects all connected users, including you. – Don’t panic if you type the command and your Administrator client now says

“Server not responding”, that’s what should happen● Just connect to the server again

● Be aware that server_restricted applies not just to clients but also to other servers. Whilst a server is sat in restricted mode─ No mail will route to it─ Nothing will replicate with it─ No cluster traffic will be sent to it

● When coming out of server_restricted, you want to immediately force a full replication to catch up on missed data

226


Restricting A Domino Server● Let us test● Set Server_Restricted=1 on Atlantic/Svr/Demo (Paul’s home server)● Type Drop All on the server console● Paul will now be failed over to Pacific/Svr/Demo to open his mail

─ I will still be able to access and work on Atlantic via Domino Administrator● If Paul tries to open Atlantic/Svr/Demo directly, he is denied access

227


Preventing Domino Server Being Overloaded● Domino can fail new sessions based on the following”

─ Hitting maximum number of sessions specified─ Hitting low availability (the low setting is set by administrator)

● Maximum sessions─ Server_Max_users─ Server_Max_Sessions─ these are very “hammer to crack a nut” type settings. Do you rarely want to restrict

server access to specific exact numbers of users?– Let me tell you a sad story about clusters, an evil admin and max users gone

bad● Availability index

─ How much resource the server has available to it

228


Let’s Talk SAI ..● Server Availability Index (SAI) is core to Domino cluster logic. ● The server monitors itself, checking the response times of requests continually● As the response times start decreasing so does the SAI● SAI is meant to represent the % of server capacity that is available● SAI starts at 100 - meaning, “I’m fully available and ready to take on all comers”

─ SAI falls to 0 - meaning, “go away I can’t handle any more traffic”

● The SAI is relative to the actual capacity of the server – 100% SAI on one server won’t give the same response times as 100% SAI on another more powerful server

229


Preventing Domino Server Being Overloaded● You can set a notes.ini setting of Server_Availability_Threshold=X where X

represents the lowest acceptable value of availability you want the server to reach before it starts failing users over to a cluster mate

● But the thing about SAI calculation─ It kind of , sort of, works─ and sometimes doesn’t

● The biggest problem appears to be with powerful hardware where the SAI miscalculates its own availability showing itself as being on low availability when the server is barely utilised

● There are lots of reasons for this, mostly due to the server trying to record when it’s “busiest” as a tracking point and base everything off that

230


Preventing Domino Server Being Overloaded● What you need to know

─ It gets better with 8.0.2 and onwards. Earlier Domino version SAI calculations are less reliable

─ If you have brand shiny new hardware which Domino insists is maxed out - that may not be true

─ You can override Domino’s calculation of available resources by modifying a notes.ini setting called Server_Transinfo_Range

● HERE BE TIGERS!!!

● We will not be going into any more detail on changing Server_Transinfo_Range because we sincerely hope none of you will be doing it─ If you want to know more read the Admin help─ Then read it again─ Then make a coffee , sit down and ask yourself if you really want to do this

231


Disabling database cluster replication● You may not want to cluster replicate a specific database temporarily, even

though it is present on multiple servers─ By default Domino will cluster replicate, and make available for failover, any server

sharing a replica ID between source and destination● If it’s an application where evil developers have hardcoded server names and it

simply won’t work on another server, you don’t want it clustered because you don’t want users failing over and trying to use it─ However you may want a replica stored on a cluster mate and kept up to date as a

live back up● If a single database is taking up too much cluster replication activity disable it

for clustering─ Disabling / Enabling cluster replication doesn’t affect standard replication so if a

database doesn’t need cluster performance take it out of cluster replication

● You can disable cluster replication for a individual database or group of databases via the Domino Administrator client

232


Disabling database cluster replication● In the Domino Admin client select server Atlantic/Svr/Demo● Select the Files tab● Select the file you want to mark unavailable for clustering and choose cluster

from the right hand menu (or right click)

233


Disabling database cluster replication● You have five options on this dialog, all are radio buttons so only one can be

selected at a time● Out of Service - do not allow client access to this database or fail clients over

to it. Existing clients can remain in it until they finish their session but can’t reconnect

● In Service - allow clients back into this database or allow clients to fail over into it from another server

● Pending Delete - as Out of Service but Pending Delete cannot be marked “In Service” again

234


Disabling database cluster replication● You have five options on this dialog, all are radio buttons so only one can be

selected at a time● Enable Cluster Replication - allow the cluster replicator task to continue

sending changes to and from this database.● Disable Cluster Replication - halt the cluster replicator from sending or

receiving changes to this database. No changes will be held in memory for it. The scheduled replicator task ignores this setting

235


Troubleshooting Cluster Traffic● Always use “pause” on the cluster replicator if you want to troubleshoot rather

than stopping it completely● Server Commands

─ Tell CLRepl Pause ─ Tell CLRepl Resume

● A pause / resume enables the cluster replication tasks to continue to build queues of changes in memory

● A “Quit” on the CLRepl destroys any in memory tasks and no more are accumulated until CLRepl is restarted

● If you quit CLRepl there will be a gap in your data until the scheduled replication catches up or until you force a replication

● If you have mulitple cluster replicators running, a “Quit” or “Pause” command applies to them all

236


Examples Of Cluster Topologies

237


Clustering topologies● Many ways of implementing clustering● Active - Active

─ All servers with all data● Active - Active Data split

─ Three way cluster - all data exists over two of them● Active - passive

─ Two way cluster - all activity on customer facing one. ● Backup cluster

─ n-way cluster─ Data only exists on the production server and a single backup server contains

everything

238


Centralised Servers● Large hardware in a single central location● Users assigned to different servers so no one server services all users by default● All data is clustered across all servers so each provides service to any user

239


Disaster Recovery Solution● Small remote sites can cluster their data to large central hardware ● Multiple servers can cluster their data to a single remote DR server● The DR server can be forced to make itself unavailable to clients to no requests are

failed over to it unless all other servers are unavailable completely

240


Disabling server cluster replication● You may want cluster failover without the replication

─ low bandwidth─ often in DR situations where there is an offiste DR server

● Possible to implement in domino─ Consider the implications

– not true clustering as data isn’t replicated in realtime and is reliant on scheduled replication

– Clients still recognise the server as being in the cluster so mail be failed over to it

● Scenario─ Site replicates with DR server nightly─ Cluster enabled─ Data is up to date of last replication

241


Things To Watch Out For

242


Quotas● Quotas are server and database specific● You have to set a Quotas on every server if that’s what you want● The cluster replicator and standard replicator will ignore the file size Quota by

default

● How Quotas work● Quotas on a mail file are usually set to allow mail to continue to be delivered

and to refuse to send or save mail unti the filesize is reduced● File sizes will vary from server to server within a cluster for the same database

so make sure you schedule a nightly compact on each server

243


Calendaring and Scheduling● A Domino server tracks the free / busytime for each of their users (those that

have that server set as their home server) in a database called busytime.nsf● In a cluster, the user’s calendar activity doesn’t necessarily happen on a single

server, it could happen on any server in the cluster● To handle this, Domino creates a free/busytime database that is replicated

throughout the cluster. This is called clubusy.nsf● When a server is first added to a cluster you will see the console delete the

existing busytime.nsf and create a replica of clubusy.nsf instead● Clubusy.nsf will be replicated by the cluster replicator task but should also

always be included in your scheduled replication connection document

244


Out Of Office● In v8 of Domino the Out of Office agent was replaced by the OOO router

service● Whereas the agent settings by default run on “Any Server”, the OOO service is

triggered by whichever router task delivers the message to the user’s mail file● You configure in your server configuration document, whether OOO shoud use

the agent or the service● All servers in a cluster should be configured the same to prevent mulitple OOO

messages being sent out● If you configure OOO as a service, and the users are on pre v8 mail templates,

they will still run the agent─ In which case you want to ensure the agent is set to run on one server only

245


Clustering and Browsers● Clustering is about Notes traffic only and isn’t for browser traffic● Other tools that use Notes traffic such as Blackberry servers or Traveler, will

recognise your cluster and failover appropriately● Browsers themselves are not “cluster aware”, they ask for specific hostnames● If you want to cluster browser traffic (or POP or IMAP) you have to manage this

at a level above Domino─ DNS ─ IP Load Balancers─ Websphere Edge Server─ lots more...

─ There’s also ICM for Domino which handles load balancing via Domino for browser traffic, but there are better network level solutions now

– It does work– We would deploy it for an enterprise requiring high availability– It’s very clever but you still have a single point of failure - your ICM server

246


Many things to consider before implementing● Server Names

─ So much in clustering logic is based on finding the first server alphabetically● Security

─ in the clustering world, all servers are equal and this means roles too● Network Connectivity and WAN Hops

─ Domino doesn’t take account of bandwidth when deciding which server you should use, just server load and the name of the server (ridiculous as that 2nd one sounds)

● Failover vs Load Balancing─ Domino clustering is especially clever as it can load balance users dynamically but

is that really what you want● Agents - and terrible, terrible developer code in general

─ Only run on server ─ Hardcoded server names

● Server versions and templates─ Run design refresh from one server only─ Disable the design task on other servers

247


Many things to check periodically ● Replicas

─ If they aren’t there they won’t cluster● Security

─ Not too much, not too little (Editor vs Manager) but always the same for every server● Connectivity

─ How much bandwidth do you need─ Domino can’t help with client / server network latency issues

● Server Load─ At what point short of crashing does the server think it’s busy

● Failover─ Avoiding single points of failure (I’m looking at YOU SAN admins!)

● Agents─ Lazy developers are a cluster servers worst enemy

248


Where Things Can Go Wrong● Know your alphabet● Unread marks● ACLs - especially roles in applications● Design updates on different server versions● Not standardising on OOO service or agent● Not setting up connections from clients to all other cluster mates or having

server names in DNS (internal and external)● Having a cluster mate at a DR site on a lousy connection but beginning with the

letter A● Inbound mail routing● Transfer through any hops

─ Strange things can happen● MAX_USERS - eek!● SAI and powerful servers

249


Wrap Up

250


Questions

251

● Paul Mooney● [email protected]

● Gabriella Davis● [email protected]


Legal Disclaimer● © IBM Corporation 2009. All Rights Reserved● The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information

contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

● References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

● IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

● Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

● Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

252