Rhel cluster basics 1

Cluster Basics

A cluster is two or more computers (called nodes or members) that

work together to perform a task.

Types of clusters

• Storage

• High availability

• Load balancing

• High performance

StorageStorage clusters provide a consistent file systemimage across servers in a cluster, allowing theservers to simultaneously read and write to asingle shared file system. A storage clustersimplifies storage administration by limiting theinstallation and patching of applications to one filesystem. Also, with a cluster-wide file system, astorage cluster eliminates the need for redundantcopies of application data and simplifies backupand disaster recovery. Red Hat Cluster Suiteprovides storage clustering through Red Hat GFS.

High availabilityHigh-availability clusters provide continuousavailability of services by eliminating single pointsof failure and by failing over services from onecluster node to another in case a node becomesinoperative. Typically, services in a high-availabilitycluster read and write data (via read-writemounted file systems). Therefore, a high-availability cluster must maintain data integrity asone cluster node takes over control of a servicefrom another cluster node. Node failures in ahigh-availability cluster are not visible from clientsoutside the cluster.

Load balancingLoad-balancing clusters dispatch network servicerequests to multiple cluster nodes to balance therequest load among the cluster nodes. Loadbalancing provides cost-effective scalabilitybecause you can match the number of nodesaccording to load requirements. If a node in aload-balancing cluster becomes inoperative, theload-balancing software detects the failure andredirects requests to other cluster nodes. Red HatCluster Suite provides load-balancing through LVS(Linux Virtual Server).

High performanceHigh-performance clusters use cluster nodes toperform concurrent calculations.

A high-performance cluster allows applications towork in parallel, therefore enhancing theperformance of the applications.

High performance clusters are also referred to ascomputational clusters or grid computing.

Red Hat Cluster Suite

Red Hat Cluster Suite (RHCS) is an integratedset of software components that can bedeployed in a variety of configurations tosuit your needs for performance, high-availability, load balancing, scalability, filesharing, and economy.

RHCS major components

• Cluster infrastructure — Provides fundamentalfunctions for nodes to work together as acluster: configuration-file management,membership management, lock management,and fencing.

• High-availability Service Management —Provides failover of services from one clusternode to another in case a node becomesinoperative.


• Red Hat GFS (Global File System) — Provides acluster file system for use with Red Hat ClusterSuite. GFS allows multiple nodes to sharestorage at a block level as if the storage wereconnected locally to each cluster node.

• Cluster Logical Volume Manager (CLVM) —Provides volume management of clusterstorage.


• Global Network Block Device (GNBD) — Anancillary component of GFS that exports block-level storage to Ethernet. This is an economicalway to make block-level storage available to RedHat GFS.

• Linux Virtual Server (LVS) — Routing softwarethat provides IP-Load-balancing. LVS runs in apair of redundant servers that distributes clientrequests evenly to real servers that are behindthe LVS servers.


• Cluster administration tools — Configurationand management tools for setting up,configuring, and managing a Red Hat cluster.The tools are for use with the ClusterInfrastructure components, the High-availabilityand Service Management components, andstorage.

• You can configure and manage other Red HatCluster Suite components through tools forthose components.


Cluster Infrastructure

The Red Hat Cluster Suite cluster infrastructure providesthe basic functions for a group of computers (called nodesor members) to work together as a cluster. Once a clusteris formed using the cluster infrastructure, you can useother Red Hat Cluster Suite components to suit yourclustering needs.

The cluster infrastructure performs the followingfunctions:• Cluster management• Lock management• Fencing• Cluster configuration management

Cluster Management

• Cluster management manages cluster quorumand cluster membership. CMAN (anabbreviation for cluster manager) performscluster management in Red Hat Cluster Suite forRed Hat Enterprise Linux.

• CMAN is a distributed cluster manager and runsin each cluster node; cluster management isdistributed across all nodes in the cluster

Cluster Management

CMAN keeps track of cluster quorum bymonitoring the count of cluster nodes. If morethan half the nodes are active, the cluster hasquorum. If half the nodes (or fewer) are active,the cluster does not have quorum, and all clusteractivity is stopped. Cluster quorum prevents theoccurrence of a "split-brain" condition — acondition where two instances of the same clusterare running. A split-brain condition would alloweach cluster instance to access cluster resourceswithout knowledge of the other cluster instance,resulting in corrupted cluster integrity.

Cluster Management

Quorum is determined by communication ofmessages among cluster nodes via Ethernet.

Optionally, quorum can be determined by acombination of communicating messages viaEthernet and through a quorum disk.

For quorum via Ethernet, quorum consists of 50percent of the node votes plus 1. For quorum viaquorum disk, quorum consists of user-specifiedconditions.

Cluster Management -Quorum

Lock management is a common cluster-infrastructure service that provides a mechanismfor other cluster infrastructure components tosynchronize their access to shared resources.

In a Red Hat cluster, DLM (Distributed LockManager) is the lock manager.

DLM is a distributed lock manager and runs ineach cluster node; lock management is distributedacross all nodes in the cluster.

Lock Management

GFS and CLVM use locks from the lock manager.GFS uses locks from the lock manager tosynchronize access to file system metadata (onshared storage).

CLVM uses locks from the lock manager tosynchronize updates to LVM volumes and volumegroups (also on shared storage).

Lock Management

Fencing

Fencing is the disconnection of a node from thecluster's shared storage.

Fencing cuts off I/O from shared storage, thusensuring data integrity.

The cluster infrastructure performs fencingthrough the fence daemon, fenced.

Fencing

When CMAN determines that a node has failed, itcommunicates to other cluster-infrastructurecomponents that the node has failed.

Fenced, when notified of the failure, fences thefailed node.

Other cluster-infrastructure componentsdetermine what actions to take — that is, theyperform any recovery that needs to done.

Fencing

For example, DLM and GFS, when notified of anode failure, suspend activity until they detectthat fenced has completed fencing the failednode.

Upon confirmation that the failed node is fenced,DLM and GFS perform recovery.

DLM releases locks of the failed node; GFSrecovers the journal of the failed node.

FencingThe fencing program determines from the clusterconfiguration file which fencing method to use.

Two key elements in the cluster configuration filedefine a fencing method: fencing agent and fencingdevice.

The fencing program makes a call to a fencing agentspecified in the cluster configuration file.

The fencing agent, in turn, fences the node via afencing device. When fencing is complete, thefencing program notifies the cluster manager.

Fencing Methods• Power fencing — A fencing method that uses a

power controller to power off an inoperable node.

• Fibre Channel switch fencing — A fencing methodthat disables the Fibre Channel port that connectsstorage to an inoperable node.

• GNBD fencing — A fencing method that disables aninoperable node's access to a GNBD server.

• Other fencing — Several other fencing methodsthat disable I/O or power of an inoperable node,including IBM Bladecenters, PAP, DRAC/MC, HP ILO,IPMI, IBM RSA II, and others.

Power Fencing

Fibre Channel Switch Fencing

Fencing a Node with Dual Power Supplies

Fencing a Node with Dual Fibre Channel Connections

Cluster Configuration System

The Cluster Configuration System (CCS) managesthe cluster configuration and providesconfiguration information to other clustercomponents in a Red Hat cluster. CCS runs in eachcluster node and makes sure that the clusterconfiguration file in each cluster node is up todate. For example, if a cluster systemadministrator updates the configuration file inNode A, CCS propagates the update from Node Ato the other nodes in the cluster.

Cluster Configuration System

Cluster Configuration SystemOther cluster components (for example, CMAN) access configuration information from the configuration file through CCS.

Cluster Configuration FileThe cluster configuration file (/etc/cluster/cluster.conf) isan XML file that describes the following clustercharacteristics:• Cluster name — Displays the cluster name, cluster

configuration file revision level, and basic fence timingproperties used when a node joins a cluster or is fencedfrom the cluster.

• Cluster — Displays each node of the cluster, specifyingnode name, node ID, number of quorum votes, andfencing method for that node.

• Fence Device — Displays fence devices in the cluster.Parameters vary according to the type of fence device.

• Managed Resources — Displays resources required tocreate cluster services. Managed resources includes thedefinition of failover domains, resources (for example anIP address), and services.

High-availability Service Management

• High-availability service management providesthe ability to create and manage high-availabilitycluster services in a Red Hat cluster.

• The key component for high-availability servicemanagement in a Red Hat cluster, rgmanager,implements cold failover for off-the-shelfapplications.

• A high-availability cluster service can fail overfrom one cluster node to another with noapparent interruption to cluster clients.

Failover Domains

• A failover domain is a subset of cluster nodesthat are eligible to run a particular clusterservice.

• Cluster-service failover can occur if a clusternode fails or if a cluster system administratormoves the service from one cluster node toanother.

Failover Priority• A cluster service can run on only one cluster

node at a time to maintain data integrity.

• Specifying failover priority consists of assigninga priority level to each node in a failoverdomain. The priority level determines thefailover order.

• If you do not specify failover priority, a clusterservice can fail over to any node in its failoverdomain.

Failover Domains Example

Failover Domains ExampleFailover Domain 1 is configured to restrict failover within thatdomain; therefore, Cluster Service X can only fail over betweenNode A and Node B.

Failover Domain 2 is also configured to restrict failover with itsdomain; additionally, it is configured for failover priority. FailoverDomain 2 priority is configured with Node C as priority 1, Node B aspriority 2, and Node D as priority 3. If Node C fails, Cluster Service Yfails over to Node B next. If it cannot fail over to Node B, it triesfailing over to Node D.

Failover Domain 3 is configured with no priority and no restrictions.If the node that Cluster Service Z is running on fails, Cluster Service Ztries failing over to one of the nodes in Failover Domain 3. However,if none of those nodes is available, Cluster Service Z can fail over toany node in the cluster.

Web Server Cluster Service Example


• In the example, a high-availability cluster servicethat is a web server named "content-webserver".

• It is running in cluster node B and is in a failoverdomain that consists of nodes A, B, and D.

• In addition, the failover domain is configured witha failover priority to fail over to node D beforenode A and to restrict failover to nodes only in thatfailover domain.


• Clients access the cluster service through the IPaddress 10.10.10.201, enabling interaction with theweb server application, httpd-content.

• The httpd-content application uses the gfs-content-webserver file system.

• If node B were to fail, the content-webserver clusterservice would fail over to node D. If node D were notavailable or also failed, the service would fail over tonode A.

• Failover would occur with no apparent interruption tothe cluster clients.

Technology

Rhel cluster basics 1