17285826 Clustering and Storage With Windows Server 2003

8/6/2019 17285826 Clustering and Storage With Windows Server 2003

1/20

Clustering and Storage withWindows Server 2003

an Storage eBook


2/20

1

contents

This content was adapted from Internet.com'sServerWatch Web site and was writtenby Marcin Policht.

Clustering and Storage with Windows Server 2003, an Internet.com Storage eBook.

2008, Jupitermedia Corp.

2 Server Clustering

10 Disk Storage (SCSI)

13 SAN-Based Storage

15 iSCSI Storage

19 Conclusion

10

2

13

15 19

Clustering and Storage with Windows Server 2003

[ ]


3/20

I

n this eBook we're going to take a look at clusteringand storage with Windows Server 2003. You can usethe information we present here as a foundation for

creating high-availability solutions.

Although some of the technologies we will be describ-ing have been available in earlier version of Windows(as inherent components, add-on programs, or third-partyofferings that made their wayinto Microsoft's portfoliothrough acquisitions) their latestincarnations are superior fromfunctionality, stability, and man-ageability perspectives.

Server ClusteringTwo basic approaches to reach-ing high availability have beenbuilt into the Windows Server2003 operating system. Thefirst, known as Server Clustering, requires WindowsServer 2003 Enterprise and Datacenter Editions. Thesecond one, known as Network Load Balancing (NLB),was incorporated into all Windows Server 2003 ver-sions (including Standard and Web).

Each represents a unique approach to eliminating "a

single point of failure" in computer system design. Theyalso share one important underlying feature that servesas the basis for their redundancy: Both increase avail-

ability by relying on multiple physical servers, hostingidentically configured instances of a particular resource(such as a service or application). The main differencelies in the way these instances are defined and imple-

mented.

In case of NLB, each instance ispermanently tied to the host-ing of its physical server, and itremains active as long as thisserver is functional. In otherwords, all of them operate

simultaneously during clusteruptime. With Server Clustering,on the other hand, there is onlya single active instance foreach highly available resource,regardless of the total numberof servers that are members of

the entire cluster. The server that currently hosts thisresource becomes its owner and is responsible for pro-cessing all requests for its services.

These underlying architectural principles introduce anumber of challenges. Since the NLB cluster consists ofup to 32 instances running in parallel servers, there is a

2 Clustering and Storage with Windows Server 2003, an Internet.com Storage eBook. 2008, Jupitermedia Corp.

Clustering and Storage with Windows Server 2003[ ]

Clustering and Storage withWindows Server 2003

By Marcin Policht

With Server Clustering, there is only a single active instance for each highly availableresource, regardless of the total number of servers that are members of the entire clus-ter. The server that currently hosts this resource becomes its owner and is responsible

for processing all requests for its services.

Jupiterimages


4/20

need for additional mechanisms that enable them todecide which is responsible for handling the processingof client requests targeting any of the highly available

resources at any given time. This determination mustbe made for every new incoming request and, depend-ing on the configuration, might have to be performedindependently for each of them.

With Server Clustering, the equivalent process is trivialsince there is only one instance of each highly availableresource. The cost, however, is increased complexity ofthe logic that dictates which member server hosts thisresource, especially following the failures of its previousowner.

To function as a unit, servers participating in a cluster(also referred to as nodes) must be able to interact witheach other. This is accomplishedby setting up redundant networkconnections so as to minimizethe possibility of failure. Thus,each node should have at leasttwo network adapters. The con-nections are organized into twogroups, private and public, alsoreferred to as "Internal Clustercommunications only" and "AllCommunications," respectively.They are identified and config-ured during cluster installationon each member server.

The first one contains links dedicated to internode, intr-acluster traffic. Although the primary purpose of thesecond one is to carry service requests and responsesbetween clients and the cluster, it also serves as a back-up to the first one. Depending on the number of nodesin a cluster (and your budget), you can employ differenttechnologies to implement node interconnects. In thesimplest case (limited to two nodes), this is possible

with a crossover cable. When a larger number ofservers participate in a cluster (up to a total of eightsupported by Windows Server 2003 Enterprise andDatacenter Editions) a preferably dedicated hub or aswitch is needed.

To optimize internode communication, which is criticalfor a cluster to operate properly, we recommendedeliminating any unnecessary network traffic on the pri-

vate network interfaces. This is accomplished by:

Disabling NetBIOS over TCP/IP. Relevant options

are listed in the NetBIOS section on the WINS tab ofthe Advanced TCP/IP settings dialog box of the inter-face properties

Removing file and printer sharing for MicrosoftNetworks. Configurable on the General tab of theinterface properties dialog box

Setting appropriate speed and duplexity mode.Rather than relying on Autodetect option - Donefrom the Advanced tab of the network adapterProperties dialog box

Ensure that statically assigned IP addresses areused. Instead of using DynamicHost Configuration Protocol orAutomatic Private IP Addressing.

There should be no defaultgateway. Entries should becleared for the "Use the follow-ing DNS server addresses"options, present on the InternetProtocol Properties dialog boxfor the connection

It is no longer necessary to dis-able the Media Sensing feature

on Windows Server 2003. This was accomplished byregistry modification on Windows 2000-based clustermembers.

Despite these extra measures, communication betweennodes can still fail. This makes it necessary to providean additional safety mechanism that would prevent aso-called "split-brain" scenario, where individual nodes,unable to determine status of clustered resources,

attempt to activate them at the same time. This wouldviolate the principles of server clustering describedabove and result in potentially serious implications,such as data corruption in the case of disk-basedresources.

Quorum DesignationsTo prevent this, every cluster contains one designatedresource, called Quorum, implemented as a dedicated



Every cluster contains onedesignated resource, calledQuorum, implemented as a

dedicated disk volume. Mostfrequently, this volume

consists of a pair of mirroreddisks, which increases level

of its fault tolerance.


5/20

disk volume. Most frequently, this volume consists of apair of mirrored disks, which increases the level of itsfault tolerance. Its optimum size is 500 MB (due to

NTFS characteristics), although its use typically consti-tutes only a fraction of this capacity. Like with otherresources, only one server owns the Quorum at anygiven time. The Quorum owner has the ultimateresponsibility for making decisions regarding ownershipof all other resources.

More specifically, nodes exchange "heartbeat" signals,formatted as User Datagram Protocol (UDP) packets atpre-configured intervals (every 1.2 seconds) to confirmtheir network interfaces are operational. The absenceof two consecutive packets triggers a reaction that is

supposed to address potential cluster problems. InWindows 2000 Server-based implementations, this con-sisted of activating all resources on the current ownerof the Quorum and, simultaneously, deactivating themon all other nodes. This effectively ensured only a sin-gle instance of each resource remained online.However, under certain circumstances, it could lead toan undesirable outcome.

Although a rather rare occurrence, it is possible for theQuorum owner to lose connectivity on all of its inter-faces and, at the same time, the remaining nodesremain able to communicate with the client's network.As the result, user requests will not be able to reachcluster resources, which are still active but reside on thenode that is no longer accessible. Remaining nodes,however, would be fully capable of handling theserequests, if they can take ownership of the Quorumand all other resources.

The introduction of additional logic in the wayWindows Server 2003-based clusters handle theabsence of heartbeat traffic resolved this issue. Ratherthan following the legacy procedure when missingheartbeat signals are detected, nodes first check

whether any of their network interfaces designated aspublic are operational and, if so, whether client net-works are still reachable. This is accomplished by send-ing ICMP (Internet Control Message Protocol) echorequests (i.e., executing PING) to external systems -typically the default gateway configured for these inter-faces. If the node hosting the Quorum fails any of thesetests, it will voluntarily deactivate all its resources,including the Quorum. If the remaining nodes discovertheir network links are still working, they will have no



There's still plenty of mileage left in Microsoft

Windows Server 2003, but it doesn't hurt to

look ahead. You won't find any killer features

in Windows Server 2008, but that's not to say there's

nothing to get excited about. There's a great deal

that's new, and depending on the set up of your

organization, it's almost certain you'll find some or

all of it extremely valuable.

Any ranking is bound to be subjective, and bearing

that in mind, here are what we believe to be the 10

most interesting new features in Windows Server

2008.

1. Virtualization

Microsoft's Hyper-V hypervisor-based virtualization

technology promises to be a star attraction of Server

2008 for many organizations.

Although some 75 percent of large businesses have

started using virtualization, only an estimated 10

percent of servers are running virtual machines.This means the market is still immature. For

Windows shops, virtualization using Server 2008

will be a relatively low-cost and low-risk way to dip a

toe in the water.

At the moment, Hyper-V lacks the virtualized infra-

structure support virtualization market leader

VMware can provide. Roy Illsley, senior research

analyst at U.K.-based Butler Group, noted that

Microsoft is not as far behind as many people seem

to think. "Don't forget Microsoft's System Center,

which is a fully integrated management suite and

which includes VM Manager. Obviously it onlyworks in a Wintel environment, but if you have

Server 2008 and System Center, you have a pretty

compelling proposition.

"What Microsoft is doing by embedding virtualiza-

tion technology in Server 2008 is a bit like embed-

ding Internet Explorer into Windows," said Illsley.

"This is an obvious attempt to get a foothold into the

virtualization market."

10 Coolest Features in

Windows Server 2008by Paul Rubens


6/20

problem establishing a new Quorum owner and transfercontrol of all cluster resources to it.

Besides assisting with resource arbitration followingcommunication failure, Quorum serves another impor-tant function - providing storage for up-to-date clusterconfiguration. This configuration resides in two files inthe MSCS folder on the quorum volume - the clusterhive checkpoint file (Chkxxx.tmp) and Quorum log(Quolog.log). The first one stores a copy of configura-tion database, which mirrors the content of the Clusterregistry hive on the server hosting the Quorum resourceand stored in the %SystemRoot%ClusterCLUSDB file onthat server. This database is replicated to all remainingnodes and loaded into their Registry (maintaining a sin-

gle "master" copy of this information ensures its consis-tency). Replication takes place for every new clusterconfiguration change, as long as all nodes are opera-tional. If this is not the case, timestamped changes arerecorded in the Quorum log file and applied to config-uration database once the offline nodes are broughtback online. Being familiar with these facts is importantwhen troubleshooting some of the most severe clusterproblems.

As already mentioned, Quorum is implemented as avolume on a physical disk. However, details of thisimplementation vary depending on a number of fac-tors, such as number of nodes, server cluster type, orstorage technology.

Maintaining a single instance of each clustered resource(ensuring at the same time its fault tolerance and pre-venting "split-brain" scenarios) is accomplished throughtwo basic mechanisms, resource virtualization and inter-node communication.

Resource virtualization requires each clustered serviceor application be represented by a number of relatedsoftware and hardware components, such as disks, IP

addresses, network names, and file shares, which canbe assigned to any server participating in the clusterand easily transferred between them, if necessary. Thisis made possible by setting up these servers in a veryspecific manner, where they can access the same set ofshared storage devices, reside on the same subnet, andare part of the same domain. For example, to create ahighly available network file share, you would identify ashared disk drive hosting the share, an IP address (withcorresponding network name) from which the share can


Clustering and Storage with Windows Server 2003[ ]10 Coolest continued

At launch, Microsoft is unlikely to have a similar

product to VMware's highly popular VMotion (which

enables administrators to move virtual machines

from one physical server to another while they are

running), but such a product is bound to available

soon after.

2. Server Core

Many server administrators, especially those used

to working in a Linux environment, instinctively dis-

like having to install a large, feature-packed operat-

ing system to run a particular specialized server.

Server 2008 offers a Server Core installation, which

provides the minimum installation required to carryout a specific server role, such as for a DHCP, DNS,

or print server. From a security standpoint, this is

attractive. Fewer applications and services on the

server make for a smaller attack surface. In theory,

there should also be less maintenance and manage-

ment with fewer patches to install, and the whole

server could take up as little as 3Gb of disk space

according to Microsoft. This comes at a price -

there's no upgrade path back to a "normal" version

of Server 2008 short of a reinstall. In fact there is no

GUI at all - everything is done from the command

line.

3. IIS

IIS 7, the Web server bundled with Server 2008, is a

big upgrade from the previous version. "There are

significant changes in terms of security and the

overall implementation, which make this version

very attractive," said Barb Goldworm, president and

chief analyst at Boulder, Colo.-based Focus

Consulting. One new feature getting a lot of atten-

tion is the ability to delegate administration of

servers (and sites) to site admins while restricting

their privileges.

4. Role-Based Installation

Role-based installation is a less extreme version of

Server Core. Although it was included in 2003, it is

far more comprehensive in this version. The con-

cept is that rather than configuring a full server

install for a particular role by uninstalling unneces-

sary components (and installing needed extras), you

simply specify the role the server is to play, and

Windows will install what's necessary - nothing

more. This makes it easy for anyone to provision a


7/20

be accessed remotely, and target file share, with itsname and access permissions.

Although this sounds complicated and time consuming,all necessary resources are pre-defined, making thisprocedure fairly straightforward. Once resources areidentified and configured (by specifying disk drive let-ters, assigning unique IP addresses, network names, orfile share characteristics), they can be assigned to anyserver participating in the cluster (as long as each one iscapable of supporting them). Resources can then beeasily moved between nodes in case the one currentlyhosting them fails.

The Importance of Quorum

Inter-node communication is facilitated through heart-beat signals carried over redundant network connec-tions between cluster members and through Quorum'spresence, which determines how resource ownershipshould be handled. As we pointed out, Quorum hasthe additional important function of storing the mostup-to-date cluster configuration, copied subsequentlyto a dedicated registry hive on each node. Local copiesare referenced when nodes join the cluster during start-up. Because of its significance in clustering architecture,Quorum also serves as the basis for three main serverclustering models:

Single Shared Quorum: Quorum is implemented asthe Physical Disk clustered resource.

Single Local Quorum: Quorum is implemented asthe Local Quorum clustered resource.

Majority Node Set Quorum: Quorum is implement-ed as the Majority Node Set clustered resource.

Single Shared Quorum clusters are by far most popularamong server cluster implementations. They most

closely match the traditional clustering design (which isreflected by continuing support for this model sinceintroduction of Microsoft Cluster Server in Windows NT4.0 Server Enterprise Edition), offering high-availabilityof resources representing wide variety of services andapplications as well as simplicity of installation and con-figuration.

As their name indicates, Single Shared Quorum clustersuse storage design, which enables them to access the



particular server without increasing the attack sur-

face by including unwanted components that will

not do anything except present a security risk.

5. Read Only Domain Controllers (RODC)

It's hardly news that branch offices often lack

skilled IT staff to administer their servers, but they

also face another, less talked about problem. While

corporate data centers are often physically secured,

servers at branch offices rarely have the same phys-

ical security protecting them. This makes them a

convenient launch pad for attacks back to the main

corporate servers. RODC provides a way to make an

Active Directory database read-only. Thus, any mis-chief carried out at the branch office cannot propa-

gate its way back to poison the Active Directory sys-

tem as a whole. It also reduces traffic on WAN links.

6. Enhanced Terminal Services

Terminal services has been beefed up in Server

2008 in a number of ways. TS RemoteApp enables

remote users to access a centralized application

(rather than an entire desktop) that appears to be

running on the local computer's hard drive. These

apps can be accessed via a Web portal or directly by

double-clicking on a correctly configured icon on

the local machine. TS Gateway secures sessions,which are then tunnelled over https, so users don't

need to use a VPN to use RemoteApps securely over

the Internet. Local printing has also been made sig-

nificantly easier.

7. Network Access Protection

Microsoft's system for ensuring that clients connect-

ing to Server 2008 are patched, running a firewall

and in compliance with corporate security policies -

and that those that are not can be remediated - is

useful. However, similar functionality has been and

remains available from third parties.

8. Bitlocker

System drive encryption can be a sensible security

measure for servers located in remote branch

offices or anywhere where the physical security of

the server is sub-optimal. Bitlocker encryption pro-

tects data if the server is physically removed or

booted from removable media into a different oper-

ating system that might otherwise give an intruder

access to data that is protected in a Windows


8/20

same set of disks from every cluster member. Whileunderlying hardware varies widely (and might involvesuch types of technologies as SCSI, SANs, NAS, or

iSCSI, which we will review more closely later), the basicpremise remains the same.

Only one instance of any specific resource is permittedat any given time within the cluster. The same appliesto Quorum, located on a highly available disk volume,physically connected via a SCSI bus, Fibre Channellinks, or network infrastructure to all servers participat-ing in the cluster. Ownership of the shared volume isarbitrated to ensure it is granted only to a single node,thus preventing other nodes from accessing it at thesame time (such situation would likely result in data cor-

ruption).

This arbitration is typically handled using internal SCSIcommands (such as SCSI reserve and SCSI release) aswell as bus, Target, or Logical Unit Number (LUN)resets. The specifics depend on the type of storage

technology implemented. Note that support for a clus-tering installation is contingent on strict compliancewith the Hardware Compatibility List (which is part ofthe Windows Server Catalog, containing all clusteringsolutions certified by Microsoft). Therefore it is criticalthat you verify which system you intend to purchaseand deploy. Quorum, in this case, is implemented asthe Physical Disk resource, which requires having a sep-

arate volume accessible to all cluster nodes (clusteringsetup determines automatically whether the volumeyou selected satisfies necessary criteria).

Unfortunately, the majority of hardware required to setup clustered servers is relatively expensive (althoughprices of such systems are considerably lower than theywere a few years ago), especially if the intention is toensure redundancy for every infrastructure component,including Fibre Channel and network devices, such as



environment. Again, similar functionality is avail-

able from third-party vendors.

9. Windows PowerShell

Microsoft's new(ish) command line shell and script-

ing language has proved popular with some server

administrators, especially those used to working in

Linux environments. Included in Server 2008,

PowerShell can make some jobs quicker and easier

to perform than going through the GUI. Although it

might seem like a step backward in terms of user

friendly operation, it's one of those features that

once you've gotten used to it, you'll never want to

give up.

10. Better Security

We've already mentioned various security features

built into Server 2008, such as the ability to reduce

attack surfaces by running minimal installations,

and specific features like BitLocker and NAP.

Numerous little touches make Server 2008 more

secure than its predecessors. An example is Address

Space Load Randomization - a feature also present

in Vista - which makes it more difficult for attackers

to carry out buffer overflow attacks on a system by

changing the location of various system services

each time a system is run. Since many attacks relyon the ability to call particular services by jumping

to particular locations, address space randomization

can make these attacks much less likely to succeed.

It's clear that with Server 2008 Microsoft is treading

the familiar path of adding features to the operating

system that third parties have previously been pro-

viding as separate products. As far as the core serv-

er product is concerned, much is new. Just because

some technologies have been available elsewhere

doesn't mean they've actually been implemented.

Having them as part of the operating system can be

very convenient indeed.

If you're running Server 2003 then, now is the time

to start making plans to test Server 2008 - you're

almost bound to find something you like. Whether

you decide to implement it, and when, is up to you.

Besides assisting with resourcearbitration following communication

failure, Quorum serves another

important function - providing storagefor up-to-date cluster configuration.


9/20

adapters and switches, or disk arrays and their con-trollers. The cost might be prohibitive, especially forprogrammers whose sole goal is developing cluster-

aware software or exploring the possibility of migratingexisting applications into clustered environment.

To remediate this issue, Microsoft made such function-ality available without specialized hardware setup, byallowing the installation of a cluster on a single serverwith local storage only (also known as a single nodecluster). Obviously, such configuration lacks any degreeof high availability, but it has all features necessary forapplication development and testing. Since local disksare not represented as Physical Disk resources, thisclustering model requires using a distinct resource type

called Local Quorum when running New Server ClusterWizard during initial setup, which we will review indetails later.

Despite the benefits mentioned earlier (such as a signif-icant level of high availability and compatibility with avariety of hardware platforms, applications, and servic-es), Single Shared Quorum has limitations. The first oneis inherent to the technologies used to implement it.For example, configurations relying on SCSI-basedshared storage are restricted by the maximum length ofthe SCSI bus connecting all cluster nodes to the samedisk array (which typically forces you to place them inthe same or adjacent data center cabinets). This dis-tance can be increased considerably by switching to aFibre Channel infrastructure, but not without significantimpact on hardware cost. Introducing iSCSI and NASinto the arsenal of available shared storage choicesprovides the same capability at lower prices, but thereare still some caveats that restrict their widespread use(e.g., NAS devices are not supported as the Quorumresource). The second limitation is that despite redun-dancy on the disk level (which can be accomplishedthrough RAID sets or duplexing, with fault-tolerantdisks and controllers), Single Shared Quorum still con-

stitutes a single point of failure.

There are third-party solutions designed to addressboth of these limitations, and with release of Windows2003 Server-based clustering, Microsoft introduced itsown remedy in the form of Majority Node Set (MNS)Quorum. Like Local Quorum, MNS is defined as a sep-arate resource that must be selected during clustersetup with New Server Cluster Wizard. Also like LocalQuorum model, dependency on the shared storage

hosting Quorum resource is eliminated, without havinga negative impact on high availability.

The level of redundancy is increased by introducingadditional copies of Quorum stored locally on eachnode (in the %SystemRoot%ClusterMNS.%ResourceGUID%$%ResourceGUID%$MSCS folder, where%ResourceGUID% designates a 128-bit unique identifi-er assigned to the cluster at its creation). As you canexpect, having more than one Quorum instancerequires a different approach to preventing "split-brain" scenario. This is handled by defining a differentrule that determines when the cluster is consideredoperational (which, in turn, is necessary to make itsresources available for client access). For this to hap-

pen, more than the half of cluster nodes must be func-tioning properly and able to communicate with eachother. The formula used to calculate this number is:

[(total number of nodes in MNS cluster)/2] + 1

where the square brackets denote Ceiling function,returning smallest integer equal to or larger than theresult of dividing total number of nodes by two. Forexample, for a five-node cluster, three nodes wouldneed to be running and communicating for itsresources to be available (the same would apply to afour-node cluster). Clearly, setting up a two-node MNScluster, although technically possible, does not makemuch sense from availability perspective (since onenode's failure would force the other one to shut downall of its resources). For an MNS cluster to function, atleast two servers (in a three-node cluster) must be oper-ational (note that with a Single Shared Quorum, a clus-ter might be capable of supporting its resources evenwith one remaining node).

Effectively, the rule guarantees that at any given pointthere will be no more than a single instance of everycluster resource. Clustering service on each node is

configured to launch at boot time and to try to estab-lish communication with majority of other nodes. Thisprocess is repeated every minute if the initial attemptfails.

This solution introduces additional requirements, sinceits architecture implies existence of multiple copies ofthe clustered data (unlike with the Single SharedQuorum model), which must be consistently main-tained. Although the clustering software itself is respon-




10/20

sible for replication of Quorum configuration across allnodes, this does not apply to services and application-specific data. In general, there are two ways of han-

dling this task. The first one relies on mechanisms builtinto the application (e.g., log shipping in SQL Server2000/2005 deployments). The second one involves set-ting up replication on file system or disk block level.This can be handled through software or hardware, atopic we plan to elaborate on later in this eBook.In addition, since clustered resources are virtualized,some of the restrictions placed on the Single SharedQuorum model still apply. In particular, for resourcefailover to take place, nodes must be able to detectfailure of others through the absence of heartbeat sig-nals. This requires round trip latency between nodes be

no longer than 500 ms -- affecting, in turn, the maxi-mum allowed distance between them. They also mustbe members of the same domain and their public andprivate network interfaces have to reside on the samesubnets (which can be accomplished through settingup two VLANs spanning multiple physical locationshosting cluster nodes).

Furthermore, since Quorum updates are handled vianetwork file shares called %ResourceGUID%$ (associat-ed with the Quorum location listed earlier), both Server

and Workstation services (LanManServer andLanManWorkstation, respectively) must be running onall cluster nodes and File and Printer Sharing forMicrosoft Networks must be enabled for both privateand public network connections.

Thus, when designing architecture it is important tokeep in mind the impact the architectural design willhave on availability of the MNS cluster. For example,setting up two sites separated by a network link with anequal number of nodes in each will cause both to fail ifcommunication between them is severed (since neither

one contains majority of nodes). It might be beneficialin such situation to set up a third site with a single clus-ter node in it (and dedicated network links to the othertwo sites), dedicated exclusively to establishing majoritynode count when needed. Alternatively, you can alsoforce some of the cluster nodes to host resources,although this requires manual intervention.




11/20

So far, we've provided a brief overview of serverclustering, and described its three basic types,categorized according to characteristics of the

Quorum resource (i.e., Single Shared, Single Local, and

Majority Node Set). Next up, we will examine one ofthe most important of clustering components - diskstorage.

The importance of disk storage stemsfrom its role in the server clusteringarchitecture. As you might recall fromearlier discussions, the Quorumresource must be implemented as anNTFS volume, hosting Quorum logand Checkpoint files (for moredetails, refer to our earlier article).

Just as relevant is the ability to imple-ment the Physical Disk resource (sep-arate from the Quorum resource),which is required in the overwhelm-ing majority of typical clustered appli-cations.

To comply with server clustering prin-ciples, storage must have certaincharacteristics. More specifically, thevolumes it hosts must be accessible to all clusternodes; a critical requirement for the Single Shared clus-

ter category is concerned. This applies to many deploy-ments but not Single Local or Majority Node Set types.

Cluster Service CommunicationThe storage must also be able to communicate withthe Cluster Service, an instance of which runs on every

node, via SCSI protocol. This does not limit hardwarechoices to SCSI disks, channels, and controllers; howev-er, disks and controllers must becapable of properly processing (andsharing a channel for proper transmit-ting) such SCSI commands as Reserve(used by individual cluster nodes toobtain and maintain exclusive owner-ship of a device), Release (which relin-quish reservation of a device, allowinganother cluster node to take owner-ship of it), and Reset (forcibly remov-

ing existing reservation of an individ-ual device or all devices on the bus).

These commands serve a very impor-tant purpose - they prevent a situa-tion where two hosts would be per-mitted to write simultaneously to thesame disk device. This is likely to hap-pen otherwise, considering bothhosts share a physical connection to

it. When the first cluster node is brought online, itsCluster Service (with help of the Cluster Disk DriverClusdisk.sys) scans the devices of the shared storagebus and attempts to bring them online. It issues theReserve command to claim ownership. The same com-



Disk Storage (SCSI)

A SCSI controller is typically installed in a host system as the host adapter, but it canalso reside in an external storage subsystem.

Jupiterimages


12/20

mand gets re-sent in subsequent, regular intervals(every three seconds). This ensures ownership is main-tained, and the owning node has exclusive access to all

volumes on the target disk.

Reserve also plays a critical role in cases when networkcommunication between nodes fails. As mentioned ear-lier, such a situation is handled by first establishingwhich node is the owner of the Quorum (potentiallytriggering new election, if the previous owner is nolonger operational) and transferring all cluster resourcesto it. A successful outcome of this process relies on twosettings that control SCSI commands issued by clusternodes. The first one forces the Quorum owner to renewits reservation every three seconds. The second one,

inter-node communication failure, causes non-Quorumowners to initiate bus-wide Reset, followed by a seven-second waiting period.

If the Quorum remains available after the wait period isover (which indicates the previous Quorum ownerfailed), the challenging node takes over ownership ofthe Quorum (by sending Reserve signal) as well as allremaining resources. Another purpose of Reset com-mand is to periodically terminate reservations to detectsituations in which a node becomes unresponsive (with-out failing completely). Providing that this is not thecase, reservations are subsequently re-established.

Now that we have established functional requirementsof storage in Single Shared Quorum clusters, let'sreview technologies that satisfy criteria outlined above.Regardless of your choice, the actual hardware selectedmust be Microsoft-certified, which can be verified byreferencing Microsoft Windows Server catalog). In gen-eral, storage clustering solutions belong to one of fourcategories:

Shared SCSI Fibre Channel Storage Area Networks (SANs)

NAS (Network Attached Storage) iSCSI

SCSISCSI (Small Computer System Interface) is the bestknown and most popular storage technology for multi-disk configurations. The term SCSI also refers to thecommunication protocol, providing reliable block-leveldata transport between a host (known as the initiator)

and storage (known as the target), which is independ-ent of the way data is stored. Its architecture consists ofa parallel I/O bus shared between multiple (frequently

daisy-chained) devices (including controllers), andenclosed on both ends with terminators, which preventelectrical signals from bouncing back (terminators arefrequently built directly into SCSI devices).

A SCSI controller is typically installed in a host systemas the host adapter, but it can also reside in an externalstorage subsystem. Each device on the bus is assigneda unique identifier referred to as SCSI ID that is num-bered from 0 to 7 or from 0 to 15, for narrow and wideSCSI bus types, respectively. In addition to providingaddressing capabilities, the SCSI ID determines priority

level (with an ID 7 being the highest and assigned typi-cally to the controller, ensuring proper bus arbitration).

A limited range of SCSI IDs (which restrict the numberof devices on the bus to 15) is extended through theassignment of Logical Unit Numbers (LUNs), associatedwith each individual storage entity, which is able toprocess individual SCSI commands. Typically, they rep-resent individual disks within a storage subsystem, con-nected to the main SCSI bus via an external SCSI con-troller. In addition to LUN and SCSI ID, the full addressof such Logical Unit also contains a bus identifier, whichcommonly corresponds to a specific SCSI interfacecard. A server can have several such cards installed.The total number of available LUNs ranges from 8 to254, depending on the hardware support for LargeLUNs. For more information on this subject, refer toMicrosoft Knowledge Base article 310072.

Implementing SCSI technology for the purpose ofshared clustered storage adds an extra layer of com-plexity to its configuration. Since the bus must beaccessible by clustered nodes, install a SCSI controllercard in each (and disable their BIOS). Furthermore,since these controllers will be connected to the same

bus, they cannot have identical SCSI IDs. Typically, thisdilemma is resolved by setting one to 7 and the otherto 6, which grants the latter the next-highest prioritylevel. To ensure the failure of a single component (suchas a device, controller, or host) does not affect theentire cluster, use external (rather than device's built-in)terminators. Keep in mind that number of nodes in aSCSI storage-based clustered implementation cannotexceed two.




13/20

As part of your design, you should ensure sufficientlevel of storage redundancy by implementing RAID,which enables individual disk failures to not affect over-

all data accessibility. Although Windows 2000 and 2003Server products support software-based fault tolerantRAID configurations (RAID 1 and 5, known also as mir-roring and striping with parity, respectively), thisrequires setting up target disks as dynamic, which inturn are not permitted - at least not without installingthird-party products (e.g., Symantec StorageFoundation for Windows add-in) as shared clusteredstorage This restriction does not apply to local clusternode drives.

Although this means that you must resort to more

expensive, external storage arrays, which implementhardware-based RAID, you can benefit not only fromsignificantly better performance but also fromimproved functionality, including such features asredundant hot swappable fans, power supplies, extradisk cache memory, and more complex and resilientRAID configurations (such as RAID 10 or 50, whichcombine disk mirroring with striping or striping withparity, protecting from losing data access even in casesof multiple disk failures).

Unfortunately, the SCSI technology, despite its relativelylow cost, widespread popularity, and significant transferspeeds of up to 320 MBps with SCSI-3 Ultra320 stan-

dard is subject to several limitations. They result mainlyfrom its parallel nature, which introduces a skew phe-nomenon (where individual signals sent in parallel arriveat a target at slightly different times), restricting themaximum length of the SCSI bus (in most implementa-tions, remaining within 25 meters range, requiringphysical proximity of clustered components, whichmakes them unsuitable for disaster recovery scenarios).A recently introduced serial version of SCSI (SerialAttached SCSI, or SAS) addresses the drawbacks of itsparallel counterpart, but it is unlikely to become ameaningful competitor to Fibre Channel or iSCSI. The

SCSI bus is also vulnerable to contention issues, wherea device with higher priority dominates communication.Finally, storage is closely tied to the hosts, whichincreases the complexity of consolidation and expan-sion efforts.

Although shared SCSI technology is a viable option forlower-end server clustering implementations on theWindows 2003 Server platform, other types of storagesolutions offer considerable advantages in terms of per-formance, scalability, and stability.




14/20

Fibre Channel storage area network (FC SANs) rep-resent a considerable shift from the directlyattached storage paradigm. They offer significant

functionality and performance improvements. The basic

idea is to use a network infrastructure for connectingservers to their disks, allowing physical separation ofthe two by far greater distances than was previouslypossible. But there are also other,equally important, advantages of thisseparation. Managing storage inlarger environments no longerrequires dealing with each individualsystem, as was the case with directlyattached models. Disks are groupedtogether, simplifying their administra-tion (e.g., monitoring, backups,

restores, provisioning and expansion)and making it more efficient, throughsuch inventions as LAN-free or serv-er-free backups and restores, orbooting from a SAN.

In addition, since large number ofservers and storage devices can par-ticipate in the same SAN, it is possi-ble to attach new ones as needed,making allocation of additional space a fairly easy task.This is further simplified by the DISKPART.EXE

Windows 2003 Server utility, which is capable ofdynamically extending basic and dynamic volumes, asexplained in Microsoft Knowledge Base Article

Q325590. This is especially true when comparing theSAN with a SCSI-based setup, where the limitedamount of internal or external connectors and adjacentphysical space available must be taken into account.

Fibre Channel SAN technology leverages SCSI-3 speci-fications for communication between hosts and target

devices, since its implementation isbased on the SCSI command set.Their transmission, however, is han-dled using FC transport protocol.This is done in a serial manner, typi-cally over fiber optic cabling(although copper-based media areallowed), which eliminates distancelimitations inherent to parallel SCSI.

Note, however, that the term "net-work" should not be interpreted inthe traditional sense, since SANs donot offer routing capabilities, primari-ly because they are intended forhigh-speed, low-latency communica-tion. SANs also use a distinct endnode identification mechanism,which does not rely on Media Access

Control (MAC) addresses associated with each networkadapter, but instead employs 64-bit (expressed usually

in the form of eight pairs of hexadecimal characters)World Wide Names (WWN), burned into fibre host busadapters (HBAs) by their manufacturers. FC intercon-



SAN-Based Storage

FC SANs represent a considerable shift from the directly attached storage paradigm.They offer significant functionality and performance improvements.

Jupiterimages


15/20

necting devices handle dynamic address allocation onthe fabric level. In addition, unlike majority of IP-basednetworks, FC SANs have primarily asymmetric charac-

ters, with active servers on one end connecting mostlyto passive devices, such as disks arrays or tape driveson the other, arranged in one of the following topolo-gies:

Point-to-point: Links a single host and its storagedevice directly via a fiber connection. This type ofconfiguration is the simplest and least expensive todeploy and manage, but it lacks the flexibility andexpandability of the other two, since it is conceptual-ly equivalent to SCSI-based directly attached disks. Itis, therefore, rarely implemented.

Shared, also known as Fibre Channel ArbitratedLoop (FC-AL): Takes the shape of a logical ring (butphysically forming a star), with an FC hub or a loopswitch serving as the interconnecting device. Thedesign is similar to Token Ring architecture. This simi-larity is also apparent when it comes to arbitration ofloop usage.

Since FA-CL devices share the same media, whenev-er one of them needs to communicate with another,it is must send an arbitration packet around the loop,which once returned back to the sender, signalsexclusive loop access can be granted. Should con-flicts occur when multiple devices attempt to commu-nicate at the same time, the one with the lowestaddress wins. Addresses, which differentiate amongall nodes participating in the loop, can be hard-coded or assigned dynamically. The majority of loopswitches provide this capability. Although dynamicallocation simplifies configuration in multi-node sce-narios, it might also cause instability when devicesare restarted or new ones added, since such eventstrigger loop reinitialization and node readdressing.

Although considerably less expensive than theirswitch-based counterparts, FC-ALs are not as effi-cient. Access to fabric is shared across all intercon-nected devices, which allows only two of them com-municate at any given time. They are also not as scal-able and support fewer nodes - the maximum is 126.As with SCSI-based shared storage, FC-AL-basedWindows 2003 Server clusters are limited to twonodes. In addition, Microsoft recommends using arbi-

trated loops for individual cluster implementations,rather than sharing them with other clusters or non-clustered devices. Larger or shared implementations

require switched configuration.

Switched, referred to as Switched Fibre ChannelFabric (FC-SW): These networks use FC switchesfunctioning as interconnecting devices. This topologyaddresses the efficiency limitations of the loop con-figuration by allowing simultaneous, dedicated pathsat the full wire speed between any two Fibre-attached nodes. This is based on the same principleas traditional LAN switching. Scalability is greatlyincreased due to hierarchical, fully redundant archi-tecture. It consists of up to three layers with core

employing highest speed and port density switches,distribution relying on midrange hardware, andaccess characterized by low-end switches, arbitratedloops, and point-to-point connections.

Switches keep track of all fabric-attached devices,including other switches, in federated and cascadedconfigurations, using 3-byte identifiers. This sets thetheoretical limit of roughly 16 million unique address-es. Stability is improved as well, since restarts andnew connections are handled gracefully, withoutchanges to an already established addressingscheme or having a negative impact on the status ofthe fabric. This is partially because of the introductionof less disruptive, targeted LUN and SCSI ID resets,which are attempted first before resorting to the bus-wide SCSI Reset command. Previously, this was theonly available option in Windows 2000 Server clusterimplementations. Keep in mind, however, that theavailability of this feature depends on the vendor-developed HBA specific miniport driver, which mustbe written specifically to interact with the Microsoft-provided StorPort port driver. This is a new feature inWindows 2003 Server. It is designed specifically totake advantage of performance enhancing capabili-

ties of FC adapter, rather than legacy SCSIPort.

Increased performance, flexibility, and the reliability ofswitched implementations come with their own set ofdrawbacks. Besides considerably higher cost (com-pared to arbitrated loops) and interoperability issuesacross components from different vendors, one of themost significant ones is the increased complexity ofconfiguration and management. In particular, it is fre-




16/20

quently necessary to provide an appropriate degree ofisolation across multiple hosts connected to the samefabric and shared devices with which they are sup-

posed to interact.

As mentioned earlier, this exclusive access is requiredto avoid data corruption, which is bound to happenwith unarbitrated, simultaneous writes to the same diskvolume. In general, three mechanisms deliver this func-tionality - zoning, LUN masking (known also as selectivepresentation), and multipath configurations.

Zoning can be compared to Virtual LANs (VLANs) intraditional networks, since it defines logical boundaries(known in SAN terminology as zones) that encompass

arbitrarily designated switch ports. Zone definitions inclustered deployments are typically stored andenforced by the switch port ASIC (Application-SpecificIntegrated Circuits) firmware, with communication per-mitted only between nodes attached to the switchports that belong to the same zone. They can also beimplemented by referencing WWN of host busadapters. In addition to preventing accidental data cor-ruption, zoning offers also an additional level of securi-ty. It protects the server from unauthorized access. Inclustered configurations, cluster nodes, along with theshared disks that constitute clustered resources, shouldbelong to the same zone.

LUN (an acronym for Logical Unit Number, describing alogical disk defined in a FC SAN) masking makes itpossible to limit access to individual, arbitrarily selectedLUNs within a shared storage device. Such functionalityis typically required in configurations involving largemultidisk systems, where port-level zoning does notoffer sufficient granularity. LUN masking provides nec-essary isolation in cases of overlapping zones, wherehosts or storage devices belong to more than onezone. The relevant configuration is performed andstored on the storage controller level.

Multipath technology is the direct result of the strive forfull redundancy in SAN environment. Such redundancyis available on the storage side (through fault-tolerantdisk configurations, dual controllers with their own ded-icated battery-backed caches and power supplies) andon the server side (through server clustering, with eachof the member servers featuring dual, hot-swappablecomponents). It is reasonable to expect the same when

it comes to SAN connectivity.

Unfortunately, the solution is not as simple as installing

two FC host bus adapters (HBAs) and connecting themto two redundant switches, each of which in turn,attaches to separate FC connections on the storagecontroller. This is because without additional provisions,Windows would detect two distinct I/O buses and sep-arately enumerate devices connected to each (resultingin a duplicate set of drives presented to the operatingsystem), which could potentially lead to data corrup-tion. To resolve this issue, Microsoft Windows 2003Server includes native support for Multipath I/O, whichmakes it possible to connect dual HBAs to the sametarget storage device with support for failover, failback,

and load balancing functionality.

Each implementation of a Windows 2003 Server clustermust belong to a dedicated zone, to eliminate poten-tial adverse effect of the disk access protection mecha-nism included in the clustering software on otherdevices. This does not apply, however, to storage con-trollers, which can be shared across multiple zones, aslong as they are included on the Cluster/Multi-ClusterDevice HCL. In addition, you should avoid collocatingdisk and tape devices in the same zone, as the SCSIbus reset commands can interfere with normal tapeoperations.

Remember, the rule regarding consistent hardware andsoftware setup across all cluster nodes extends to SANconnections - including host bus adapter models, theirfirmware revision levels, and driver versions.

You should also ensure that automatic basic disk vol-ume mounting feature is disabled. This does not applyto volumes residing on dynamic disks or removablemedia, which are always automatically mounted. Earlierversions of Windows would spontaneously mount everynewly detected volume. In a SAN environment, thiscould create a problem if zoning or LUN masking was

misconfigured or if prospective cluster nodes hadaccess to the shared LUNs prior to installation of theclustering software. This feature is configurable, anddisabled by default, in Windows 2003 Server. Runningthe MOUNTVOL command or using AUTOMOUNToption of the DISKPART utility can control it.




17/20

So far, we discussed the two most common choices- direct-attached SCSI (with popularity resultingfrom its long-lasting, widespread commercial pres-

ence and low pricing) and Fibre Channel storage-area

networks (FC SANs), which are frequently chosenbecause of their superior performance and reliability.

Unfortunately, the cost associatedwith FC SAN deployments is pro-hibitive for most smaller or less-critical environment, whoserequirements cannot be satisfiedwith parallel SCSI because of itsperformance and scalability limita-tions. The introduction of iSCSIresolves this dilemma by combin-

ing the benefits of both technolo-gies and at the same time avoid-ing their biggest drawbacks.

iSCSI is an acronym derived fromthe term Internet SCSI, which suc-cinctly summarizes its basic prem-ise. iSCSI uses IP packets to carrySCSI commands, status signals,and data between storage devicesand hosts over standard networks. This approach offerstremendous advantage by leveraging existing hardware

and cabling (as well as expertise). Although iSCSI fre-quently uses Gigabit Ethernet, with enterprise class

switches and specialized network adapters (containingfirmware that processes iSCSI-related traffic, offloadingit from host CPUs), its overall cost is lower than equiva-lent Fibre Channel deployments. At the same rate,

however, features, such as addressing or automaticdevice discovery built into FC SAN infrastructure, mustbe incorporated into iSCSI specifications and imple-

mented in its components.

iSCSI communication is carriedover a TCP session between aniSCSI initiator (for which functional-ity is provided in Windows 2003 inthe form of software or a mix ofHBA firmware and Storport mini-port driver) and an iSCSI target

(such as a storage device), estab-lished following a logon sequence,during which session security andtransport parameters are negotiat-ed. These sessions can be madepersistent so they are automatical-ly restored after host reboots.

On the network level, both initia-tor and target get assigned unique

IP addresses, which allow for node identification. Withnode identification, the target is actually accessed by a

combination of IP address and port number, which isreferred to as portal. In the iSCSI protocol, addressing



iSCSI Storage

Unfortunately, the cost associated with FC SAN deployments is prohibitive for mostsmaller or less-critical environments, whose requirements cannot be satisfied with paral-

lel SCSI because of its performance and scalability limitations.

Jupiterimages


18/20

is typically handled with iSCSI Qualified Name (IQN)convention. Its format consists of the type identifier(i.e., "iqn."); registration date field (in the month-year

notation); followed by the period and domain in whichthe name is registered (in reversed sequence); thesemicolon; and the host (or device) name, which can beeither autogenerated (as is the case with Microsoftimplementation, where it is derived from the computername), preassigned, or chosen arbitrarily, serving as adescriptor providing such information as device model,location, purpose, or LUN.

Targets are located either by statically configuring soft-ware initiator, by specifying target portal parameters(and corresponding logon credentials), by leveraging

functionality built into HBAs on the host, or discoveredautomatically, using information stored on an InternetStorage Name Server (iSNS). This server offers a cen-tralized database of iSCSI resources, where iSCSI stor-age devices are able to register parameters and status,which subsequently can be referenced by initiators.Access to individual records can be restricted based ondiscovery domains, serving a purpose similar to FCSAN zoning.

In a typical Microsoft iSCSI implementation, the initiatorsoftware running on a Windows host server (with acompatible NIC or an HBA that supports MicrosoftiSCSI driver interface), is used to mount storage vol-umes located on iSCSI targets and registered with iSNSserver.

Installation of the initiator includes iSNS client andadministrative features, in the form of the iSCSI Initiatorapplet in the Control Panel and Windows ManagementInstrumentation and iSCSI Command Line interface(iSCSICLI). The software-based initiator lacks some ofthe functionality that might be available with hardware-based solutions (such as support for dynamic volumesor booting from iSCSI disks).

To provide a sufficient level of security and segregation,consider isolating iSCSI infrastructure to a dedicatedstorage network (or separating the shared environmentwith VLANs), as well as applying authentication andencryption methods. With Microsoft implementation,authentication (as well as segregation of storage) ishandled with Challenge Handshake AuthenticationProtocol (CHAP), relying on a password shared

between an initiator and a target, providing that thelatter supports it. Communication can be encrypteddirectly on end devices, using built-in features of high-

end iSCSI HBAs, third-party encryption methods, orMicrosoft's version of IPSec.

Although network teaming is not supported on iSCSIinterfaces, it is possible to enable communicationbetween an initiator and a target via redundant net-work paths that accommodate setup with multiple localNICs or HBAs and separate interconnects for each.Implementing multiple connections per session (MCS),which leverage a single iSCSI session, can do this. Itcan also be done with Microsoft Multipath I/O (MPIO),which creates multiple sessions. The distribution of I/O

across connections (applied to all LUNs involved in thesame session) or sessions (referencing individual LUNs),for MSC and MPIO, depends on Load Balance Policiesconfigured by assigning Active or Passive type to eachof network paths. This results in one of the followingarrangements:

Fail Over Only uses a single active path as the pri-mary and treats all others as secondaries, which areattempted in round-robin fashion in case the primaryfails. The first available one found becomes the pri-mary.

Round Robin distributes iSCSI communicationevenly to all paths in round-robin fashion.

Round Robin with Subset functions with one set ofpaths in the Active mode and the other remainingPassive. The traffic is distributed according to theround robin algorithm across all active paths.

Weighted Path selects a single active path by pick-ing the lowest value of arbitrarily assigned weightparameter.

Least Queue Depth, available only with MCS, sendstraffic to the path with the fewest number ofrequests.

The multipathing solution selected depends on a num-ber of factors, such as support on the target side,required level of granularity of Load Balance Policy(individual LUN or session level), and hardware compo-nents (MCS is recommended in cases where a software-




19/20

based initiator - without presence of specialized HBAson the host side - is used). Regardless of your decision,take advantage of this functionality as part of your clus-

tering deployment to increase the level of redundancy.When incorporating iSCSI storage into your Windows2003 Server cluster implementation (note that Microsoftdoes not support it on Windows 2000), also ensuresthat components on the host side fully comply withiSCSI device logo program specifications and basicclustering principles. Take domain and networkdependencies into account. Also bear in mind thatbesides SCSI RESERVE and RELEASE commands(which provide basic functionality), iSCSI targets mustsupport SCSI PERSISTENT RESERVE and PERSISTENTRELEASE to allow for all of the Load Balance policies

and persistent logons.

The latter requires a persistent reservation key be con-figured on all cluster nodes. Choosing an arbitrary 8-byte value, with the first 6 bytes unique to each cluster

and the remaining 2 bytes varying between its nodes,does this. Data is entered in thePersistentReservationKey REG_BINARY entry of the

HKLMSystemCurrentControlSetServicesMSiSCDSMPersistentReservation registry key on each cluster member.In addition, the UsePersistentReservation entry ofREG_DWORD type is set to 1 in the same registry loca-tion. You should also enable Bind Volumes InitiatorSetting (in the Properties dialog box of the iSCSIInitiator Control Panel applet), which ensures all iSCSIhosted volumes are mounted before the ClusterService attempts to bring them online.

To avoid network congestion-related issues, considersetting up dedicated Gigabit Ethernet network or

implementing VLANs with non-blocking switches sup-porting Quality of Service. Optimize bandwidth utiliza-tion, by implementing Jumbo frames and increasingvalue of Maximum Transmission Unit.




20/20

We've reviewed the general principles of server clustering and presented hardware and software criteria that mustbe taken into consideration in its design. While the cost of implementing this technology has decreased signifi-cantly in recent years, making it affordable outside of high-end environments, there are still scenarios where its usemight not be economically viable (such as in development, testing, or training).

Fortunately, it is possible to overcome constraints imposed by its storage or network requirements without any sig-nificant hardware investments by leveraging widely popular server virtualization methodology.

This content was adapted from Internet.com's ServerWatch Web site and was written by Marcin Policht.


Conclusion

Internet.com eBooks bring together the best in technical information, ideas and coverage of important ITtrends that help technology professionals build their knowledge and shape the future of their IT organizations.For more information and resources on storage, visit any of our category-leading sites:

www.Enteprisestorageforum.comwww.internetnews.com/storagewww.linuxtoday.com/storagewww.databasejournal.comhttp://news.earthweb.com/storagehttp://www.internet.com/storage

For the latest live and on-demand Webcasts on storage, visit: www.internet.com/storage

Documents

17285826 Clustering and Storage With Windows Server 2003