How do you spell relief? If you’re northeastTennessee’s largest health care system and

you’re distressed by an aging, temperamental, under-powered ATM network that hiccups at the least provo-cation, you build a Gigabit Ethernet backbone and run itover dark fiber you’ve coaxed out of a reluctant telco.

Then you add a little Course Wavelength DivisionMultiplexing (CWDM), the passive optical technologytargeted at metro networks, and you’ve built yourself anenterprise network infrastructure that’s good for at leastfive or six years. At least, that’s the prediction of DarrenRamsey, the senior network specialist who shepherdedthe migration from ATM to Gigabit Ethernet for theWellmont Health System, a leading health care providerin northeast Tennessee and southwest Virginia.

When fully operational this summer, the new multi-service infrastructure will solve a plethora of business-critical issues for Wellmont. Not the least of these isdelivering a simplified, all IP-based network withenough bandwidth to easily transport the health careorganization’s patient records, as well as graphics filescreated by the hospital system’s Agfa electronic med-ical imaging system.

The new infrastructure will also add high-availabil-ity failover redundancy for disaster recovery betweenWellmont’s three main facilities; Differentiated ServicesCode Point (DSCP) for end-to-end QoS; and the abilityto deploy several bandwidth-intensive applications, in-cluding medical imaging, health information systems,VoIP, and an IP SAN that supports both Internet SCSI

(iSCSI) and Fibre Channel over IP (FCIP) over the newbackbone. All of this will be carried over three pairs ofleased fiber optic circuits from Sprint, which initiallybalked when Wellmont inquired about renting its un-used dark fiber capacity.

When matched with the appropriate CWDM boxesand modules from Cisco Systems, Wellmont can scale

in practice

48 NETWORK MAGAZINE I 07.04 I www.networkmagazine.com

+

by Jim Carr

Darren Ramsey shepherded Wellmont’s migration from an aging ATMbackbone to a new Gigabit Ethernet infrastructure run over CWDM.

The Wellmont Health Systemis a leading health careprovider in northeast Ten-nessee and southwest Vir-ginia. Created in a merger oftwo equals—Bristol RegionalMedical Center and HolstonValley Medical Center—in1996, the health care systemconsists of five medical cen-ters and hospitals, 40 clinics,

an assisted living center, ahospice, and a wellness cen-ter. Its services range fromcommunity-based acutecare to specialized tertiaryservices that include neona-tal intensive care and twotrauma centers.

Because the merger oc-curred so quickly, the ensu-ing entity’s IT departments

had little time for long-termplanning, instead deployinga 25Mbit/sec ATM backbonebetween the two main facili-ties. By 1999, that backbonewas reaching its limit, andwith the addition of severallarge mission-critical applica-tions, it became apparentthat the hospital needed torebuild its infrastructure.

To that end, the organiza-tion staged a phased de-ployment of Gigabit Ether-net and plans to add CWDMcapabilities in the near fu-ture. This should ensure thatits infrastructure is “good forfive or six years,” accordingto Darren Ramsey, WellmontHealth System’s senior net-work specialist.

he long and short of it : : :

Healing Network IllsThe Wellmont Health System turns to Gigabit Ethernet and CWDM over dark fiberto rebuild its antiquated ATM infrastructure.

its Gigabit Ethernet backbone up to 8Gbits/sec over itsexisting fiber circuits, although initial plans call forusing CWDM just for 2Gbit/sec connectivity.

These changes didn’t happen quickly, Ramsey ad-mits. In fact, he likens the task of migrating Wellmont’sinfrastructure from ATM to Gigabit Ethernet to that ofturning an ocean liner: “Everything is so big and inter-dependent, it’s difficult to make a drastic change—it’shard to turn the ship quickly.”

But turn it around Wellmont did, and here’s how.

MERGED ENTITYWellmont, one of northern Tennessee’s largest employerswith about 4,500 health care professionals, was formed in1996 in a merger between Bristol Regional Medical Center(BRMC) and Holston Valley Medical Center (HVMC). Eightyears later, Wellmont’s integrated health care system con-sists of five medical centers and hospitals, 40 clinics, anassisted living center, a hospice, and a wellness center.

According to Craig Rogers, Wellmont’s manager oftechnical services, the merger occurred so quickly thatthe two organizations’ IT departments had little time forlong-range design and planning. In the interest of expe-diency, they deployed an ATM backbone, the de-factobackbone technology at the time, with 25Mbit/secpoint-to-point Permanent Virtual Circuits (PVCs) be-tween the two main hospitals.

Despite the fact that BRMC was primarily a Unixshop and HVMC an IBM mainframe house, and thatboth had a mixture of Novell NetWare 3 and 4 servers,

everything worked fine for a few years, says Ramsey.But by 1999, the ATM backbone had started showingits age. Several factors were at work here.

First, the organization had purchased a data centerthat would allow it to consolidate all of its IT resources,including servers, into a single location. That, of course,necessitated an architectural change in the ATM back-bone. As a result, Wellmont moved from a point-to-pointenvironment to a multipoint environment with two PVCsrather than one, adding bandwidth and complexity.

Wellmont also added several new enterprise-wideapplications to its existing health information system,which provides patient records and financial function-ality already in use. This included Microsoft Outlookfor e-mail, and Agfa’s Picture Archiving and Communi-cations System (PACS), which is used at four of Well-mont’s facilities as a film-free diagnostic imaging solu-tion. In addition, the workstation count grew to morethan 2,000, according to Wellmont CIO Steve Hill.

Despite this growth, Wellmont’s network consistedof a single layer-2 unrouted subnetwork. With Novell’sIPX and IP running across this backbone, it was nosurprise that the network began to experience severeperformance degradation, including broadcast storms,says Ramsey.

It got to the point where Wellmont’s networking staffwas troubleshooting the organization’s ATM switches on adaily basis. The staff experienced “strange issues” that theATM vendor was unable to fix. It was obviously time forsome changes, according to Ramsey.

www.networkmagazine.com I 07.04 I NETWORK MAGAZINE 49

Core Topology

HVMC CORP BRMCRouters

1 Gigabit

1 Gigabit

1 Gigabit

1 Gigabit

1 Gigabit

1 Gigabit

CWDM 1550nmCWDM 1590nm

Routers Routers

Wellmont’s new Gigabit Ethernet infrastructure consists of leased fiber optic circuits (denoted by the solid white lines) and two 1 GigabitCWDM “lambdas” (indicated by the blue and yellow lines). Expanding the backbone’s CWDM capacity requires adding modules to thehospital’s Cisco Gigabit Interface Converter CWDM devices, which operate in conjunction with the hospital’s Cisco routers.

MOVE TO ROUTINGThe first step in remedying Wellmont’s infrastructureproblems was to implement a routed network. Becausethis required “touching” almost every device on the net-work (minus workstations, which are assigned IP ad-dresses automatically via DHCP) and because many ofthese devices, including servers, were moved around,the process took almost two full years, says Ramsey.

Once the network was routed and somewhat stable,Wellmont began eliminating IPX and its NetWareservers, shutting down the last one in April 2002.

By then, it was evident that Wellmont had outgrownits 25Mbit/sec ATM backbone, says Ramsey. In studyingthe options, Wellmont rejected going with a bigger ATMpipe. “We thought ATM was probably dead as an enter-prise networking technology and Ethernet is much sim-pler, with a smaller learning curve,” he says.

Wellmont also looked at the Multilink Point-to-PointProtocol (MLPPP), which aggregates up to eight T1 cir-cuits into a 12Mbit/sec link. “But 12Mbits/sec is a farcry from 25Mbits/sec, so we knew that wouldn’t work,”Ramsey explains.

Gigabit Ethernet, on the other hand, had recentlybeen standardized, “and we knew the prices wouldcome down in 2000/2001,” he says. “We knew wecouldn’t trench it ourselves,” so Wellmont started look-ing for dark fiber to lease.

“Initially, we went to Sprint, but they didn’t want tooffer the service,” says Ramsey. “They didn’t want to doit because there was no competition.”

After six to nine months of negotiations, however, theentry of several Competitive Local Exchange Carriers(CLECs) into the area market changed Sprint’s attitude.

Although several CLECs had fiber for lease, “Sprint madethe deal attractive because it could bundle all voice, data,and long distance into the contract,” says Ramsey. “Sprintfound out we might go with someone else, it made us agood deal, and we ended up paying less a month for fourfiber optic links than our two ATM PVCs had cost.”

Wellmont subsequently leased fiber circuits betweenits three main sites—BRMC, HVMC, and the corporatedata center—in a multi-point-to-point arrangement, withtwo independent rings between the three sites connectingAlcatel switches at each site. That configuration gave

Wellmont’s link redundancy with a failover connectionshould one fiber line go down, but not switch redundancy.

From then on, “The network was like night and day,”says Ramsey. “It went from everything being the net-work’s fault to the point where the network was no longerguilty until proven innocent when an application failed.”

ELIMINATING ATM Even with these improvements, Wellmont still relied onATM for wiring closet-to-backbone connectivity, withabout 35 closets per facility. Its next step was to elimi-nate ATM entirely and move strictly to Ethernet.

With a slew of new internetworking equipment to pur-chase, Wellmont standardized on Cisco as its “next-gener-ation vendor,” as Ramsey puts it. He says sticking with onevendor simplifies implementation in enterprise networks.

In 2002 and 2003, Wellmont deployed a wide varietyof Cisco boxes to begin rolling out its Gigabit Ethernetbackbone. This included placing Catalyst 3550 10/100Ethernet switches in the wiring closets, Catalyst 4506Ethernet switches in Wellmont’s medium-sized facili-ties, and Catalyst 6509 multiprotocol switches in each ofthe BRMC, HVMC, and corporate data center locations.

By the start of 2004, Wellmont’s Gigabit Ethernet back-bone was up and running smoothly, performance issueswith the health information systems had vanished, anddownload times—particularly those involving medicalimages—were dramatically improved. At that point, Ram-sey says Wellmont’s management brought up several for-ward-looking issues to the IT department. These includeddeploying a disaster recovery plan in tandem with a SAN,and running VoIP over the new infrastructure.

“With the backbone up, we felt a lot of pressure todevelop a business continuity plan,” he explains. Thatlead to the deployment of 80Tbytes of mirrored storage

money matters

in practice

50 NETWORK MAGAZINE I 07.04 I www.networkmagazine.com

Sprint initially balked when Wellmont inquired aboutrenting its unused dark fiber capacity.“ ”

Price of a two-year upgrade to a Gigabit Ethernet backbone

Monthly savings migratingfrom 25Mbit/sec ATM toleased fiber optic circuitsvia Sprint

Cost of CWDM equipmentto expand the backbone to 8Gbits/sec

>$1Million

>$1K$50K

in EMC Symmetric, Clarion, and Centara optical stor-age arrays. All are linked via IP over the Gigabit Ether-net backbone. The SAN currently supports FCIP, butRamsey says Wellmont plans to use a CWDM circuit toextend the hospital system’s Brocade Fibre Channelswitches to its three main sites. That will give it redun-dant SAN transport with a pure Fibre Channel overFCIP, he says.

Meanwhile, Wellmont also began moving some of itsvoice traffic onto the IP network. First, it added Nortel Net-works’ Internet Telephony Gateway (ITG) VoIP cards to itsphone system, which is based on Nortel Option PBXs.This allowed site-to-site calls within the organization totraverse Wellmont’s backbone network, eliminating severalTime Division Multiplexing (TDM) Primary Rate Interfaces(PRIs) and the charges associated with many long distancephone calls. In patient care areas, the hospital also inte-grated SpectraLink’s 802.11b-compatible handsets into theVoIP system.

“Once we started doing all that—we have our PACSimages, patient files, and SAN and voice traffic on thenetwork—we were in a position where the networkcan’t fail,” says Ramsey. “We needed 99.999 percent up-time, so we started looking at what we needed for thedata center to survive a catastrophic failure.”

That meant making yet another architectural shift inthe backbone. Wellmont worked with Sprint to redeploy

its fiber circuits from a strictly point-to-point topology toa ring topology and added a second redundant Catalyst6509 switch to each of the three main sites.

This left Wellmont with direct links between itsmain facilities. Such a configuration ensured that a fail-ure on any communications link or switch wouldn’tnegatively impact the core of Wellmont’s backbone, giv-ing Ramsey his “five nines” reliability.

It also paved the way for Wellmont to deployCWDM. Ramsey says all he has to do is drop a CiscoCWDM Gigabit Interface Converter (GBIC) with the ap-propriate Optical Add-Drop Modules (OADMs) betweenthe routers in the health care provider’s three main sitesand the fiber optic circuit, “and the routers think welease eight pairs of fibers.” He plans to double the back-bone’s capacity, saving the potential eightfold band-width increase for future growth.

All this was done at bargain prices. By eliminatingthe need for a second fiber circuit, which costs $10,000to $15,000 per month to lease, Ramsey says he’s lookingat a three-month payback on his $50,000 investment inCWDM equipment.

NOT INEXPENSIVERamsey admits the backbone project itself wasn’t so in-expensive, however. He says from 2002 to 2004, it costthe organization more than $1 million: “More than wespent in the previous four years, but it’s money we hadto spend,” he says.

“We could not afford to not do it,” he adds. “We’relooking at half a million dollars lost per hour if our sys-tems go down. How many downtimes would it take topay for $1 million? I’d rather not have any,” says Ramsey.

“Besides, it gives us the ability to do things we could-n’t do before—VoIP, wireless, the SAN Fibre Channel overCWDM—and we’re set for the next five or six years.”

www.networkmagazine.com I 07.04 I NETWORK MAGAZINE 51

2

1 10Gbit/sec Ethernet that’s moremature and comparable in price perport to 1Gbit/sec Ethernet. A 10Gbit/sec backbone would be hard tosaturate in the foreseeable future

That wireless switches had beenmainstream when standaloneaccess points were deployed. Had itbeen available, a different designapproach might have been taken

» wish list : : :

I s y o u r e n t e r p r i s e m a k i n g i n n o v a t i v e u s e o f a n e t w o r k -i n g t e c h n o l o g y o r s e r v i c e t h a t y o u ’ d l i k e u s t o w r i t ea b o u t ? C o n t a c t J i m C a r r , a n A p t o s , C A - b a s e d f r e e l a n c eb u s i n e s s a n d t e c h n o l o g y w r i t e r , a t j e c a r r 1 3 @ c h a r t e r . n e t .

lessons learnedTIPS FOR

DEPLOYING A GIGABIT

ETHERNETBACKBONE

WITH CWDM

1) Always make network monitoring a priority. Wellmont lost aredundant core switch and didn’t figure out what went wrong forabout a week. 2) Always purchase more bandwidth than youthink you need today. You’ll use it sooner than later. 3) Documen-tation is key on the day of the switchover. 4) Standardizing on asingle network vendor in the enterprise simplifies implementa-tion. 5) Design, test, and prove your solution in a lab environment.