View
640
Download
1
Category
Preview:
DESCRIPTION
Citation preview
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 1
Smart Networking Techniques in Implementing Broadband Hybrid Wide Area Networks
by
John Puetz, President MasterWorks Communications and David Blanks, CTO, Avalon Inc.
Hybrid networks, comprising satellite, fiber-cable and wireless-local-loop technologies, are becoming more commonplace as service providers and enterprise IT managers seek out the best-fit network implementation for Internet access, wide area networking and virtual private networks. Economics and performance factors demand that hybrid systems be investigated and seriously evaluated. This paper presents two case studies of hybrid networks that have been implemented in the past year. For each case an overview of the technology trade-offs is presented, along with the network details and underlying technology advances that made the network implementation possible. In addition, the economics of the implementation are provided.
Introduction
Capacity, network reach, and economics are arguably the three most important
considerations faced by service providers and enterprise IT managers today, as they rollout
or expand their networks. The amount of data used in business, and the information
required to stay competitive, is roughly doubling every six months. Access to the Internet,
and a presence on the web, have become a necessity in today’s corporate environment.
Moving large amounts of data into, around, and out-of the enterprise network for national
and multi-national corporations is becoming increasing more difficult in a cost constrained,
bandwidth-limited environment.
While fiber and other broadband technologies have become more widely deployed, no
single technology can meet the capacity, geographical reach, and financial considerations
that face my many IT managers. Service providers, especially network and Internet access
providers, are faced with increasing demands for higher capacity access closer to the end-
users and a wider offering of services. For economic and performance considerations a
combination of network technologies is becoming more prevalent as the “best-fit” approach.
In concept, hybrid networking makes use of the “best” features and benefits of each
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 2
technology, while minimizing the “rough spots” that might occur at the seams where these
technologies fit together. There-in lies the challenges facing equipment manufactures,
integrators, service providers and users of hybrid networks.
This paper presents two cases studies of cutting-edge hybrid network implementations. The
first network involves using broadband satellite VSATs (very small aperture terminals) to
greatly extend the reach of an existing international frame relay network for real-time data
collection and infrastructure communications in an oil exploration and services company.
Mobility and high data capacity were key requirements for this IP based virtual private
network. A new satellite routing protocol (SRP) was implemented and integrated with a
flexible rate, IP on-demand satellite networking capability to implement a transportable WAN
environment. Additional enhancements were implemented to permit automatic remote node
registration and network logon. The resulting network is international in scope, covering
North America and portions of South America and Central Africa.
The second network details a fiber/satellite/microwave network implemented by a start-up
network access provider (NAP) in bringing services to ISPs and businesses in Australia. A
wireless domestic network is implemented using microwave and satellite that bypasses the
incumbent telecom carriers. This allows the service provider to implement an alternative
service that brings a new level of performance, pricing and service to the Australian
marketplace.
Case Study 1 – Broadband WAN with a long reach
Travel expenses and lost employee productivity due to travel time are significant to many
companies around the world. But to a U.S. based Fortune-100 oil services company, lost
productivity of critical expert resources was the key motivation for finding a better way to do
business. Before they deployed their wireless broadband WAN across The Americas and
South Africa, their experts went to where the geological data was—the well sites. These
experts spent half of their time on a job in non-productive travel, getting to exploration sites
at sea or in hard-to-reach land locations. Once on site, they’d analyze large amounts of
data captured in real-time using specialized computing applications, and make drilling or
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 3
process recommendations on the spot yielding immediate results. This highly specialized
service has brought in big money for the company, but the number of jobs that each expert
can support limits company revenues—that is until now. Since they’ve deployed their
broadband WAN infrastructure, the high-volume, real-time well-site data comes to their
experts, who are regionally located in offices and who can now support several operations
at multiple sites concurrently. Results—increased productivity, larger service revenues and
happier employees. And when local conditions warrant, expert on-call help is just a
videoconference or a telephone call away for the drill crews. And for field crews that remain
on-site for offshore rig or shipboard operations, the new network gets used off-hours as
well—for calls home. Thus, the company has significantly altered the model they use to
deploy geological services and plan on expanding services into new markets.
Demanding network requirements for a “mobile” WAN
Oil field exploration operations are a very demanding telecommunications environment. The
virtual network must “reach” hundreds of miles from the nearest frame relay access point, to
provide a consistent, high-quality of service IP pipe, at data rates above 400 Kbps. Also
needed is the ability to support flexible bandwidth capacity to accommodate video feeds and
video conferencing as well as “basic” telephone, email and file transfers services.
Additionally, operational sites may exist only for a few hours or a few days; so network setup
and activation must be fast, less than 30 minutes, and simple—trained oil field personnel
only, no satellite field engineers allowed. For all practical purposes, this is a mobile WAN
environment; although except for shipboard installations, the network won’t operate while
mobile. An additional wrinkle to the mobile environment is that networking equipment
configurations are not always the same. Some field operations required more special
purpose computing equipment, so the local network topologies vary. The network could
have a large number of high-bandwidth applications on line simultaneously, so to make it
economically attractive, the company couldn’t use dedicated links for each. Thus a way to
automatically allocate bandwidth on demand and based on application need (some low
speed, some high speed) was required.
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 4
Implementation Trade-offs
With a worldwide frame relay network infrastructure already in place, the “trick” was
extending the network’s reach wirelessly and at reasonably fast data rates. Prior to the
implementing the broadband VSAT network, lower bandwidth technologies were
experimented with. These included CDPD wireless and low-speed interactive VSATs. While
data rates were only 4.8 to 14.4 Kbps at best, these trials provided a useful function in
proving out the concept and providing an environment for specialized software applications
development. Digital line-of-site microwave links were considered, but except for a few
locations, were deemed too expensive and too restrictive in “reach” capability, with only 20
miles coverage at best without using link repeaters.
Smart broadband VSATs
After evaluating a number of leading DAMA VSAT equipment providers, the oil services
company selected ViaSat as the technology vendor of choice. Key to ViaSat’s technology
edge is the ability to combine intelligent IP routing techniques with a sophisticated
bandwidth on demand control system—both are integrated directly into their broadband
VSAT terminal. IP routes are defined based on ‘Internet Application Profiles’ that take into
account the destination IP address, the requesting application port, and satellite channel
parameters like transmit and receive data rates and the type of connection desired (one-
way, two-way equal rate or two-way unequal rate). Thus when a file transfer is required, the
FTP/IP packets arrive at the VSAT based router and a two-way IP connection established
between the two networks. In this example, the circuit might be configured for a 1 Mbps
transmit rate and a 32 Kbps receive rate. Once the connections are made between the
routers, data transfer begins. The IP route remains active as long as there is traffic transiting
the connection and automatically terminates after a designated inactivity period has expired.
Dozens of possible data rates are available from 9.6 to 2048 Kbps.
Another capability required for successful field operations was the ability to dynamically up-
speed (and down-speed) connection rates as applications come and go, without interrupting
active applications. For example, a baseline WAN connection is established at 64 Kbps in
both directions to support email and general networking functions. But when real-time data
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 5
needs to be transferred from the well-site to the processing center, higher speed
asymmetric IP connection is established. This approach ensures that all of the critical real-
time data gets to the processing center immediately and without congestion or packet loss.
Now if a videoconference connection is also desired, the broadband VSAT can establish yet
another circuit (if to a different destination) or increase the existing 64 Kbps connection to a
rate high enough to accommodate both the video conferencing and the basic WAN
connectivity.
The Hybrid Network Implementation
Critical to the success of deploying wireless networking is the ability to accommodate the
reduced traffic capacity of the wireless networks (e.g., 256Kbps to 2 Mbps) as they are
encountered by the higher speed wired WANs with their 10 to 100 Mbps capacity. Not only
are wireless pipes smaller, but also when standard protocols are used over the increased
delays of satellite, the effects of congestion are amplified, since the feedback mechanisms
within TCP take longer. Thus congestion management and mitigation processes are
important when implementing satellite based WANs. Router queue depth management and
protocol responses play a very significant role in making certain data throughputs are high.
Proprietary TCP/IP throughput enhancement algorithms and quality-of-service (QoS)
features were implemented to ensure the desired throughput levels could be achieved.
The following diagram provides an overview of the overall network. The wired WAN
represented by the network cloud on the right while the remote LANs are located on the left,
with the VSAT nodes and the satellite gateway equipment forming the VSAT WAN cloud.
Multiple VSAT WANs may exist for different regions (e.g., one for North America, a second
for South America).
In a mobile WAN environment, the remote VSAT nodes come and go from the wide area
network as they join and leave the network. For fixed site locations, the remote nodes join
upon initial equipment installation. Routes can either be static or dynamic. Static routing
means the routes are administrated manually (added and deleted) across the entire
network. Unless a network is very small, static routing is difficult to support.
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 6
RemoteNet ‘A’RemoteNet ‘A’
RemoteNet ‘B’RemoteNet ‘B’
RemoteNet ‘C’RemoteNet ‘C’
RemoteNet ‘X’RemoteNet ‘X’
RemoteNet ‘Y’
RemoteNet ‘Y’
Servers
Video Conferencing
WAN - Internet
GatewayLAN
VSATWAN #2
VSATWAN #1
With bandwidth on demand WAN systems, and especially in a mobile WAN environment
where network nodes come and go, it’s vital that the wired WAN network “knows” which
VSAT nodes are reachable and what routes are active. This is the job of routing protocols—
software routines that run on routers to determine who their “neighbors” are and how to
connect to them.
Dynamic routing protocols like RIP (routing information protocol) and OSPF (open shortest
path first), automatically keep the routing tables in all participating routers aware of the
network topology, available connections, and route performance. Internet control message
protocol (ICMP) messages are used to indicate route and connections status among
routers. Because these routing protocols were designed for wired networks where full-time
connections are available, these protocols require non-trivial amounts of data to be
transferred among routers. (While OSPF is more efficient than RIP and provides for better
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 7
routing performance, there is still several hundreds of bytes of information required on a
relatively continuous basis between routers.)
With on-demand satellite WAN networking, connections are not always “on”, as once a
connection becomes inactive, the circuit is deactivated and must be reinitiated if needed
again. Thus to achieve efficient use of bandwidth, distinctions in routing metrics are made
so that active connections are preferred over inactive ones. Furthermore it’s highly desirable
for system capacity and bandwidth utilization reasons, to have the IP routing protocol
operate over the network’s inherent network management and control infrastructure. To
achieve this, ViaSat uses a proprietary routing protocol (SRP) to support routing
management over the satellite network in a “background mode”—all routing housekeeping
is performed using the network control channel bandwidth so that on-demand services are
used exclusively for WAN data traffic. At the Gateway LAN, where the VSAT WAN and the
wire WAN meet, RIPv2 is used to communicate routing updates to the terrestrial routers.
The network supports mesh, star, inverse star (for data collection) and hybrid mesh-star
architectures. To achieve maximum throughput of gateway resources, any gateway VSAT
can communicate to any of remote VSATs. There are no pre-assigned channel allocations
needed or bandwidth restrictions, so approach all gateway resources can be used before
service blocking occurs. Full mesh connectivity ensures that remote VSATs can route IP
packets directly to other remotes. And the network supports voice/fax communications as
well. One common remote-to-remote connection path is for the company’s Gulf of Mexico
port office to communicate directly with the exploration ships that it controls. Data
connections are used for planning and logistic support activities.
The broadband satellite WAN network has been deployed in North America, northern South
America and Central America and in portions of Africa. The African VSAT WAN network
interfaces into the global frame network using a teleport located in the United Kingdom,
while the two America VSAT WAN networks interface at a single facility in Houston, Texas.
Since the networks operate as a single entity, email traffic originating 50 miles offshore
Africa, reaches its destination in a Houston processing facility in a matter of a few seconds.
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 8
A look at the economics
Almost without exception the locations requiring high-speed data connectivity are well
outside the range of existing terrestrial wireless infrastructures. Mobile satellite systems
such as INMARSAT can provide the service coverage but are currently limited in speed to
64 Kbps or 128 Kbps using special HSD (high-speed data) services. Typically these
services run from $6 to $9 per minute with effective data throughputs limited to 50 to 70
percent. This service was evaluated as being far to costly by the energy services company
to be used for anything sort of an emergency. (They do employ INMARSAT and GlobalStar
services for emergency phone communications.)
Another approach to consider is implementing a private wireless infrastructure using
terrestrial digital microwave links. This approach is in use for off-shore installation were a
number of sites are progressively further from shore on a relatively straight line and
separated from one another by less than the line-of-sight range of a microwave link (15 to
20 miles). However, for transportable land use and more typical offshore usage, microwave
is not practical.
As a practical matter, Ku-band and C-band fixed satellite services are economically
appealing. As can be see from the figure below, a 25 node network employing a variety of
technologies can range from $138K to over $600K per year for space segment. The figures
were derived according to the following:
� dedicated: full-time asymmetric connections are used operating at 384 Kbps transmit and 64 Kbps receive. Bandwidth is not shared among terminals.
� BOD: bandwidth on demand configuration whereby bandwidth sharing occurs on a 95% availability figure, with the same transmit/receive rates as noted above.
� BOD with flexible IP rates: same as BOD except that 64 Kbps circuits are used until a high-speed connection is required, then a 384 Kbps transmit circuit is initiated. When the high-speed service is no longer needed then the circuit returns to 64 Kbps.
The assumed space segment rate is $6K per MHz per month, a typically rate paid in the
United States for Ku-band space segment for partial transponder use over a two to three
year period.
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 9
$619
$248
$138
$0
$100
$200
$300
$400
$500
$600
$700
Space Segment Price ($K/yr)
Dedicated BOD BOD with Flex IP
As can be seen from this chart, there’s approximately a 60 percent savings using
bandwidth-on-demand (BOD) compared with dedicated operation (with no perceivable
impact to service quality) and a further 45 percent savings using a dynamic rate approach
that increases (and decreases) link capacity as required.
Case Study 2 – National Network Access Provider goes Wireless and Hybrid
Integrating traditional fiber-wire capacity with satellite and terrestrial wireless technologies is
enabling new business opportunities and services. Case in point is a new start-up network
access provider (NAP) in Australia. From a business perspective, the NAP start-up has
launched three service offerings in designated markets across Australia:
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 10
Fast Internet – with the first implementation for Australian customers of the DIRECT
PEERING concept pioneered in the USA by InterNap Inc – removing the
delays and losses associated with traditional free, public peering models used
by legacy carriers
Virtual Private Networks (VPN’s) – offering Web access, email, IP based Video
Conferencing and Database applications on a single service equipped with
QoS definitions for each traffic type
Streaming Media services – based on local servers supporting Microsoft and Real Networks
streaming products
Market environment overview
Two Tier 1 carriers presently dominate Internet access and distribution in Australia. Adding
to the lack of competition are the market dynamics of insufficient bandwidth and volume
based usage tariffs imposed by the incumbent telcos. Currently high tariff rates exist with an
added encumbrance of penalizing ISP’s for asymmetric bandwidth usage. Both Telstra and
Cable & Wireless/Optus have dominated the supply side of the market, creating a new
environment of opportunity for new service providers.
Within Australia, the duopoly has buried many thousands of route miles of fiber cable, which
is currently inactive and being held as dark fiber. As a result, multiple carriers have wired the
major capital cities (four of which account for 70 percent of Australia’s population). Until
Macrocom installed a Digital Microwave Network, there was no alternative inter-capital
carrier. Once Macrocom provided competition, the duopoly dropped the price of E1 inter-
capital connections by 67 percent.
The lack of competition within the Australian marketplaces has led to a number of thin-route,
independent satellite service providers. Several are using PanAmSat satellites to supply
Internet access to ISPs, although in most cases the return paths are routed terrestrially.
Intelsat provides an alternative satellite source, but until Intelsat’s future privatization occurs,
landing rights are still restricted to C&W Optus and Telstra.
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 11
Network considerations
The network architecture was based on the following considerations:
(a) Independence from the incumbent Telcos within Australia for the core and distribution elements
(b) Route diversity and policy routing of traffic by Class-of-Service criteria (c) End-to-end Quality of Service management (d) Robust and resilient, fault tolerance (e) High availability (fiber networks have a poor reliability record within Australia)
The original network concept was conceived around an ATM platform to implement the
desired quality of service features. But with the availability of multi-protocol label switching
(MPLS) from Cisco in late 1999, a core IP design was selected. The opportunity to have a
simplified routing implementation based upon an “ingress/egress” point definition was a
definite appeal to support the VPN services.
Network implementation
The service is essentially implemented as a wireless virtual domestic network using a
combination of satellite and microwave with fiber distribution in metropolitan areas. The
network connects into the Internet backbone in the United States in two ways. The first route
uses a 45 Mbps DS-3 undersea fiber connection that terminates in Sydney. (The Western
Australia cable route was used due to the non-availability of the new Southern Cross Cable;
Pacrim East and West have no spare capacity.) From Sydney, terrestrial microwave
connects Melbourne and Brisbane. Within each city, local fiber is used to complete the WAN
connection.
The second Internet backbone route is a 45 Mbps satellite link from Verestar’s teleport in
Brewster, Washington that feeds all four cities with Internet service. A lower-speed, 10 Mbps
Internet return connection is uplinked from Sydney. A full 36 MHz transponder is used on
Intelsat 701 (180 E) with high throughput rates achieved from enhanced modulation
techniques and FEC coding. For the other cities the return path to the Internet is made over
the microwave connections back to Sydney. Because the high-speed satellite connection
drops simultaneously into all sites, there is no need for expensive 45 Mbps intra-country
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 12
terrestrial connections. The satellite link also serves as an alternate routing path for the
main fiber connection, providing a level of redundancy and load sharing.
Brisbane
Sydney
Melbourne Undersea Fiber
45 Mbps - Satellite
10 Mbps - Satellite
Brisbane
Sydney
Melbourne Undersea Fiber
45 Mbps - Satellite
10 Mbps - Satellite
As mentioned previously Cisco’s Multi Protocol Label Switching (MPLS) is used and
implemented with Cisco 7500 and 7200 series routers that provide a virtual “cloud” between
the Australian cities and Brewster / Seattle in the USA. To optimize web traffic, intelligent
caching engines were used. Cisco Model 590s were installed at Sydney and Brewster. And
to support streaming video and audio, specialized InfoLibria Media Mall streaming servers
where installed on both ends of the fiber.
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 13
Network Economics
In providing Internet access as a network access provider (NAP), connectivity into the U.S.
backbone is essential. A comparative analysis of satellite and fiber rates indicates that
satellites can provide substantial savings in service provisioning, especially when
considering the asymmetric aspect of Internet access into the backbone. The following
figure provides indicative telecom pricing for T1 or E1 (full-duplex) services between the
designated country and the U.S.
International T1/E1 to US
0 20 40 60 80 100 120
Monthly Rate ($K, US)
FranceSpainVenezuelaPanamaHong KongAustraliaJapan
A number of satellite services providers supply connectivity to numerous ISPs
internationally, with representative pricing to Europe of $18K per month for a 2 Mbps
outbound and a 512 Kbps return link. These rates are very favorable when comparing to the
$25K to $30K for a terrestrial service into Europe and almost $90K into Australia.
Specific to this network, indicative pricing for a DS-3 (45 Mbps) full duplex fiber connection
to the U.S. from Australia ranges from $400K to $600K per month. Using satellite, an
asymmetric service of 45 Mbps outbound and 10 Mbps return link can be established for
approximately $240K per month providing a hefty 40 to 60 percent savings. Furthermore,
since multiple cities are served by the single 45 Mbps outbound, no additional connectivity
charges are incurred to reach them—in short the incremental cost for distributing to
additional sites is minimal. This is in sharp contrast to using fiber to reach the additional
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 14
cities. This approach would require a separate connection for each city thereby increasing
the connectivity charges in a multiplying factor.
Network expansion
The start-up intends to expand beyond the launch network with additional capabilities that
will expand performance or services. These include:
(a) VPN Provisioning partnerships with US Service Providers to extend the scope and reach of VPN’s beyond the POP sites in Seattle and Brewster – also to overseas countries beyond the USA
(b) Network based ASP offerings such as on the fly virus protection, encrypted secure IP sub-networks with encryption nodes at all co-location sites, and IP network based call centers and ACD systems
(c) Advanced compression systems that allow 54:1 JPEG image compression and FTP session compressions at lesser rates – currently widely used in Europe
(d) Widespread installation of streaming media servers at Australian ISP sites– this will lift the majority of ISP’s away from the threatened revenues of simple Internet access
(e) The opening up of the Telstra controlled copper network to competitive ADSL suppliers has resulted in further “last mile” access options to current and prospective customers. Other last mile options include LMDS / MMDS radio ( point to point and multipoint), SDH fiber access ( usually at E1 rate) and HDSL over copper.
(f) IP performance enhancement, using Mentat Sky-X protocol on the Intelsat transponder to ensure constant throughput at maximum rates
Other applications fit satellite based data networking
The two case networks just discussed are but a sampling of types of applications that the
new broadband satellite technology advances are enabling. Additional applications include:
� “instant” communications infrastructure for locations that are geographically remote or have limited capacity or facilities (e.g., only cell phone coverage, dial-up data services, etc.).
� backup data communications and media diversity for critical business operations such as publishing, production and manufacturing industries. With VSAT terminals located directly at the business, a direct end-to-end link is automatically established that bypasses all wired facilities.
� quick service deployment (within hours or days) of temporary or permanent services for expanding network coverage or for new operations. Facilities base service
Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001
Page 15
deployment typically run 2 to 6 months depending on location, and much, much longer in some areas of the world.
� service augmentation of existing wire/fiber infrastructure to alleviate network congestion and provide a “must-get-there” data path for time critical content and information.
� emergency communication services in responding to natural and man-made disasters. Either fixed site, portable or mobile networking environments can be quickly activated or deployed.
� “Best value” transport for multicasting (distributing) data, video, multimedia and Internet.
As more entrepreneurs and IT managers around the world become exposed to the cutting
edge of broadband networking possibilities, the number of hybrid satellite-fiber-WLL
networks will increase dramatically.
Recommended