AARNet’s Australian Network

©Copyright AARNet Pty Ltd

AARNet’s Australian Network

Positioning Australia in a Global Gigabit Research Network Environment


What’s different about AARNet?Australia’s Academic and Research Network

• Has been a user-pays network since inception in 1989• Until mid 1995 provided all internet access in Australia (including to ISPs)• In mid-1995 sold its commercial customer base to Telstra (Australia’s

dominant carrier) – used proceeds to develop ATM-based network• In 1998 became a not-for-profit company limited by shares, shareholders

are 37 Australian universities and the CSIRO• In 1999 started to roll out a national VoIP service – now switching 30,000

calls per day• In 2000 became a licenced carrier in Australia (number 61)• In 2001 acquired an IRU between Australia and the US (155Mbps)• In 2002 started to develop non-traditional alliances with power utilities to

gain access to regional fibre• 2003 was a big year (more later)


Key Strategic Relationships

• with Powerlink (North Queensland) and TransGrid (NSW) – power utility companies that provided fibre to regional areas – resulted in gigabit capacity where not previously economically feasible

• with Australian Government and the Australian Research and Education Network Initiative (AREN) – provided funding to catalyse key regional initiatives

• with Southern Cross on SXTransPORT (more later)• with Leightons (construdtion company) – 2 fibre pairs across

Australia as basis for AARNet3 and regional gigabit connectivity (eg to remote telescopes)

• with AUSAID (Australian Aid Agency) to provide a connection from the Fiji Campus of the University of the South Pacific to AARNet and global R&E networks

• with Procket providing high capacity core routing for new network


Gigabit capacity within Australia – challenges and solutions

• Poor response to request to carriers to make dark fibre available– Solution: use AARNet’s carrier licence to carry out civil works

to lay our own fibre– Only needed to do once in any area (Canberra) after this some

enough players willing to sell/lease dark fibre• Poor response from traditional carriers for high capacity connectivity

to regional areas– Solution: form strategic alliances with power utilities for mutual

benefit– Now have affordable gigabit capacity in North Queensland and

inland NSW• Tasmania, an island state south of Melbourne, monopoly carrier, no

affordable high capacity option– Solution: form strategic alliance with State govt, gas pipeline

group to deploy alternative undersea cable capacity


Grasp the rare opportunities - NextGen

• Both of the two new fibre build initiatives in Australia went into receivership in 2003

• One of these (NextGen) had a footprint of fibre that was of great interest to AARNet’s clients (the universities, research bodies, especially the astronomers)

• AARNet made a bid in its own right initially, but ended up partnering with the construction company that built the network (Leightons)

• Concluded agreement on Christmas eve after a 5 months process• Provides 2 fibre pairs Australia wide (Brisbane to Perth)• 1st pair lit as 10 Gbps for inter-capital trunks• 2nd pair equipped by APL as regional fibre loops• Access to Space in all capital city central offices and controlled

environment vaults (huts) along the routes


Overview - 10Gbps Backbone

• Two fibre pairs on each path– STM-64 (OC192) service provided on first pair for inter capital trunks– Second pair to be lit by APL to allow Gigabit Ethernet drop off to

regional members– Member must provide tail to the regional network


Overview - Equipment

• Core Router– Procket Networks PRO/8812– 40Gbps capable– Packet over SDH to STM-64– Gigabit and 10 Gigabit Ethernet

• Core Switch– Cisco Systems Catalyst 6509-NEB-A with Sup720– Pure L2 switching– Fast, Gigabit and 10 Gigabit Ethernet only

• Member Edge and POP based “Legacy” routers– Cisco Systems 7304 with NPE-G100– 3 x Gigabit Ethernet (Member, POP “A”, POP “B”)– 1 x Fast Ethernet dedicated to flow accounting– Capability to handle legacy (slow) interfaces


Design Issues

• Redundancy & Resilience• Support for IPv4 and IPv6

–unicast and multicast• Traffic Accounting and Monitoring• End to end performance measures• Support QoS (diffserv)• Support for large traffic flows, jumbo frames


Redundancy & Resilience

• Dual points of presence (POP) in major capital cities• Diverse, dual unprotected national links

–Will use MPLS Fast Reroute for protection–Provides ability to burst above protected capacity

• Use single metro dark fibre pair to connect intra city POP sites

• Creates rings between cities• Provides opportunity for members and customers to

build diverse, redundant connections to AARNet


IPv4 and IPv6

• Native IPv4 and IPv6 (Dual Stack) network–Unicast and Multicast for both IPv4 and IPv6–EFT IPv6 Multicast (initially intra-domain only)–Line rate performance for IPv4 and IPv6–Peering to both R&E and Commodity Internet–Hexago IPv6 Migration Broker to aid member and

client IPv6 deployment• DNS, AARNet Mirror and USENet News accessible

over IPv4 and IPv6• Jumbo frames, 9000 byte


Traffic Accounting and Monitoring

• Flow based accounting• Differentiate traffic into classes for billing• Scaling issues require accounting function to be

moved to the edge of the network• Use anycast addressing so data supplied to a central

collector in an emergency• Centralise reporting to a POP based server• Also provides AARNet with measurement device on

network edge to improve performance monitoring


Australian Intercapital Backbone Network


GigaPOPs

• Dual Multiple Gigabit Points of Presence (GigaPOPs) in Capital Cities

– Brisbane• UQ Prentice Centre & QUT Gardens Point

– Sydney• UTS Broadway & Nextgen Rosebery

– Canberra• ANU Leonard Huxley & TransACT

– Melbourne• UniMelb Law Building & Nextgen West Melbourne

– Adelaide• University of Adelaide, 10 Pulteney Street & Hostworks

– Perth• CSIRO ARRC & AMNet IX


Connections through the GigaPOPs


Overview - International Links

• Add drop offs to existing STM-1/OC3 (155Mbps) – Mixed traffic - R&E and Commodity– University of South Pacific, Fiji– Possibly Auckland, New Zealand– Connects to 155Mbps path to Tokyo from Hawai‘i

• Dual STM-4 (622Mbps)–Commodity Internet transit - NTT/Verio and MCI– PAIX Palo Alto (Silicon Valley)– Los Angeles

• “SX TransPORT” - Dual STM-64 (10Gbps) – Research and Education (R&E) traffic only - AUP– Hawai‘i - Manoa and Seattle (Abilene, CA*net 4)– Los Angeles (Abilene, CENIC, CUDI)– Look to add Mauna Kea to Los Angeles path later


Dual 155Mbps circuits – regional and Pacific Island connectivity

• 4-drop ring (4 unprotected circuits, MPLS, fast failover)–Sydney-Suva (new)–Suva-Oahu (Hawaii) – and then to U Hawaii, Manoa–Oahu-Hillsboro – and then to Pacific Wave, Seattle–Hillsboro-Sydney

• AUSAID to provide funding for the University of the South Pacific to contract AARNet to provide connections to AARNet and the global R&E Networks (monopoly carrier issues)

• Interconnect at Hawaii with new Hawaii –Tokyo link• USPNet to other Pacific Islands to be upgraded later• Possible connection of NZ later


Dual 155Mbps circuits – regional and Pacific Island connectivity


Dual 622Mbps circuits – commodity traffic

Access Commodity Internet in Palo Alto–Connected to the PAIX peering fabric–Obtain transit from MCI and NTT/Verio–Peer with other organisations at PAIX

Add second commodity POP in Los Angeles–Need to determine

– data centre location– backhaul from Morro Bay (San Luis Obispo)• transit providers, at least one in common with

Palo Alto site


Dual 622Mbps circuits – commodity traffic


Dual 10Gbps circuits – SXTransPORTTrans Pacific Optical Research Testbed

• Major new partnership announced between AARNet and Southern Cross Cable Network (SX) on 11 December 2003

• Support from Australian Government• Dual 10Gbps circuits between Australia and US West Coast• Activation mid-2004, Initial term 5 years, option for 5-year term• AUP applies similar to Abilene• Advisory Board to be set up with reps expected from SX,

AARNet, global research interests• Will provide affordable high throughput access allowing

Australia to effectively participate in global e-science initiatives• Will strengthen case for locating unique research facilities in

Australia, eg Square Kilometre Array


SXTransPORT and Hawaii in the Pacific


Dual 10Gbps circuits – SXTransPORTAustralia Connections

• Southern Cross Configuration–First circuit

• Sydney - Oahu - Hillsboro, Oregon–Second circuit

• Sydney – San Luis Obispo, California• In Australia, each circuit will be linked to a different AARNet

PoP in Sydney–AARNet PoPs will be linked together as well as to

AARNet3’s 10Gbps backbone infrastructure in Australia


Dual 10Gbps circuits – SXTransPORTU.S. Connections

• First circuit: Sydney - Oahu - Hillsboro, Oregon–Oahu connection to Hawaii GigaPoP

at University of Hawaii, Honolulu–Oregon connection to Pacific Wave in Seattle

via NLR from Portland• Second circuit: Sydney – San Luis Obispo, California

–Initial peering via CalREN-2 at Cal Poly SLO?–Moving to LA-based peering via NLR–Planning for new connection on Big Island of Hawaii to

include Mauna Kea Observatories


International Points Of Presence

• Existing–University of Hawai‘i, Manoa, Oahu–Pacific Wave, Westin Building, Seattle–PAIX, Palo Alto

• Planned–Los Angeles (exact location yet to be determined)–University of the South Pacific, Suva, Fiji

• Possible–Auckland–Singapore–Tokyo


Support Services

• DNS Cache and Secondary Servers• Usenet News• Hexago IPv6 Migration Broker• DDoS Detection and Mitigation

–Investigate appliances–Interest in automatic detection and filtering–Locate next to transit (and peering) links

• AARNet Mirror• VoIP Gateways• NLANR and/or RIPE Test Traffic Measurement


Indicative timeframe to June 2004 – initial phase

• Build PAIX Palo Alto POP – Done (Feb 04)• 1st Commodity link - PAIX to UTS – Done (Feb 04)• Intra Sydney link - March 04• 1st Melbourne-Sydney link - March 04• Canberra-Sydney link - April 04• 1st Brisbane-Sydney link - May 04• 1st Melbourne-Adelaide-Perth link - May 04• 1st “SX TransPORT” connection (Seattle) - June 04• 2nd Commodity link - LA to SYD #2 - June 04


AARNET’s International Connections


QUESTNet 2004 and 18th APAN meeting 2-7 July 2004 Cairns Australia


QUESTNet 2004 and 18th APAN meeting 2-7 July 2004 Cairns Australia

www.aarnet.edu.au

Documents

AARNet’s Australian Network