Wide Area Network Project

Overview of the Forensic Science Service:

• It is an executive agency of the UK Home Office• It is the market leader in the supply of forensic science

services to police forces in England and Wales, as well as being a source of training, consultancy, and scientific support for many overseas and private sector customers

• It is in the vanguard of forensic science technology and has an unrivalled reputation for the integrity, impartiality and accuracy of its findings

Forensic Science Locations and Staff Numbers:

Site Current Staff per SiteBirmingham (Doranda Way)Birmingham (Oldbury)Birmingham (Priory House)Birmingham (Scenesafe)ChepstowChorleyHuntingdonJFK (Warrington)LondonManchesterSheffieldSunderlandTrident Court (HQ)Wetherby

1525310202252303105670151010525280

Total 2650

Legacy Wide Area Network:

• The current WAN was primarily based around a hub and spoke ATM PVC design with Trident Court the HQ acting as the hub

• All sites where dual connected over diversely routed links • The existing network carried a mix of traffic including data,

voice, video conferencing and Internet browsing • All traffic was ToS (Type of Service) marked and QoS (Quality

of Service) had been implemented across all LAN and WAN network devices

• The WAN was a single OSPF Area 0

Legacy Wide Area Network Diagram:

Customer Brief for New Wide Area Network:

• A seamless transition from the current WAN to the provided solution

• To support rapid changes in connectivity and the demands of the business

• Minimise impact of latency: target < 20ms single hop; < 50ms any site to any site

• Easily re-configurable solution to accommodate new locations (Geographically anywhere in the UK) or the removal of existing locations

• Solution which minimises the requirement and impact of planned and unplanned maintenance

New Wide Area Network Solution:

Core Solution was based on a BT IP Clear MPLS Network

Some of the advantages to this solution where:

• Provides the performance, reliability and security of a leased-line network with the any-to-any scalability and flexibilities of an IP network

• Provides differentiated performance levels and prioritisation of delay and non-delay sensitive traffic

• As IP Clear is any-to-any IP VPN network each site needs only one-access line and via this line can access all the other sites in the network

Specifications of the MPLS Solution:

Each connection from an FSS site to the IP Clear networkconsisted of:

• Bearer Circuit – total capacity of circuit connecting site to MPLS network

• Committed Data Rate – actual amount of Bearer Circuit available for use

• Class of Service Data Rates – allocated EF and AF capacity for priority traffic

Trident Court (HQ) and London Offices:

Secure + was installed at the Head Office and at the London Officeas these sites required network traffic to be routed equally over bothlinks

(Secure+ provides a second diversely routed access circuit to a different PoP)

Remote Sites:

ADSL contended IP Clear Backup resilience links were provided as well as themain circuit

Hardware Used:

Ethernet Presented Circuits – 3750 Metro Switch was used - Mainadvantages to use this switch was because it supported Hierarchical QoSovercoming the limitations where the committed data rate on an interface isless then the interface speed

ADSL Circuits – Cisco 1801 Router was used - Main advantages that thismodel of ADSL switch could be rack mounted and has 2 Ethernet interfaces

Technical Design:

Routing (Edge Sites and London)

• BGP would be the routing protocol within the MPLS cloud • All CE Routers would be a EBGP Peer to the PE Routers• Initially I designed the new LAN to redistribute BGP into OSPF for

internal edge site subnets by use of statically configured network statements

Routing (Edge Sites) continued

• FFS however decided they wanted a dynamic approach to redistribution at the edge sites

• Final Design – Redistribute OSPF into BGP• Advantages:

• Eliminate the need for BGP Network Statements and also allow for future new subnets to be advertised automatically

• Each Edge site became a single OSPF Area 0 autonomous system

Failover (Edge Sites)

All the remote sites had two routers one being the 3750Metro Switch (Primary Link) and the second being the 1801 Router(Backup Link) both where EBGP Neighbours of each other

Ingress Traffic ◦ I used route maps to make sure all Ingress traffic from the MPLS was passed

across the high bandwidth link◦ I altered the BGP MED Attribute which set the 3750 to use a MED of 100 and

the 1801 to use a MED of 200

Failover (Edge Sites) continued

Egress Traffic ◦ To ensure all outgoing traffic was passed across the high bandwidth link I

sent a default route to the FSS internal L3 Switches using the default-information originate command within OSPF

◦ The 3750 Metro Switch used a metric of 100 and the 1801 Router used a metric of 200

◦ This approach was feasible as the edge sites only had only one exit point out to the MPLS

London Technical Design

◦ London like Trident Court (HQ) was using Secure + and these sites needed both Ingress and Egress traffic to be load shared

◦ Difficult to come up with a solution, however after much research I found the following article from Cisco: http://www.cisco.com/en/US/tech/tk365/technologies_configuration_example09186a00800945bf.shtml

◦ The above provides a link to sample configurations for Load Sharing with BGP in Single and Multihomed Environments, the one I selected was Load Sharing when Dual-Homed to one ISP Through Multiple Local Routers

London Technical Design continued

Ingress Traffic Configuration:◦ On the IBGP connections between the two 3750’s, a secondary IP Address

was assigned.◦ Primary IBGP Address is announced from the first 3750 with a shorter

AS_Path then announced from the Second 3750◦ Secondary IBGP Address was announced from the second 3750 with a

shorter AS_Path then announced from the first 3750

London Technical Design continued

Egress Traffic Configuration:◦ Load sharing of egress traffic was similar to the edge sites◦ A default route was sent to the FSS internal L3 switches using the default-

information originate command within OSPF◦ Metrics of the command left a default so both 3750’s had equal cost

Trident Technical Design

Ingress Traffic Configuration:◦ Identical to the setup of London no difference

Egress Traffic Configuration◦ This site was the Hub which had multiple paths out to other Government

Networks, sending a default route to the FSS Layer 3 Switches would of caused routing issues to the other Government Networks

◦ All Prefixes where advertised from Trident Court using BGP Network Statements and BGP was redistributed into OSPF

◦ OSPF by default load shared over the equal cost paths◦ A default route from Trident Court had to be injected into the MPLS network

so the PE Routers knew where to send unknown traffic too. Using the command under BGP neighbor xxx.xxx.xxx.xxx default originate this was achieved

Trident Technical Design continued

Staging

◦ I could not emulate a true MPLS network with PE routers as the equipment I had did not support MPLS

◦ As an alternative method I setup one of the 3750 Metro Switch’s to emulate BT’s MPLS cloud running AS 2856

◦ By using the 3750 Metro Switch it allowed me to connect up to 24 routers via Ethernet

◦ I setup Trident Court and a number of example spoke sites for the testing

Staging continued

◦ Testing the design went smoothly apart from the failover to the Backup Circuits

◦ Issue was after failover to the Backup Circuit when the Primary Circuit was brought backup traffic did not use it, instead it would keep using the Backup Circuit

◦ After a lot of research the reason this was caused because the traffic was using the preferred route of EBGP as this has an administrative distance of 20 where as OSPF has an administrative distance of 110

◦ The solution was to increase the Administrative Distance of EBGP to 120 from 20 using the command distance 120 200 200 which fixed the issue

Implementation (Phase 1)

◦ Phase 1 was to install the 3750 Metro Switches at Trident Court to check if the Secure + Circuits where operational, running concurrently with the ATM Routers

◦ Advertise all prefixes using BGP Network statements into the MPLS Network from Trident Court – this was needed so when we cut over to the Pilot Site it would be able to see all advertised networks and full connectivity testing could be carried out

◦ Install all the spoke sites Switch’s/Routers and leave them in place for roughly 2-3 weeks so the MPLS circuits could be monitored to see if they were functioning correctly

◦ By testing the MPLS circuits in this manner, it allowed me to identify that nearly 40% of the circuits had issues and most required re-visits from BT Commissioning Engineers to correct the faults

Implementation (Phase 2) Pilot Site Cut Over

◦ Phase 2 was to cut over a Pilot site, Warrington was identified as a good candidate as it only had 5 users

◦ Key steps where: Disconnect LAN interfaces on legacy WAN routers. Ensure that the edge site routes

disappear from the Trident Court routing tables Connect MPLS router LAN interfaces Verify that MPLS routers have learned internal subnets via OSPF Verify that MPLS routers have learned all BGP Advertised routes from Trident Court Test Failover Carry out full connectivity testing with the Warrington Users

◦ The Pilot site Cut over went well with no major issues

Implementation (Phase 3) Live Cut Over

◦ All spoke sites where cut over with no major issues, using a checklist I had pre-prepared from the pilot cut over in Warrington I carried out the same checks at every site, if all checks were successful then the site would be signed off by FSS

◦ Only issue I had was a the London site, as this site was using Secure + I found it not to be advertising all of its internal networks via BGP.

◦ This was caused by the route map access list not containing all the internal subnets. Once they where added to the access list, it corrected the fault

◦ Once all spoke sites where cut over to the MPLS WAN, I then just disconnected the legacy ATM routers at Trident Court, FSS was then natively on the MPLS WAN for all sites

Support

◦ A dedicated Frame Relay Router was installed so we could Manage/Monitor the FSS WAN remotely from the Alfred McApline NOC which was located in Glasgow, using SNMPc and Cisco Works

◦ TACACS Authentication was setup on all the FSS Devices so only authorised individuals could get access to the equipment

◦ Being Lead Engineer for the Project it was my duty to write up Managed Network Reports which included utilisation graphs, service issues and monthly events that needed to be recorded

MPLS Wide Area Network Diagram: