57
JOINT WARRIOR INTEROPERABILITY DEMONSTRATION 2004 (JWID 2004) NETWORK OPERATIONS WORKING GROUP (NOWG) CFBLNet FINAL REPORT NETWORK OPERATIONS WORKING GROUP July 30, 2004

COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JOINT WARRIOR INTEROPERABILITY DEMONSTRATION 2004

(JWID 2004)

NETWORK OPERATIONS WORKING GROUP(NOWG)

CFBLNetFINAL REPORT

NETWORK OPERATIONS WORKING GROUPJuly 30, 2004

Page 2: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Table of Contents

1. Purpose...............................................................................................................................4

2. Scope..................................................................................................................................4

3. Summary............................................................................................................................4

4. JWID 2004 CFBLNet........................................................................................................64.1 Network Engineering..............................................................................................7

4.1.1 Telecommunications Circuits.....................................................................74.1.2 IP Routing Protocols ..................................................................................8

4.1.2.1 IP Routing Prioritization.................................................................94.1.2.2 IP Routing Summarization..............................................................104.1.2.3 Use of Internal BGP Routing Reflectors........................................104.1.2.4 Control Over Incoming/Outgoing EBGP Routes...........................10

4.1.3 Ethernet LAN Improved at the AITS-JPO .................................................104.1.4 REL ROK Network.....................................................................................114.1.5 Quality of Service.......................................................................................124.1.6 Universal “Read-Only” password...............................................................124.1.7 Network Time Protocol...............................................................................12

4.2 Provisioning............................................................................................................134.2.1 Lessons Learned (Provisioning).................................................................16

4.3 Network Accreditation............................................................................................164.3.1 Lessons Learned (Network Accreditation) ................................................17

4.4 COMSEC................................................................................................................174.5 Network Testing .....................................................................................................18

4.5.1 Lessons Learned (Network Testing) .......................................................19

5. Network Services...............................................................................................................195.1 E-Mail.....................................................................................................................19

5.1.1 Improvements.............................................................................................215.1.2 Lessons Learned ........................................................................................22

5.2 DNS.........................................................................................................................225.2.1 Improvements.............................................................................................235.2.2 Lessons Learned..........................................................................................23

5.3 IP Phones................................................................................................................235.3.1 Lessons Learned..........................................................................................24

5.4 Web.........................................................................................................................245.4.1 Overall Web Structure and Format.............................................................255.4.2 Newsgroups.................................................................................................26

2

Page 3: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

5.4.3 Bubble Chart...............................................................................................265.4.4 Lessons Learned Database..........................................................................28

5.3.3.1 Database Lessons learned...............................................................295.4.5 Master Scenario Events List (MSEL) & MSEL Web Based Tool (MWBT)

Servers........................................................................................................305.5 Defense Collaboration Tool Suite (DCTS).............................................................31

5.5.1 DCTS on the CFBLNet...............................................................................315.5.2 Lessons Learned..........................................................................................33

5.6 Network Engineering Management........................................................................345.6.1 Network Monitoring Tools.........................................................................34

5.6.1.1 Netscout..........................................................................................355.6.1.2 HP Openview Network Node Manager (NNM).............................365.6.1.3 Open NMS......................................................................................365.6.1.4 Network Traffic Analysis System (NTAS).....................................385.6.1.5 Flow Tools/Flowscan......................................................................39

5.6.2 Lessons Learned..........................................................................................415.6.2.1 Future Considerations.....................................................................42

5.7 Network Statistics Collection.................................................................................42

Network Engineering Figures

Figure 1. Backbone Bandwidth Utilization JWID Day 9 (24 June 2004).............................5Figure 2. JWID 2004 Site & Network Diagram....................................................................6Figure 3. CFBLNet Backbone Topology...............................................................................7Figure 4. JWID 2004 Routing Plan (6 Eyes Domain)...........................................................9Figure 5. Topology/connectivity of the ROK domain...........................................................11Figure 6. NTP topology.........................................................................................................13Figure 7. Correctly Functioning Pipe.....................................................................................18Figure 8. Faulty Circuit dropping packets.............................................................................19

Network Services Figures and Tables:

Table 1.Classification Markings for Mail Messages.............................................................21Table 2. Attachment Types for Messages..............................................................................26 Figure 9. JWID 2004 Email Implementation.........................................................................21Figure 10. Example Network Bubble Chart..........................................................................27Figure 11. Example CIT Bubble Chart..................................................................................28Figure 12. Lessons Learned Database...................................................................................29Figure 13. CFBL MCU Topology.........................................................................................32Figure 14. NetScout...............................................................................................................35Figure 15. Open NMS............................................................................................................37Figure 16. NTAS....................................................................................................................38Figure 17. Flowscan...............................................................................................................40

3

Page 4: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

1. PURPOSE

This report provides a summary of the Joint Warrior Interoperability Demonstration (JWID) 2004 CFBLNet engineering, operations, and network services to the extent established and supported by the AITS-JPO.

2. SCOPE

The data used to produce this report was based on observations and the operation of the CFBLNet and Network Services implemented to support JWID 2004.

3. SUMMARY

The CFBLNet was successfully implemented to support JWID 2004. Designed as a Secret-Releasable overlay on the Defense Information Systems Network-Leading Edge Services (DISN-LES) and DISA ATM Services - Unclassified (DATMS-U) ATM backbone networks, the CFBLNet incorporated a sophisticated ATM backbone, capable of supporting high-speed data transmission requirements of up to 45Mbps. In the US, the CFBLNet shared between 3 and 6Mbps of bandwidth on the DISN-LES with the US National and the US Military Assistance to Civilian Authority (MACA) domains. Details of the US National and US MACA domains will be discussed in separate documentation and are only mentioned where necessary for clarification.

JWID 2004 officially began 2 June 2004 and ended 25 June 2004. Scenario execution started 14 June and continued until 25 June; the JWID 2004 Day totaled six hours from 1600Z to 2200Z.

Throughout the exercise, the overall CFBLNet availability was 99.9998%. Availability rates are based on IP connectivity during the JWID Day to each site. Although sporadic network outages occurred during the execution period, only network outages that affected operations during the JWID Day were used for site availability reporting.

In general, network traffic for JWID 2004 was nearly double that of the traffic generated during JWID in previous years. The addition of the US National and US MACA domains significantly increased bandwidth utilization across the DISN-LES. During JWID 2004, traffic being transmitted from the JPO to other sites on all three networks peaked between 5-6 Mbps throughout the JWID Day. This increase was due in a large part to greater use of the Defense Collaborative Tool Suite (DCTS) and the larger number of trials participating within the Coalition and the US domains. DCTS was designated a core service for JWID 2004. Figure 1 shows the backbone (US) bandwidth utilization between the AITS-JPO and Dahlgren during JWID Day 9 (23 June 2004).

4

DISA, 01/03/-1,
Apparently there is a Presidential directive stating that we cannot have an “exercise” with NZ. Therefore all references to “exercise” must be removed.
DISA, 01/03/-1,
No need to use “increase” three times.
DISA, 01/03/-1,
Redundant.
DISA, 01/03/-1,
“Increase” is used twice in the same sentence at this point.
DISA, 01/03/-1,
Using first-person perspective is not necessary for this report. Who does “we” refer to anyway?
Page 5: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

At the beginning of scenario execution, a few engineering and operational issues impacted network performance. When discussing network performance, network problems are defined as any network event that affected the ability of a Coalition Interoperability Trial (CIT) to successfully complete its presentation as planned.

Additional problems identified during JWID 2004 will be addressed within each reported area.

Figure 1 Backbone Bandwidth Utilization JWID Day 9 (23 June 2004)

4. JWID CFBLNET

5

DISA, 01/03/-1,
No first-person.
Page 6: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

The CFBLNet for JWID 2004 consisted of 35 sites, connected with a sophisticated network of ATM transmission paths capable of providing between 3-10 Mbps of bandwidth for data requirements.

Figure 2JWID 2004 Site & Network Diagram

While the majority of sites were ATM-connected, some sites were connected via IP Ethernet or serial data links. All sites on the CFBLNet backbone were secured to the Secret-Releasable level with the use of KG-75 FASTLANEs for ATM-connected links, KG-175 TACLANEs for Ethernet-based links, and KG-84 or KIV-7 encryption devices on serial data links.

The CFBLNet for JWID 2004 facilitated additional NATO site participation through Allied backside connections. Many of these sites were connected to and participated on the CFBLNet with extensive local infrastructures. However this document will only discuss the directly connected backbone sites. See Figure 2 for the CFBLNet topology in support of JWID 2004.

6

DISA, 01/03/-1,
Above we say we are using DISN-LES and DATMS-U paths, and here we mention PATM. Using “ATM transmission paths” is more general and encompasses the variety of ATM transmission media used.
Page 7: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 3 CFBLNet Backbone Topology

4.1.0 Network Engineering

4.1.1 Telecommunications Circuits

The JWID 2004 network utilized circuits on the DISN-LES and DATMS-U Networks, along with leased ISDN, DS-1, fractional DS-1, E-1, DS-3, and E-3 communications circuits. In the United States, these communications facilities supported US Unclassified, US Secret, and US Secret Releasable to Allies (AUCANZUKUS + NATO) services. The classified security levels were separated from each other and from the unclassified service by Type-1 encryption devices. Outside the United States, these lines supported the US Secret Releasable to Allies (AUCANZUKUS + NATO) service only. Encryption devices used were KG-75 FASTLANEs, KG-175 TACLANEs, KIV-7s, and KG-84s.

The use of separate unclassified ATM switches at select US sites using their own Permanent Virtual Paths (PVPs) allowed the separation of JWID traffic from other user site program traffic using the same physical layer access circuit. This allowed better tracking of JWID user traffic and bandwidth requirements since JWID traffic was on JWID PVPs and not mixed in with other user site program traffic. It also enhanced network fault isolation.

7

Page 8: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

To carry IP traffic over the ATM portion of the JWID 2004 network, static IP to Network Service Access Point (NSAP) tables were entered into all ATM-enabled routers on the JWID 2004 network. ISDN, DS-1, E-1 and fractional DS-1 circuits were directly IP-connected to serial router interfaces at other JWID sites. Those sites were then connected to the DISN-LES ATM network to support JWID 2004.

4.1.2 IP Routing Protocols

Two routing protocols were used during JWID 2004. Open Shortest Path First (OSPF) Version 2 was used within a particular Autonomous System (AS), and Border Gateway Protocol (BGP) Version 4 was used between ASs. From a US perspective, OSPF connected US Sites and BGP connected those sites to all of the allied participants (see Figure 3). These two IP routing protocols were kept completely separate, or not “redistributed”, during JWID 2004. By keeping these routing protocols separate, not only did each AS have more control of what IP routes were propagated and accepted, but troubleshooting was made much easier because the origin of an IP route could be very quickly associated with an AS.

Figure 4JWID 2004 Routing Plan (6 Eyes Domain)

8

Page 9: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

4.1.2.1 IP Route Prioritization

The JWID 2004 network had few redundant circuits. The IP routing on the redundant circuits that did exist was designed by the network engineers to continue to provide connectivity in the event of a failure on the primary communications circuit. Unless the highest bandwidth links were deliberately prioritized, it was left up to the IP routing protocol to determine which link was used. JWID 2004 network engineers agreed to use EBGP (External BGP) Local Preferencing (LP) to prioritize IP links among sites (see Figure 4). EBGP LP assigned a preference to all EBGP-learned routes. The default LP assigned was 100. The higher the LP, the more desirable the route was to the remote AS. All the routers within the remote AS then learned the LP assigned to the route. Therefore, if an AS has more than one route to a remote AS (circuit redundancy), LPs were compared, and the route with the highest LP was preferred while the other route served as a dynamic backup route in the event of a failure.

4.1.2.2 IP Route Summarization

Instead of the US advertising several, separate IP subnets to its Allies, a single subnet encompassing the entire US IP address space, called a “supernet,” was advertised to the Allies. Likewise, the Allies consolidated their IP address space into a small number of supernets. This was advantageous for several reasons. When a small number of IP routes were advertised, EBGP updates required less bandwidth. Moreover, one supernetted IP address could often represent several, if not hundreds, of subnets. This made it much easier for a network engineer to troubleshoot IP routing problems because fewer IP routes needed to be verified as being present in the router’s IP table.

4.1.2.3 Use of Internal BGP Route Reflectors

Once a router learned an EBGP Route, the JWID network was designed so that each AS propagated the learned EBGP route to other routers in its AS via Internal BGP (IBGP). However, to avoid propagating invalid routes, IBGP would only propagate an EBGP route to its “neighbors”. This meant that all routers within an AS would have to be BGP neighbors with every other router within that AS. This was undesirable, because within the US there were six site routers on the network. Therefore the US implemented an alternative solution using IBGP dual route reflectors. Once a route reflector learned a route, it distributed or “reflected” this information to all route reflector “clients”. During JWID 2004, network engineers configured two route reflectors on the routers located at the AITS-JPO and the Space and Naval Warfare (SPAWAR) Systems Center – San Diego (SSC-SD). In the event one route reflector failed, the second route reflector took over updating the route reflector clients. This added an extra layer of redundancy to the IP routing design.

9

Page 10: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

4.1.2.4 Control Over Incoming/Outgoing EBGP Routes

EBGP allowed the network engineer to easily filter incoming or outgoing EBGP routes. This gave complete control to the network engineer to what IP routes were distributed to neighbors and what IP routes were learned from neighbors.

4.1.3 Ethernet LAN Improved at the AITS-JPO

The Top-Provider CUseeMe server and several core network servers were located at the AITS-JPO for JWID 2004. It was surmised that a lot of data traffic would be transmitted and received at the AITS-JPO during this and future JWIDs. Therefore, the LAN on which these servers were connected needed to be optimized for maximum bandwidth efficiency. The engineering staff improved upon the 10/100Mbps Switched Ethernet LAN implemented at AITS-JPO for JWID 2002, and upgraded all segments of the LAN to 100Mbps Full Duplex.

4.1.4 REL ROK Network

For JWID an additional network (8-eyes) was established to facilitate the participation of the Republic of Korea in Seoul, KR. The only two sites that had visibility to this 8-eyes domain were Seoul, KR, and the AITS-JPO. DNS and email were hosted by the AITS-JPO with email traffic passing through a DII Guard for classification security. GCCS COP traffic passed through a Radiant Mercury Guard. Traffic flow from the 8-eyes domain to the JWID domain was also possible via the DII and Radiant Mercury guards. Both the DII and Radiant Mercury Guards were located at the AITS-JPO. The ROK domain was cryptographically separated from the rest of CFBL using Type-1 TACLANE encryptors and no additional services were available.

10

KOREAROK LAN

JPOROK LAN

DISN-LES

KG 75KG 75

ASX 200ASX 200

TAC Lane TAC Lane

Cisco 3550

JPO-CFBL ROK-CFBL

35503550

IEC InnerQuad C

Page 11: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 5Topology/connectivity of the ROK domain

4.1.5 Quality of Service

During JWID 2004, the possibility existed that the low bandwidth link to New Zealand would become saturated with data. To allow IP phone traffic to pass unhindered, Priority Queuing (PQ) was implemented. PQ is a Quality of Service method, in which the three precedence bits located in the IP header of the data packet are colored by the call manager to indicate the packet is critical. The serial interface on the AITS-JPO router for the New Zealand link was configured to recognize this coloring and process these packets as priority packets.

4.1.6 Universal “Read-Only” Password

In order to perform easy collaboration among network engineers, the same password policy implemented in previous years’ demonstrations was reused during JWID 2004. All JWID network engineers used a universal “read-only” password to access JWID border routers, LAN Ethernet switches, and ATM switches. This read-only password allowed network engineers to collaborate on connectivity, routing, switching, and ATM problems that occurred during the lead up to and execution of JWID 2004. AS engineers configured a unique “enable” password for the routers, Ethernet switches, and ATM switches in their AS. The CCCC-R was the central repository for all passwords. Enable passwords were transmitted to the CCCC-R via IP phones.

All JWID 2004 network engineers agreed that the enable passwords would be used only by the CCCC-R when a configuration change would alleviate a network emergency, and only after every effort was made to contact the applicable network engineer. Such an emergency never arose during JWID 2004.

4.1.7 Network Time Protocol

In order to synchronize all network devices and provide a reliable time source for servers and applications, Network Time Protocol (NTP) was used. The NATO-NC3A was configured to recognize a stratum-1 time source located at NATO-NC3A as an authoritative time source. This made the NATO-NC3A router a stratum-2 time source. The UK, Australia, Canada, and the AITS-JPO routers peered with the stratum-2 NATO-NC3A router. Each AS engineer was responsible for propagating time down to their respective LANs from one of these routers.

11

Page 12: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

In the event that the stratum-1 time source was inoperable, the UK, Australia, Canada, NATO-NC3A, and AITS-JPO routers were configured as stratum-5 authoritative time sources. In this case, each router would recognize their respective internal timing crystal as an authoritative time source and synchronize with each other using the defined NTP to minimize clock drift.

Figure 6NTP topology

4.2 Provisioning

Provisioning is the all-encompassing term that incorporates all actions required to support a user's network requirements. Identification of user requirements, engineering design, design review and approval, equipment bills of materials, funding, equipment and circuit acquisition, equipment configuration and testing, installation, and system testing are all part of the provisioning process. The key to success lies in the early and thorough identification of user requirements.

12

RED1_NC3A_7204Loopback0

A.B.5.1

NC3A AS 1109

Portsdown WestLoopback0

E.F.245.1/32

NEW ZEALAND

AUSTRALIALoopback0C.D.217.245

JPO_7204Loopback0I.J.52.97/32

US AS 3664 UK AS 1200-1201

N.Z. AS 1003

AS. AS 1000

CANADALoopback0G.H.125.1

CANADA AS 1001

Time SourceServer Associations

Time SourcePeer Associations

NATOAuthoritative time

NATOGBS receiver

ntp source loopback0ntp master 5

ntp update-calendarntp peer A.B.5.1 (NATO border router)

ntp peer C.D.217.245 (Australian router)ntp peer E.F.245.1 (UK border router)

ntp peer G.H.125.1 (Canadian border router)ntp peer I.J.43.97 (AITS-JPO router)

ntp source loopback0ntp master 5

ntp update-calendarntp server A.B.4.5 (NATO stratum 1)

ntp peer C.D.217.178 (Australian router)ntp peer E.F.245.1 (UK border router)

ntp peer G.H.125.1 (Canadian border router)ntp peer I .J.43.97 (AITS-JPO router )

ntp source loopback0ntp update-calendar

ntp server C.D.217.245 ntp server A.B.5.1

For NZ:

For AUS/UK/AITS-JPO/CAN:

For NATO:

Network Time Protocol (NTP)

Page 13: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

The provisioning of services for JWID 2004 was based on maximum use of existing DISN-LES, DATMS-U, and Allied connectivity.

The following US sites were provided Unclassified, US Secret, and US Secret Releasable Allies (AUCANZUKUS + NATO) Coalition connectivity over the DISN-LES network:

SSC-SD, San Diego, CA Unclass & CoalitionNORTHCOM, Peterson AFB, CO Unclass, Secret, & CoalitionESC, Hanscom AFB, MA Unclass, Secret, & CoalitionMARFOR/NSWC-DD, NSWC Dahlgren, VA Unclass, Secret, & CoalitionNGA, Bethesda, MD Secret, & CoalitionDTRA, Alexandria, VA CoalitionJITC, NSWC Indian Head, MD CoalitionDISA EAGLE, Falls Church, VA Unclass, Secret, & CoalitionAITS-JPO, Arlington, VA Unclass, Secret, & Coalition

The following US sites were provided coalition connectivity over the DATMS-U Network:

NCTAMSPAC, Wahiawa NCS, Wahiawa, HI UnclassUSFK/ROK, Yongsan Main Post, Seoul, S. Korea Coalition

The following allied sites were provided coalition connectivity using a combination of the DISN-LES, DATMS-U, dedicated Point-to-Point (Pt-to-Pt) US leased circuits, allied Public ATM (PATM) networks, and dedicated Pt-to-Pt allied leased circuits:

NATO NC3A, The Hague, NetherlandsNATO Contingent NORWAY, Lillehammer, NorwayNATO National NORWAY, Lillehammer, NorwayNATO National NORWAY, Kolsass, NorwayNATO National FRANCE, Taverny, Celar, FranceNATO National Germany, Euskirchen, GermanyNATO National ITALY, Rome, ItalyNATO National SPAIN, Madrid, SpainNATO AC-T, Norfolk, Virginia

UNITED KINGDOM, DSTL, Portsdown West, UKUNITED KINGDOM, DGIA, Feltham, UKUNITED KINGDOM, JARIC, RAF Brampton, UK

CANADA, CTDC, Tunneys Pasture, Ottawa, CanadaCANADA, CFEC, Shirleys Bay, Ottawa, Canada

13

Page 14: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

CANADA, J2IM, Ottawa, CanadaCANADA, SBML, Shirleys Bay, Ottawa, CanadaCANADA, DRDC, Valcartier, Quebec, CanadaCANADA, 1CAD, Winnepeg, Canada

AUSTRALIA, DSTO Fern Hill, Canberra, AustraliaAUSTRALIA, DNC4ISREW, Campbell Park Offices, Canberra, AustraliaAUSTRALIA, ADF Warfare Center, Newcastle, AustraliaAUSTRALIA, DSTO Edinburgh, Adelaide, Australia

NEW ZEALAND, JISA Porirua, Wellington, New ZealandNEW ZEALAND, HQJFNZ, Trentham, New ZealandNEW ZEALAND, TAC NOC, Auckland, New Zealand

UK leased E-3 (34Mbps) access circuits to the British Telecom Public ATM network from RAF Molesworth, UK, and Portsdown West, UK. A 2Mbps virtual path (VP) was established through those circuits to provide CFBLNet access from Portsdown West, UK to RAF Molesworth, UK. This provided connectivity to the coalition Point Of Presence (POP) at RAF Molesworth for the UK. From Portsdown West, the UK extended service to RAF Brampton and Feltham, UK.

NATO leased a dedicated E-3 (34Mbps) circuit to CFBLNet connectivity from NC3A, The Hague, Netherlandsto the coalition POP at RAF Molesworth, UK. An E-1 (2.048Mbps) ISDN dial-up connection to the Public Switched Telephone Network (PSTN) remained in place at both RAF Molesworth, UK, and at NC3A, The Hague, NL, to support NATO connectivity should the E-3 line fail. The E-1 ISDN was not used during the JWID 2004 event. NATO operated a T-1 connection from ACT, Norfolk, VA, to the AITS-JPO in Arlington, VA. From NC3A, The Hague, NL, NATO provided connectivity to France, Germany, Italy, Norway, and Spain. At Lillehammer, Norway, both a NATO contingent and a Norway national contingent operated within the facility.

Australia leased a dedicated DS-3 (45Mbps) line connection between HMAS Harmon, Canberra, Australia and NCTAMSPAC, Wahiawa NCS, Oahu, HI. The DS-3 supported an ATM cell-bearing virtual path (VP) between Australian-owned NORTEL PassPort ATM switches at HMAS Harmon, Canberra, Australia and NCTAMSPAC, Wahiawa NCS, Oahu, HI. The Australian PassPort ATM switch connection was extended from HMAS Harmon, Canberra, Australia, to a PassPort switch at DSTO Fern Hill, Canberra, Australia. The DSTO Fern Hill PassPort switch was then connected to a collocated Marconi (FORE) ATM switch. The Wahiawa PassPort switch was connected to the collocated DISN-LES CFBLNet ATM switch. A 10Mbps PVP was established from DSTO Fern Hill to DISN-LES CFBLNet Wahiawa over this path. The use of a DISN-LES node at Wahiawa allowed traffic from Australia to choose whether it needed to go to

14

Page 15: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

USFK, or directly to the CONUS, thus preventing potential node isolation and path loading. From DSTO Fern Hill, Australia provided connectivity to other sites in Australia as well as the primary connection to New Zealand. The connection to New Zealand was an IP router over an ATM-based connection.

Canada leased an OC-3c (155Mbps) access circuit at DRTC, Tunney's Pasture and a DS-3 (45Mbps) access circuit at AITS-JPO, Arlington, VA, to the Bell Nexxia Public ATM (PATM) network. A 15Mbps PVP was ordered by Canada via these access circuits and the PATM to support Canadian access to the CFBLNet from Tunney's Pasture, Canada, to AITS-JPO, Arlington, VA. This link used a Canadian TACLANE with JWID key material at both ends of the link to encrypt network traffic.

New Zealand established a new 512kbps dedicated connection from New Zealand to Australia for CFBLNet. The existing 512Kbps ISDN dial-up circuit from New Zealand to AITS-JPO, Arlington, VA, was left in place as a back up to the dedicated service. The ISDN dial-up was not used during JWID 2004.

4.2.1 Lessons Learned (Provisioning)

It is essential to remove any option to add new locations after the Mid Planning Conference. The changes to locations and late decision to add two sites added to the complexity of standing up and testing JWID 2004 locations prior to the JWID 2004 execution.

The use of separate Unclassified ATM switches at US user locations for JWID 2004/CFBL and other programs allowed the clean separation of user traffic for JWID from other user site programs. This was accomplished by standing up separate PVPs that terminated only on the JWID Unclassified ATM switches, allowing simultaneous program support for both JWID and other DISN-LES user programs. It also allowed better isolation of network faults by enabling personnel to focus only on the JWID PVPs.

4.3 Network Accreditation

In accordance with the DoD Information Technology Security Certification and Accreditation Process (DITSCAP) and the guidelines of the Annex C, network accreditation is two-part process. First, a site is accredited for the infrastructure, which allows for connectivity to the network and testing. Second, all initiatives that take place on the CFBLNet require an additional accreditation package from the US sites for that initiative. JWID 2004 is an example of just such an initiative. Timely site accreditation became an issue during JWID 2004 for the US sites, especially for the new sites participating in the demonstration. Another issue regarding the US sites was a lack of separation of the accreditation packages. The US sites had combined the infrastructure and the initiative accreditations into a single package. As a result, several US sites initially presented only the infrastructure accreditation package for connectivity to the network.

15

Page 16: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

4.3.1 Lessons Learned (Network Accreditation)

All US sites connecting to the DISN-LES must follow the rules and guidelines in the accreditation connection requirements and the DITSCAP. US sites connecting to the CFBLNet must follow the guidelines of Annex C of the CFBLNet charter. A number of the sites were slow to submit their completed accreditation packages, resulting in a degraded ability to meet the established timelines for network testing. Emphasis needs to be placed on utilizing the standard procedures for future initiatives and demonstrations.

Continuous tracking of each participating site’s accreditation status is essential to ensuring accreditation is completed on schedule. This requires intense management and timely intervention when a site begins to fall behind schedule. A finalized list of participating sites and trials with the network connection requirements for each site must be produced as early as possible in the planning process. This will allow the sites to present their initiative accreditation package in a timely manner.

The security accreditation briefing for CWID 2005 will be conducted differently next year. At the end of the briefing a representative from each site will be asked to sign an attendance sheet and will be given an accreditation CD, containing information that can be used as a guide to prepare the accreditation packages. JPO contact information, to include name, number and email address, will also be available for further assistance.

4.4 Communications Security (COMSEC)

All COMSEC equipment and keying material (keymat) was sent in a timely manner. The National Security Agency (NSA) has granted the AITS-JPO an alternative means of deploying keymat to Allied CFBLNet members. <what's the alternative means?> Keying devices, such as the Data Transfer Device (DTD), were used as the main storage device for all electronic keymat. With the current availability of the STU-III or STE secure telephones to the Allies, the electronic transfer of keymat to the National Distribution Accounts was performed in a matter of minutes versus days or weeks via the physical shipment method. The receipt of the COMSEC implementation messages sent via DMS continues to be an issue with Australia. At this point, the Australian AUTODIN interface seems to be the problem. This issue will continue to researched and worked during the course of this year as we continue to provide Australia with COMSEC support. Network encryption devices were loaned to NATO and New Zealand for successful connectivity to the network.

4.5 Network Testing

Bandwidth utilization testing began on May 12, 2004 and was completed May 25, 2004. A detailed test plan was created and distributed to all site engineers via email. The purpose of the

16

Page 17: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

bandwidth utilization tests was to ensure that a clean connection could be maintained to each site to the maximum of the site’s purchased bandwidth.

To perform testing, each site needed a machine able to run the MGEN 4.x program available from the Naval Research Laboratory’s website (http://mgen.pf.itd.nrl.navy.mil/).

The idea behind using MGEN for network testing was to be able to accurately detect network problems in an end-to-end manner to and from each site and the AITS-JPO.

MGEN generates UDP traffic at any size and rate that is given to the program. Given this ability, a Perl script was created to start at the lowest possible packet size and steadily increase the bandwidth used until attempting to push 5% above the known pipe size. The expected results of this test were, when the traffic received at each site was graphed, a stair-step pattern would result with the top level of the graph appearing as a flat line since more traffic was being generated than could travel through the pipe. The figure below is an example of a correctly functioning pipe.

Figure 7Correctly Functioning Pipe

During the testing, two circuits were discovered that dropped packets once maximum throughput was reached. The faulty circuits were isolated and located between Australia and the AITS-JPO and between SSC-SD and the JPO. In both instances, these packet drops were isolated to ATM rate-shaping memory buffer overflows. These problems were rectified before JWID 04 execution in both instances by reallocating memory buffer space. The figure below is an example of one site that was dropping packets as described.

17

Page 18: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 8Faulty circuit dropping packets

Several sites provided test hardware/software that was either not capable of running the MGEN 4.X program or the respective site missed their test date and provided the test hardware platform late. In these cases, we noted which links may be affected and did our best to ensure that an adequate (over 3Mbps) amount of data could be successfully sustained through the link. This was accomplished by using the much less reliable “ping” test to and from the remote site.

4.5.1 Lessons Learned (Network Testing)

The success of JWID 2004 illustrated the many benefits of competent and diligent network planning, implementation, and execution. A spirit of collaboration and understanding existed among all JWID 2004 network engineers; nonetheless, there is room for improvement in the following area:

Importance of test schedule: The test schedule must be discussed and agreed to during the three planning conferences. Site engineers should make every attempt to adhere to the test schedule or reschedule as soon as possible if something disrupts planned tests. The importance of thoroughly testing the network and services must be stressed at each of the planning conferences.

5. NETWORK SERVICES

A coalition staff representing Australia, Canada, New Zealand, the UK, the US and NATO developed, planned, engineered, implemented and maintained the network services for JWID 2004. The tasks required skills in UNIX and Windows platforms, network management tools, collaboration tools, web, Domain Name Service (DNS), and electronic mail (email).

18

Page 19: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

The network services requirements for JWID this year were extremely complex, given the establishment of multiple coalition domains. In addition to the AUSCANNZUKUS + NATO domain, three additional domains were implemented: the ROK domain, the NATO-Only domain and the Partners for Peace (PfP) domain. Each of these domains required a set of core services and restricted connectivity between domains. The NATO-Only and PfP domains were implemented on the national side of the NATO CFBLNet point of presense, and NATO held the responsibility for standing up core services in these domains and, therefore, will not be discussed in this document. The ROK domain was executed on the national side of the US CFBLNet domain with no other coalition participation; therefore, the US was responsible for implementing core services in this domain. For the ROK domain, a DII Mail guard and a Radiant Mercury guard were included in the infrastructure to allow filtered email with attachments and the Common Operating Picture (COP) to flow between the AUSCANNZUKUS + NATO and ROK domains.

5.1 Email

For JWID 2004, each nation was responsible for the implementation of an email server within their national domains as shown in Figure 9. This alleviated the requirement of managing all coalition email accounts from the CCCC-Rear.

JWID email accounts are role-based accounts. The list of role-based accounts was retrieved directly from the MSEL Web-Based Tool (MWBT) and fed into the server located at CCCC-Rear. Email accounts were also created for trial support personnel and site support staff on a per-request basis. The CCCC-R sent site engineers a list of email accounts and passwords to be distributed to the role players and support personnel. Dahlgren successfully hosted its own email accounts during JWID 2004.

19

Page 20: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 9. JWID 2004 Email Implementation

The email service used Simple Mail Transport Protocol (SMTP) to ensure interoperability between nations and organizations, but the specific implementation was at the discretion of the nation or organization. Microsoft Exchange, Qmail, and Postfix existed and successfully interoperated on the CFBLNet. IMAP was used to download messages from the email server to the clients.

The US provided a DII guard that allowed CFBLNet approved labeled email messages with attachments to flow between the CFBL and Republic of Korea (RoK) domains. Since there was only one email guard for the RoK domain, all messages destined for *.kr were routed to the US email server, and the US email server, in turn, relayed these messages to the guard. This allowed for tighter control over potentially sensitive email messages and made the guard implementation simpler. The US implemented the DII Mail Guard, version 3.0.2, in conjunction with Mime Sweeper, version 4.3, for SMTP. The JWID multinational community developed and agreed to the guard configuration and rule set during the JWID planning conferences. The guard was configured to check the classification and releasability marking in the first line of each email message, valid attachment types and a list of “dirty” words that was developed to minimize the possibility of inadvertently sending inappropriate information from CFBL to RoK. The coalition approved releasability markings are shown in Table 1. Those messages marked with the label AUSCANNZUKUS + NATO REL ROK were allowed to flow from the CFBL domain to the RoK domain. The guard rejected messages marked with any other label as well as unmarked messages.

20

CAPortsdown West, UK

NZ

AS

UK

NO

JWIDCFBL

NATO

Williamtown, AU

Shirleys Bay, CA

Porirua, NZ

Hague, NLArlington, VA

Kolsas, NO

US

NATOLillehammer, NO

SPMadrid, SP

GEEuskrichen, GE

FRBruz, FR

ITRome, IT

CFBL

NATO Only

PfP

ROK

Page 21: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

AU AU

CA CA

NZ NZ

UK UK

US US

NO NO

FR FR

IT IT

DE SECRET DE

ES CONFIDENTIAL ES

TU RESTRICTED TU

DK UNCLASSIFIED DK

HU HU

NL NL

NATO NATO

ROK ROK

RO RO

SE SE

FI FI

AUSCANZUKUS

Table 1: Valid classification markings for mail messages sent from the CFBL to Other JWID CFBL Domains

The attachment types listed in Table 2 were allowed to flow between domains after being checked for viruses and other malicious code by the guard.

Valid Attachment Types (6-to-RoK and RoK-to-6)Microsoft Office

Word - .doc, .rtf, .txt PowerPoint - .ppt, .pps Excel - .xls, .csv winmail.dat file generated by Outlook - .dat

Graphic files - .bmp, .gif, .tiff, .jpg, .jpeg, .tifAdobe Acrobat - .pdf

Table 2: Valid attachment types for messages sent between CFBL and ROK domains

21

REL

Page 22: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

The attachment types listed in Table 2 were allowed to flow between domains after being checked for viruses and other malicious code by the guard.

5.1.1 Email Improvements

- The use of IMAP this year prevented users from accidentally downloading all of their messages from the server. This was especially important in the case of shared accounts on multiple machines.

- Forcing all email for the *.kr domain to go through the US email server reduced the complexity of the guard implementation and allowed a tighter control path.

- Using an email server that stores all of its messages in plain text format made it easier to backup and restore and to verify message delivery.

- Using a highly verbose SMTP and IMAP server allowed for refined troubleshooting of client communication problems and increased auditing capabilities.

- Administrative accounts for the various top-level services were created to end the mass of e-mails that routinely filter through the [email protected] account. Accounts such as postmaster@ and hostmaster@ were used for email and DNS issues respectively and were a much more effective form of communication.

5.1.2 Lessons Learned

Recommendations

22

Page 23: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

- Modifying the IMAP server to be case-insensitive should be done for CWID 2005. Even though case-sensitivity was announced, some users still had trouble formatting the long names properly.

- Adding a web-based email client would be a nice feature to add to the email service for sites that do not wish to install an email client for whatever reason.

- The email account names need to be static after MSEL lockdown is announced. Any changes made after that time need to be communicated up the chain to the email administrator.

- A more fail-safe solution for email service needs to be applied. This comes at a cost for additional redundancy and infrastructure but would better guarantee reliability. On the UNIX-like systems, rsync or rdist over SSH may be a viable alternative when combined with an appropriate setting of round robin DNS entries.

- An LDAP/S- or SASL/LDAP/S-based authentication mechanism would greatly improve service authentication and integration. The authentication could also be passed through LDAP/S to Kerberos.

5.2 DNS

DNS was provided in a hierarchical structure with the US and the UK providing the top-level root name servers (A and B) and other coalition partners providing national root servers. Each country was responsible for providing its own DNS server unless other arrangements were made with another country or organization.

The JWID 2004 sites followed the naming convention of site.service.country with the exception of the Republic of Korea, which followed the naming convention of site.service.8e.country.

Mail Exchanger (MX) records were assigned appropriately within each DNS server with the understanding that all email servers would use the records appropriately.

5.2.1 Improvements on DNS Services

During JWID 2004, for redundancy an additional DNS server was made available via CCCC-Rear that was kept synchronized with the primary DNS server, also located at the CCCC-Rear.

5.2.2 Lessons Learned

Recommendations

23

Page 24: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

- To allow for more controlled and secure access and management, DNSSEC should be implemented. This will require private key exchanges between the various sites that wish to perform updates.

- There should be a way for an administrator at any site to manage their own DNS entries to alleviate some of the problems with personnel time differences. This method could be web-based or could rely on remote update capabilities such as Dynamic DNS.

- Start of Authority (SOA) record timeouts should be standardized across CFBL except for internal zones. This will help keep update times reasonable. Recommend the following intervals in the SOA record:

o Refresh Interval - 1 hour

o Retry Interval - 15 minutes

o Expire Interval - 1 day

o Default TTL - 12 hours

- More sites should consider hosting their own DNS. By hosting their own DNS, the sites will gain additional internal name resolution reliability and may experience fewer issues with finding hosts. This will require a person at each site to be able to implement and maintain a DNS server.

5.3 IP Telephony

The implementation of IP telephony in JWID 2004 was the most extensive JWID implementation ever. In fact, an additional call manager was fielded at Dahlgren to reduce bandwidth to the CCCC-R. The implementation used Cisco Call Managers, version 3.32, and Cisco series 79xx IP phones. The IP telephony system was configured to use the G.711 uncompressed vocoder, approximately 90kbps of IP bandwidth per call, for calls between most sites/countries. The G.711 uncompressed vocoder was chosen to support the Cisco Call Manager Conference feature.

Conference capabilities were again offered to the engineering community and were used for the daily engineering conference calls. The IP phones also proved invaluable for assisting the trials in locating their peers at other sites/countries. This allowed for secure conversations and saved on commercial long distance calls.

Another addition to JWID 2004 was the availability of a backup call manager. This call manager was available if the primary call manager experienced hardware or software failure. Redundancy is necessary because IP telephony is such a widely used service during JWID.

24

Page 25: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

5.3.1 Lessons Learned

IP phones proved to be invaluable tools for both the role players and engineers throughout JWID 2004. They should continue to be deployed extensively at each participating site. The effect of IP telephony and bandwidth usage should be evaluated at each site to determine the number of IP phones they require.

As IP phones are more widely deployed, the impact on site security should be evaluated. During JWID 2004 setup week common security tools like Nessus were run on the Cisco Call Manager and were able to bring the system down. More work should be done with the vendor to determine the correct security settings and vulnerabilities associated with the call manager.

The overall implementation of IP telephony would benefit from vendor-provided training and configuration. The current call manager software includes a number of useful features, such as voicemail and call manager clustering. These capabilities along with the many new features that are available with the latest release of the Cisco Call Manager software should be researched for use in CWID 2005.

IP phone directories for all sites should be finalized during setup week and made available to all users via the JWID Portal prior to execution. This would reduce the number of calls to the site manager requesting IP phone numbers.

The call manager’s success on the Coalition domain indicates that it would be very useful to implement on the US National and MACA Domains for CWID 2005.

5.4 Web

The JWID 2004 web site, http://www.joint.nz, was hosted by New Zealand. The hardware was a Pentium III 666 MHz with 512 MB RAM and 26 GB hard drive running Windows 2003. The web site featured the JWID Portal running on a Microsoft Internet Information Services (IIS) 6.0 web server, collaborative newsgroups served by the IIS Network News Transport Protocol (NNTP) service, and MySQL 4.0.20 database. A mirror site running an UNIX FTP service was located CCCC-R, Arlington, VA, USA, and supported the network infrastructure team. For the sake of redundancy, the JWID site was mirrored at the CCCC-R. This mirror copy was briefly utilized while NZ was still preparing to bring its network connection online. The mirror server hardware was a Sun Blade 1000 with 512 MB RAM and 73 GB of storage. Netscape Enterprise Server 6.1 was installed along with FTP, Netscape Directory Server 5.1 for authentication, MySQL 3.23.32 for database storage, and Perl version 5.005_03 to host the portal and other services.

25

Page 26: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

5.4.1 Overall Web Structure and Format

The JWID 2004 Portal web site was located at http://www.joint.nz. The JWID 2004 web site format was based on the JWID 2003 site in design and functionality as well as the JWID Public website that runs continuously on the Internet, located at http://www.jwid.js.mil. Documents posted to the site were converted to HTML and PDF due to requests from the international JWID community to publish documents in non-Microsoft specific formats. Core site functions included a search capability, hyperlinks to JWID Newsgroups, and a Perl/MySQL-based Bubble Chart and Lessons Learned database.

Figure 10 JWID 2004 Web Site

Most information published to the JWID Portal was gathered prior to scenario execution. If data needed to be updated during the execution phase, it was sent to the CCCC-R Webmaster email account or uploaded to the FTP site. The CCCC-R web administrator would make the necessary changes and then email the files to the corresponding NZ web administrator, who would receive and upload the file to the JWID 2004 web server. Each day the NZ web server would, via an automated script, export the MySQL database into an Excel spreadsheet and upload it to a designated area on the FTP site. The CCCC-R web administrator would then retrieve the spreadsheet, export the data to a comma delimited text file, and import it into the MySQL server at the CCCC-R for redundancy.

The JWID 2004 Portal design included top-level menus for All Users, CTF Staff, Site Director, Site Engineer, Links, and Lessons Learned. The front-page graphic design provided quick links to the JWID Overview, Plan of the Day, Daily SITREP Newsgroup, Bubble Chart, CTF WebTool, DCTS, and the JWID 2004 Welcome Video.

26

Page 27: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

5.4.2 Newsgroups

Newsgroups were a valuable service through which JWID warfighters, engineers and management staff could quickly exchange information. Users had the ability to upload information and read all other articles posted. The following newsgroups were set up on the newsgroup server:

Newsgroup Group of JWID Warfighters/Newsgroup function

jwid04.sitrepjwid04.nwg

SITREP NetworksWorking Group

Table 3. Newsgroups

5.4.3 Bubble Chart

The JWID 2004 Bubble Chart served as a dynamic management tool to rapidly ascertain the current status of all JWID network services and Coalition Interoperability Trials (CITs). The network services chart displayed the status of core services across all sites. The CIT chart displayed the operational status of each CIT. Due to the large number of CITs, controls were added to display only a subset of all CITs at glance. Also, a mouse-over feature was added so that the descriptive names of the CITs would appear in addition to the trial numbers. On both charts, the last time the status was updated was displayed in Zulu format. On the network services chart the US sites were rolled-up into a single line with the option to click on the link to view the status of each site.

Each site manager had the capability to mark each Bubble Chart block with green, yellow, red, or not applicable. A corresponding legend was added this year to define the four status categories. A JWID user could get an overview of the status of each site, and then click on the site name to view more detailed comments on the site’s status. In addition, a history feature enabled users to see a complete set of all issues over the course of the demonstration. The history could be filtered by site, date, network service or CIT, or status.

27

Page 28: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

The Bubble Chart content was classified as 6-eyes releasable. An Access Control List (ACL) was established to control updates to the charts. To update the chart, a site-specific password was assigned and distributed prior to JWID execution. Once the appropriate username and password were entered into the password dialogue box, the CGI script would automatically take users to the update page for their respective site. Users could only update those sites for which their username and password were assigned. In addition to updating status, the site manager could also include comments for each status box. Whenever the status was changed, or a new comment was entered, the timestamp would update automatically. The latest timestamp would then be reflected on the overview page for the network services or CIT Bubble Chart.

Figure 11 Example Network Bubble Chart

28

Page 29: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 12 Example CIT Bubble Chart

5.4.4 Lessons Learned Database

The Lessons Learned database was a CGI/Perl-based form where users could input their comments, and make suggestions to improve future JWIDs. It served as an important feedback mechanism. The form collected name, organization, email, phone number, area of comment, comments, and suggestions. Once added to the database, all users could view all comments. Sixteen comments and suggestions were received.

29

Page 30: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 13 Lessons Learned Database

5.4.4.1Lessons Learned about Lessons Learned Database

OBSERVATION: The lessons learned database provided little feedback to the user when a lesson had been successfully entered into the database.IMPACT: Users might feel confused and enter lessons multiple times resulting in redundancy and a lack of consistency between multiple versions of the same lesson.RECOMMENDATION: Develop a short response for users when data is successfully entered into the database, such as "Lesson successfully entered into database. Would you like to enter another?"

OBSERVATION: The lessons learned database does not allow users to preview a lesson before it is entered into the database.IMPACT: Incorrect or inaccurate information may be presented in a lesson.RECOMMENDATION: Develop a preview capability to allow users an opportunity to double-check lessons before committing them to the database.

30

Page 31: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

OBSERVATION: Multiple users saw variations of the same problem, but the lessons learned database does not allow for additional users to comment on a single lesson learned. Comments by additional users may bring clarity to a particular issue and facilitate resolution or improvement.IMPACT: Multiple lessons learned from multiple users are difficult to correlate and adequate resolution may not occur.RECOMMENDATION: Develop the capability to allow multiple users to comment on a single lesson learned. This will provide greater clarity on the issue at hand.

OBSERVATION: The lessons learned database is an opportunity for collaboration at a different level than the hotwash. This opportunity enables personnel with technical knowledge of a problem to collaborate in a way that hotwashes and final reports cannot capture.IMPACT: Technical collaboration on lessons learned will improve future JWIDs.RECOMMENDATION: Available weblogging software meets the above recommendations and allows for collaboration on multiple topics as well as for documenting the various lessons learned throughout the course of the demonstration. Software such as Movable Type or Wordpress enables project collaboration, moderation, and documentation of lessons learned. This software is specifically engineered for multi-user environments. The technical requirements for such software are relatively minimal, typically, a web server with a database and support for any number of scripting languages such as Perl and PHP. This software is also cost-effective. For example, Wordpress is free for commercial use, and Movable Type's fee is minimal.

5.4.5 Master Scenario Events List (MSEL) Web Based Tool (MWBT)

During JWID 2004, the AITS-JPO hosted the servers for the MWBT and provided system administration support and back up operations. The Scenario Working Group provided content design and maintenance. During the course of JWID 2004 planning it was determined that the MWBT resource needed to be available on all three classification domains. This required the AITS-JPO to stand up two additional servers, four servers in all. Prior to execution users accessed the unclassified MWBT server via the web and entered MSELs into the database. The servers for the Coalition and the US National Domains where kept off line until 4 June. At that time the Scenario Working Group locked down the unclassified MWBT, restricting changes to the MSEL database. System administrators installed copies of this database on the servers, which were accessible to both the Coalition and US National domains. The locked-down MWBT was restarted on all three domains and made available to JWID users on 4 June. At the end of JWID 2004 the servers on the Coalition and the US National domains were again taken off line. The databases were verified to be unclassified and a copy of all three is stored and accessible via the unclassified server at http://jwid.les.disa.mil/mwbt and

31

Page 32: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

http://jwid.les.disa.mil/jdcat. Tape backup copies of all three databases are stored at the AITS-JPO.

5.5 Defense Collaboration Tool Suite (DCTS)

JWID 2004 implemented DCTS, version 2, phase 1, which was the same version that was used during JWID 2003 execution. DCTS supported all three networks. The key to successful DCTS collaboration capabilities was to ensure sufficient availability of network bandwidth, properly sized servers and clients (high CPU, video card and memory capabilities), and adequate user training levels. Network statistics showed that the available bandwidth during JWID 2004 was sufficient to support the scope of DCTS for the JWID scenario despite the constraint of running three networks on the same physical access circuit. 5.5.1 DCTS ON THE CFBLNet

Fourteen CUSeeMe servers, running version 6.0.6.49, were deployed on the CFBLNet (Figure 14). The biggest change from previous years was the implementation of the domain model of conference server configuration. In previous JWIDs a configuration file was distributed to all sites in order to link conference servers. All sites were then brought online systematically in order to ensure they federated properly. If any additional conferences were requested, a new configuration file was created and distributed, which required an additional systematic reboot by all sites. In the domain model configuration, once a site’s conference server joined the domain, the administrative server updated the configuration information automatically. A configuration file and a systematic reboot were not needed. By using the domain model the administration of DCTS was centralized at the AITS-JPO. Requests for additional conferences during execution were done without additional coordination from the remote sites. Also in previous JWIDs, the MCU at the AITS-JPO served as both the top provider as well as a conferencing server. To alleviate traffic to the top provider an additional server was installed at the AITS-JPO to act solely as a conferencing server for the DCTS users in the National Capital Region.

32

Page 33: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 14CFBLNet MCU Topology

Conference server stability did not improve from JWID 2003 because the same version of CUseeMe (6.0.6.49) was used; however, each DCTS conference configuration was the same as in previous years in terms of codecs: H.323 (video), G.711 (audio). In order to attempt to improve performance with conferencing, bandwidth settings for each conference increased to 512K from client to server (256K in 2003), and 1024K between CUseeMe server to CUseeMe server (512K in 2003).

T.120 connectivity between some CUseeMe Servers in certain conferences was unreliable at times. There were instances when the T.120 portion of some conferences became unlinked from the Top Provider (AITS-JPO Server). This happened with the Hanscom, UK, and NZ servers. When this unlinking occurred, engineers performed two troubleshooting methods to fix the problem. Engineers restarted the CUseeMe service on the “problematic” server. After the restart, conferences sometimes linked, and sometimes did not. The chance of linking was greater when there was not much activity on the top provider, but some conferences did not link even when there was no activity on the top provider. Also, the engineers deleted and recreated “problematic” conferences. After the conferences were recreated, they linked in to the top provider and full functionality was restored. The UK and NZ conference server service seemed to stop for unknown reasons at the end of each execution day. At the end of each week of execution the servers were systematically rebooted to clear the servers memory and re-link all the conferences with the top provider.

While the T.120 issues provided the most numerous connectivity problems, the issues that were experienced during last year’s hotwash were not experienced during JWID 2004. During the hotwash the NetMeeting video stopped transmitting halfway through the briefing. Even though video stopped switching after Australia’s portion of the brief, audio continued to transmit clearly.

33

AITSJPO 1

UK

Dahlgren 1

AUSTRALIA

New Zealand

CANADAAAQ

Dahlgren 2

NC3ANATO

HANSCOM

SPAWARNORTHCOM

Top Provider AITSJPO2

EAGLE

Page 34: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Despite the problem with video, a reboot of the CuseeMe servers was not required. The audio transmitted very clearly and conference participants were able to view the shared application

5.5.2 Lessons Learned

In order to fully understand the issues that affected DCTS during JWID 2004, it is crucial to have First Virtual Communications, the developers of CUSeeMe, observe future JWID exercises during execution. First Virtual Communications viewed the configuration of the conferences prior to JWID and noted no problems with the configurations. They pointed out that JWID used much older versions of their conference server, and that their new version of CUSeeMe fixes many of the problems that JWID users have experienced in the past. Their on-site guidance during execution would have been crucial when anomalies like the connectivity issues described above occurred.

As experienced during JWID 2003, closing out Net Meeting incorrectly continued to cause problems the next time the user connected to a conference. When a user clicked the “X” in the upper right corner of the Net Meeting window to end a collaborative session, T.120 capabilities were sometimes not available when that user attempted to enter a new conference. Users must click the “hang-up” icon when exiting a conference.

Another issue was users not using their local conference server while in conferences. This problem was present throughout JWID, despite phone calls to the site informing the site of the problem.

The failure of users to mute their mikes caused echoing and the transmission of unwanted background noise during several collaborative sessions. Because this has been an issue for such a long time, U-Talk was developed to fix the hot mike problem during conferences. The final release of U-Talk was developed and released just before execution so there was not adequate time to train the users. The decision was made not to implement the web part on the CFBLNet and users continued to complain about the hot mike issues throughout the exercise. U-Talk was fully implemented on the MACA and US National networks, and as a result users did not experience many problems while in conferences, and conference servers were much more stable.

The troubleshooting process was greatly improved with use of the UMon tool. Administrators at the AITS-JPO could view the status of all conference servers instantaneously. Administrators could also troubleshoot problems more quickly based on the information provided by UMon. The UMon tool was also able to continuously display network latency between the conference servers, allowing the DCTS administrators to inform sites that their DCTS performance was based on intermittent bottlenecks on the network. In previous years, data like that, which UMon provides, was not available.

34

Page 35: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

DCTS user and site engineer training for all JWID sites prior to execution was imperative. Many of the problems that were experienced during JWID were user-related or user-workstation related: users not exiting the conferences properly, users not using the push-to-talk methodology, etc. If DCTS training sessions are implemented at the sites, many of the problems that were experienced by the users could have been alleviated. Also, for future JWIDs it is critical to have a DCTS trainer on-site during JWID execution. If a DCTS trainer had been on-site most, if not all, problems would have been quickly fixed or totally avoided. The U-Talk web-part could have been easily implemented had a DCTS trainer been present at each site to instruct them best on how to use the tool.

5.6 Network Engineering Management

Comprehensive network monitoring/management is critical in early detection of network problems. Additionally, Network monitoring/management tools are capable of providing feedback to sites and engineers, and demonstrations and applications as to why they were experiencing problems. During JWID execution, not only basic site connectivity, but also bandwidth utilization categorized by application, host, port number, etc…; OSPF and BGP routing protocol status; ATM SVC status; ATM signaling errors; port and interface input and output errors; multicast neighbor status; and, multicast shortest tree path (STP) status should be monitored. This information, along with the ability to monitor and initiate trouble tickets, should be readily available to any site on the JWID network.

During JWID 2004, the CCCC-R used HP OpenView Network Node Manager (NNM) and OpenNMS for general network monitoring. NetScout, Network Traffic Analysis System (NTAS), and Flow Tools and FlowScan were used for monitoring bandwidth utilization; TCPDump and TRPR were used for monitoring DCTS server-to-server traffic in both real and non-real time. For trouble ticket generation, ARS Remedy was used. All of the tools were web-based which allowed site engineers from any site to easily view and/or update the tools with the appropriate permissions. Other tools were evaluated but deemed either non-applicable or too complex for proper setup during the timeframe available.

5.6.1 Network Monitoring Tools

This year, the CCCC-R made strides to evaluate many Open Source (OSI Certified License) applications to provide greater network monitoring and reporting capabilities at lower costs. The information shown below notes some of the Open Source tools that were used along with the more traditional tools used during JWID 2004.

5.6.1.1 NetScout

35

Page 36: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

The NetScout system worked very well to monitor standard SNMP devices and to connect to probes at the JPO and at other US sites. The probes allowed a much deeper look at the traffic statistics passing across the network at various points.

Figure 15NetScout

Pros:

- A unified web-based interface made it very easy to get to all of the information from any machine.

- The newspapers were helpful in providing customized reports to the people that needed them.

- This was the only device available that was able to look into ATM traffic.- SNMP traps can be sent to other devices on noticed events

Cons:- The web page was not SSL encrypted.

36

Page 37: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

- Getting custom email alerts did not seem to be possible (this may be a configuration error).

- Applying this technology to another information domain was cost-prohibitive for a time as short as JWID execution.

Conclusion:

The NetScout product is a very effective tool and provides a good look into the traffic along any link. As such, having a probe at every ATM link would be advantageous. However, security needs to be taken into more consideration and the cost of applying the NetScout solution to other information domains needs to be analyzed.

5.6.1.2 HP OpenView Network Node Manager (NNM)

The HP OpenView NNM was mainly used to provide a high-level view of network devices that the CCCC-R wished to monitor. A map was put together that contained all of the interfaces. Each interface, or parent node, would change color if the node were unreachable via “ping”. Email alerting capabilities are present in NNM as well as the ability to integrate into ARS Remedy. However, neither of these functions was configured for this installation of NNM. From this experience with OV, it would be advisable to have one person dedicated to the management and maintenance of OV as well as having that person dedicated to extending the monitoring capabilities of the system. As with NetScout, analysis needs to be performed to determine whether or not applying an OpenView installation would be cost-effective on other information domains.

5.6.1.3 OpenNMS

OpenNMS was used throughout the JWID 2004 exercise to monitor all major CFBLNet equipment and many of the services available on the network. The program provided uptime statistics and graphs per interface, and emailed a designated account if a problem was noticed. This tool was used for statistical information while NTAS was inoperative.

37

Page 38: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 16OpenNMS

Pros:

- No costo This makes it a good candidate for short term networks or low cost deployments

- Runs on a variety of Operating Systems- Provides a great deal of required functionality out of the box- Highly configurable and customizable- Can collect SNMP data and SNMP traps and send alerts on data observed- Performs all of the basic monitoring options of HP OpenView NNM- Wrapped via Apache so there is no problem with SSL support or authentication- Can wrapper Nessus for automatic system vulnerability scanning and reporting- Commercial Support Available

Cons:

- Difficult to customizeo Utilizes JSP for all of its polling. This makes it more difficult for the average

administrator to add additional polling capabilities- No functional “Network Map” (OpenView Style)

38

Page 39: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

- Multiple installations may be required to support full data collection on different sites.o This is not difficult just more resource intensive

- Quite resource intensive (but no more than HPOV)- Only two real user ‘levels’: Administrator and View

Conclusion:

OpenNMS provided a great deal of the required functionality for network equipment monitoring. More time is needed to tailor the program for the CFBLNet environment but the amount of performance that was gained for the cost was excellent.

5.6.1.4 Network Traffic Analysis System (NTAS)

NTAS is a GOTS product that was used during JWID 2003 and again in JWID 2004. NTAS takes NetFlow input from various network devices and allows statistics to be pulled from that data. NTAS can also provide link status between devices and link utilization. NTAS was only available on CFBLNet for the second week of the demonstration.

Figure 17 NTAS

Pros:- No cost- Near real time- Gives a graphical interface to the statistical data collected- Can provide customized reports on a daily basis- Can pass the NetFlow data on to other systems for alternate collection/analysis

39

Page 40: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Cons:- Seemed difficult to keep running in a stable manner- Does not allow for user report customization

o NTAS developers were required to accomplish this task- The interface was unwieldy- The NetFlow forwarding ability failed several times during execution- Does not support SSL web connections- Does not allow for alerts to be sent via email when a problem is noticed- Does not seem to contain a framework for adding notification abilities- System functionality failed once DISA required STIGS where applied using the required

“Gold Disk”

Conclusion:

NTAS has the potential to be a useful tool in the CFBLNet network-monitoring arsenal if the cons mentioned above are addressed, particularly in the realm of automatic notification. In addition, DISA needs to develop a listing of “Not to apply” lock downs exceptions so that NTAS can function in a DISA environment.

5.6.1.5 Flow Tools/FlowScan

Flow Tools and FlowScan handle NetFlow accounting and reporting. The Flow Tools package is a suite of tools for collecting, generating, and reporting on NetFlow data. FlowScan is a package that parses the collected flow data and can generate reports based on flow content.

Pros:- No cost- Provided a very efficient means for aggregating flow collections- Could generate very detailed reports on any given set of data- Could provide graphical summaries of flows across time- Would be fairly easy to combine with a wrapper to send e-mail or SNMP traps if a

problem is noticed- Extremely fast

Cons:- Complex configuration- Cryptic (but powerful) reporting syntax

Conclusion:

These two packages were an effective way to quickly get statistics regarding network flow data and should be considered for use in the future. Given the proper reporting requirements and time to generate the report structure, this software should be able to give any amount of detail that can be extracted from a network flow.

40

Page 41: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Figure 18FlowScan

Cacti is a generic web-based interface that can be used to graph arbitrary linear time data. This is especially useful in polling SNMP statistics or statistics from log files and/or servers. This program was only briefly set up during JWID 2004 so data is limited.

Pros:- No cost- Highly configurable- Data sources easy to define and collect from any programming language- Looks to be quite scalable- Can be wrapped behind Apache for SSL support- Paired with Nagios (below) becomes a very powerful network monitoring solution- Commercial Support Available

41

Page 42: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

Cons:- Configurability at the price of complexity- Automatically collected Router interfaces may not always show correctly

o This is not a problem is they are input by hand- Knowledge of RRDTool is not necessary but useful- Cannot natively watch for threshold overruns

o This ability can be added to the collection scripts but that is fairly unwieldyo In this case, pairing with Nagios would be the best implementation

Conclusion:

Cacti is a good addition to the network management toolbox. With the ability to easily generate graphs for any time-linear numeric data, statistics can be collected from any data source. However, Cacti requires more hands-on development than OpenNMS.

5.6.2 Lessons Learned

JWID network managers have traditionally focused on monitoring the availability of network circuits and utilization of those circuits using tools such as MRTG. However, in recent years the network management challenge has expanded to include tracking the status of network services and applications. An important piece of this challenge is to characterize network traffic and account for how and where it flows. CIT 07.03 Peakflow, CIT 07.12 NetScout and the Network Traffic Analysis System (NTAS) developed by DISA’s Technology Integration support group were tools that the CCCC-R used in JWID to address this problem.

In general all the tools could be useful to managers of operational DoD networks. Peakflow provided excellent graphing of historical data, but unfortunately CCCC engineers’ evaluation of the product was limited due to a NetFlow bug in the Cisco IOS that was used in JWID. NetScout provided the best “real-time” data of the three products and was particularly valuable in monitoring DCTS conferences. NetScout’s historical data provided good high-level reports, but CCCC engineers would like to see more visualization capabilities of the historical data. NTAS provide the simplest and most intuitive graphical user interface of the three tools, but CCCC engineers would like to see more capabilities for plotting historical data.

5.6.2.1 Future Considerations

Above was a quick overview of some of the tools used during JWID 2004 by the CCCC-R for both network monitoring and network reporting. Some other Open Source Software that should be analyzed include:

- Nagioso Network monitoringo Easily customizedo Large user community

42

Page 43: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

o Graphical network mapso Automatic alertso Commercial support available

- Open Ticket Request System (OTRS)o Email basedo Allows a more user-friendly interface to trouble ticket systemso Very easy to integrate via any program that can send emailo Commercial support available

- Network Top (Ntop)o Allows for the creation of very inexpensive LAN style network probeso Advanced user interfaceo Acts as a NetFlow collector/distributoro Easily generates RMON style reportso Commercial support available

- BigSistero Monitors systems and services using custom scriptso Gives a ‘bubble chart’ type interface for status monitoringo Can send alerts or perform other actions on a service outageo Commercial support available

5.7 Network Statistics Collected During JWID 2004

During JWID execution, data regarding site I/O traffic, errors, and service uptime (if available) was monitored along with all DCTS data between AITS-JPO and any other site.

Data was collected via SNMP from the US routers and switches and from the international border routers. Additionally, core US services were monitored via OpenNMS and uptime was reported for each service monitored. This is a vast improvement over last year when services were not automatically monitored.

DCTS traffic was gathered and parsed with TCPDump and TRPR using remote spanning from the DCTS switch at the AITS-JPO. TCPDump recorded every packet, and TRPR processed this information to produce traffic information in average kbps over one-half second intervals. Live plots of the DCTS data to and from each MCU were viewed during execution for real-time troubleshooting purposes. Network data briefings for each day of execution are on file with configuration management at the AITS-JPO.

The following graphs are examples of the traffic information that was collected during JWID execution:

43

Page 44: COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

JWID 2004 Final Report

44