ppt

April 10, 2023

E2E piPEs / AMI / OWAMP StatusE2E piPEs / AMI / OWAMP Status

Eric Boyd and Jeff Boote

April 10, 2003

Eric Boyd and Jeff Boote

April 10, 2003

04/10/23 2

Agenda

• Motivation

• piPEs / AMI Overview

• piPEs / AMI Deployment Plan

• OWAMP Release

• Demo

04/10/23 3

Context for E2E Performance

Even with high bandwidth network links, the Internet2 community often does not see expected performance.

04/10/23 4

The Wizard Gap

04/10/23 5

A Problem

Applications Developer

System Administrator

LAN Administrator

CampusNetworking

Gigapop Gigapop

Backbone

CampusNetworking

LAN Administrator

System Administrator

Applications Developer

How do you solvea problem along a path?

Hey, this is not working right!

The computerIs working OK

Talk to the other guys

Everything isAOK

No othercomplaints

The network is lightly loaded

All the lights are green

We don’t see anything wrong

Looks fine

Others are getting in ok

Not our problem

04/10/23 6

Good Work Already Going On

Many projects and research is underway by others investigating performance on portions of the end-to-end path.

E2Epi seeks to bring these efforts together to look at the end-to-end path as a complete system.

04/10/23 7

Can You Go Direct to the Problem?

How can you tell where is the problem?

Need a tool to tell you:• Where the problem is.• The type of problem• Who to contact to get it fixed

Terry Gray, University of Washington“We Need a Finger Pointing Tool”

04/10/23 8

Gray Finger Pointing Tool

Applications Developer

System Administrator

LAN Administrator

CampusNetworking

Gigapop Gigapop

Backbone

CampusNetworking

LAN Administrator

System Administrator

Applications Developer

Locate the ProblemGray Finger Pointing ToolTerry GrayUniversity of Washington

Finger Pointing Tool Time!

04/10/23 9

Gray Finger Pointing Tool

Applications Developer

System Administrator

LAN Administrator

CampusNetworking

Gigapop Gigapop

Backbone

CampusNetworking

LAN Administrator

System Administrator

Applications Developer

Finger Pointing Tool Time

You ARE the Weakest Link!

OK, I’ll fix it

04/10/23 10

Agenda

• Motivation

• piPEs / AMI Overview

• piPEs / AMI Deployment Plan

• OWAMP Release

• Demo

04/10/23 11

BNI AMI

• Project: BNI Abilene Measurement Infrastructure

• Goal: Instrument Next Generation Abilene network with extensive performance measurement capabilities

• Approach: Collaborative project among BNI, Engineering, and E2E.

04/10/23 12

E2E piPES

• Project: End-to-End Performance Initiative Performance Environment System (E2E piPES)

• Goal: To allow end-users and network operators to determine performance capabilities, locate problems, and contact the right person to get a problem resolved.

• Approach: Collaborative project combining the best work of many organizations.

04/10/23 13

The Measurement System

PMP

PMP

PMP

PMP at each Campus Border

Extend the System to Campuses

AbilenePMP

Campus X

PMP

Campus Y

04/10/23 14

The Measurement System

PMP

PMP

PMP

PMP = Performance Measurement Point (at each Abilene Node)

The New Abilene will have measurement devices as part of its structure

Abilene

04/10/23 15

Performance Reporting

PMP

PMP

PMP

Perform regular tests to provide web based Performance Reports across Abilene and for Campus-Abilene paths

AbilenePMP

Campus X

PMP

Campus YCampus-Abilene CapabilityCampus W – 155 MbpsCampus X – 612 MbpsCampus Y – 1000 MbpsCampus Z – 2 Mbps

Abilene CapabilitySNVA/DNVR – 10 GbpsDNVR/IPLS – 10 GbpsIPLS/NYCM – 10 Gbps

04/10/23 16

Reaching the End-User

PMP

PMP

PMP

Provide a Testing Tool for the End-Users Computer to test across campus and beyond

AbilenePMP

Campus X

PMP

Campus Y

Host A Host B

The tool reports performance of the path to a specified destination by using active tests (like to the campus border) or using stored test results from regularly scheduled tests.

It also performs tests on the host system to check host tuning parameters

04/10/23 17

Adding Path Information

PMP

PMP

PMP

Add PMP along paths to provide additional detail

AbilenePMP

Campus X

PMP

Campus Y

Host A Host B

Regular tests are performed between adjacent PMP and stored in a Data Repository.This information is used by the End-User tool to reduce the number or active tests needed.

PMP

PMP

RegionalNetwork J

PMP

PMP

RegionalNetwork M

Gigapop S

Gigapop T

04/10/23 18

Who You Gonna Call?

PMP

PMP

PMP

E2E piPES will provide contact information when asegment in the path appears to have a problem

AbilenePMP

Campus X

PMP

Campus Y

Host A Host B

The test results obtained by the End-User will be passed on to the contact person

PMP

PMP

RegionalNetwork J

PMP

PMP

RegionalNetwork M

Gigapop S

Gigapop T

piPES Test AnalysisLooks like a problemIn Gigapop T. Pass these test result to:Joe@gigapopT.net

04/10/23 19

E2E piPES Architecture

End UserInterface

Network AdministratorInterface

PMPX

PMPY

PMPZ

CulpritDB

“Turn Yourself In”Culprit Interface

ID Self

CheckShibboleth

QueryCulprit

IDCulprit

Report “Raw” Results

Specify end points &application family

Request TestResult

Check Policy

Schedule Test

Scheduler

Performance MeasurementDB (could be local to PMP)

TestingEngine

AnalysisEngine

DispatchSchedule

Send RawResult/Refusal

Specify Test &End Nodes

Request On-Demand Test

TestArbiter

CheckShibboleth

SendConclusion

ResultArbiter Check

ShibbolethAck/Reject

ResultReady

04/10/23 20

Testing / Analysis Engine

Problem: Encode Matt Z.’s Brain

Open Questions:

•What are measurement results are “acceptable” for a given application (family)?

•What tools generate those results?

•How do you handle incomplete data?

•How do you rank multiple result-generators?

•What is the iterative decision tree to understand E2E problem?

04/10/23 21

Measurement Schema

Measurement Types?

e.g. BW, Latency, Loss, Jitter

Measurement Units?

e.g. Seconds vs. Microseconds

Map Tools to Measurements?

e.g. OWAMP -> 1-way Latency

Measurement Metadata?

Database Table Design?

04/10/23 22

Access, Authentication, Authorization

Roles for Access• Standard End-User?• Near Neighbor (Test Buddy)?• NOC Staff / Network Engineer?

Shibboleth for Implementation• Each campus decides who in each role• Others trust campus designation

04/10/23 23

Other R&D Issues

Design system to scale (eliminate centralization)?

How to “discover” PMPs?

How to insert On-Demand tests into Regularly-Scheduled test set

Balance centralization and distributed database requirements

04/10/23 24

Agenda

• Motivation

• piPEs / AMI Overview

• piPEs / AMI Deployment Plan

• OWAMP Release

• Demo

04/10/23 25

E2E piPES Architecture v1.0

End UserInterface

Network AdministratorInterface

PMPX

PMPY

PMPZ

CulpritDB

“Turn Yourself In”Culprit Interface

ID Self

CheckShibboleth

QueryCulprit

IDCulprit

Report “Raw” Results

Specify end points &application family

Request TestResult

Check Policy

Schedule Test

Scheduler

Performance MeasurementDB (could be local to PMP)

TestingEngine

AnalysisEngine

DispatchSchedule

Send RawResult/Refusal

Specify Test &End Nodes

Request On-Demand Test

TestArbiter

CheckShibboleth

SendConclusion

ResultArbiter Check

ShibbolethAck/Reject

ResultReady

04/10/23 26

piPEs / AMI Rollout

AMIPMP

AMIPMP

AMIPMP

OWAMPPERF

DB

Visualization Package

OWAMP OWAMP

OWAMP

04/10/23 27

piPEs / AMI Rollout

AMIPMP

AMIPMP

AMIPMP

OWAMPPERF

DB

Visualization Package

OWAMP OWAMP

OWAMP

Web Service

04/10/23 28

piPEs / AMI Rollout

AMIPMP

AMIPMP

AMIPMP

OWAMPPERF

DB

Visualization Package

OWAMPIPERF

TracerouteSNMP

OWAMPIPERF

TracerouteSNMP

OWAMP, IPERF, Traceroute, SNMP

Web Service

TraceroutePERF

DB

IPERFPERF

DB

SNMPPERF

DB

04/10/23 29

piPEs / AMI Rollout

AMIPMP

AMIPMP

AMIPMP

OWAMPPERF

DB

Visualization Package

OWAMPIPERF

TracerouteSNMP

OWAMPIPERF

TracerouteSNMP

OWAMP, IPERF, Traceroute, SNMP

Web Service

TraceroutePERF

DB

IPERFPERF

DB

SNMPPERF

DB

CampusPMP

CampusPMP

04/10/23 30

piPEs / AMI Rollout

AMIPMP

AMIPMP

AMIPMP

OWAMPPERF

DB

Visualization Package

OWAMPIPERF

TracerouteSNMP

OWAMPIPERF

TracerouteSNMP

OWAMP, IPERF, Traceroute, SNMP

Web Service

TraceroutePERF

DB

IPERFPERF

DB

SNMPPERF

DB

CampusPMP

CampusPMP

Scheduler

Web Service

Configuration Package

04/10/23 31

piPEs / AMI Development & Testing Environment

nmsXServer

Server

nmsXServer

Development

TestingTests Tests

New Code

Base Production Code

MeasurementNode

New Production Code

MasterMeasurementNode

MeasurementNode

MeasurementNode

MeasurementNode

AbileneMeasurementNodes

. . . . . . . . . . . . . . . . . . . . . .

Base/New Production Code

Tests Tests Tests

Server

Test Results

DatabaseServer

PerformanceVisualization Tool

MeasurementNode

Router

Switch nms1

nms3 nms2nms4

CDMA

04/10/23 32

piPEs / AMI Internal Resources

Eric Boyd – piPEs development, project coordination

Jeff Boote – OWAMP, piPEs development

Prasad Calyam – piPEs development

Chris Heermann – AMI

Matt Zekauskas - AMI

Susan Evett - Documentation

Russ Hobby – Campus Deployment

George Brett – Schema, “Grid Service”

04/10/23 33

piPEs External Resources

Yee-Ting Li – UCL (50%)

Paul Mealor – UCL (up to 100%)

PhD Student – UCL (contingent on grant)

Warren Matthews – SLAC

Jim Ferguson et al – NLANR / DAST

Many early adopter volunteers

04/10/23 34

Agenda

• Motivation

• piPEs / AMI Overview

• piPEs / AMI Deployment Plan

• OWAMP Release

• Demo

04/10/23 35

Latency Tests

Problem: Existing platforms are not interoperable (SURVEYOR,RIPE,…)

Solution: Standards

04/10/23 36

OWAMP

One-Way-Active-Measurement-Protocol• Possible due to growing availability of good time sources

• Wide deployment of “open” servers would allow measurement of one-way delay to become as commonplace as measurement of RTT using ICMP tools such as ping.

• Current Draft: draft-ietf-ippm-owdp-05.txt–Shalunov,Teitelbaum,Zekauskas

• Sample implementation under developmentBeta Release: http://owamp.internet2.edu/

04/10/23 37

OWAMP Implementation

Basically:NTP system call interface

Multiple processes for recv/send loops

Written as an API to allow one-off implementations

04/10/23 38

OWAMP Error Estimates

Rate = 1 packets/sec

Min/Median = 59/64 usec

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

TO: 1/sec95% = 0.114

FROM: 1/sec95% = 0.106

04/10/23 39

OWAMP Error Estimates

Rate = 10 packets/sec

Min/Median = 57/63 usec

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

TO: 10/sec95% = 0.080

FROM: 10/sec95% = 0.076

04/10/23 40

OWAMP Error Estimates

Rate = 100 packets/secMin/Median = 54/73 usec

0

0.5

1

1.5

2

2.5

3

3.5

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

TO: 100/sec95% = 0.098

FROM: 100/sec95% = 0.094

04/10/23 41

OWAMP Error Estimates

Rate = 1000 packets/secMin/Median = 53/176 usec

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

TO: 1000/sec95% = 0.294

FROM: 1000/sec95% = 0.279

04/10/23 42

OWAMP Errors

Preliminary Findings:• Min error estimates look to be in the 55-60 usec range.

• Serialization Delay: ~5usec x 2• Get Timestamp: ~15usec x 2• Additional error is:

–Time from userland “send” to 1st byte hits the wire–Time from kernel has packet to userland “recv” returns–Potentially recv process data processing before calling

“recv”

04/10/23 43

OWAMP implementation status

Sample implementation

http://owamp.internet2.edu/

• Beta Release:–No “policy”–No authentication/encryption–FreeBSD only

04/10/23 44

Abilene OWAMP deployment

2 overlapping full meshes (IPv4 & IPv6)• 11 measurement nodes = 220 ongoing tests• UDP singletons• Rate: 10 packets/second*• Packetsize: (32 byte payload)*• Results are continuously streamed back to “Measurement Portal” for long-term archive and data dissemination (Near real-time)

04/10/23 45

Agenda

• Motivation

• piPEs / AMI Overview

• piPEs / AMI Deployment Plan

• OWAMP Release

• Demo

04/10/23 46

piPEs / AMI / OWAMP Demo

http://owamp.internet2.edu

when available

04/10/23 47

More Internet2 Information

On the Web• www.internet2.edu• e2epi.internet2.edu

Email• Info-e2epi@internet2.edu

04/10/23 48

Questions?

www.internet2.edu