Metrics and Methods for IP Capacity

draft-ietf-ippm-capacity-metric-method-01

A. Morton, R. Geib, L. Ciavattone

Receiver Rate Measurement

2

IP-Layer Capacity

Time =

dtn

n=2

dt

dtn

n=3dt4 dtn+1

TT + I

Trial results

Complete Test

. . . . . . dtn

n=1

dtn

n=m

Define the Metric

• Maximum IP-Layer Capacity (incl headers + UDP payload)

• One of many metrics that could be defined• Def. in Words and an Equation (with variables

explained)max ( n0[dtn,dtn+1] )

[T,T+I]

Maximum_C(T,I,PM) = -------------------------

dt

where:

T < ------ Measurement Interval ------> T+I

_________________________________________

| | | | | | | | | | |

dtn=1 2 3 4 5 6 7 8 9 10 n+1

m=10

sub-intervals3

IPPM Draft Status• Many-many comments and reviews have

resulted in a very complete draft.

– New Reviews from ETSI STQ MOBILE

– Four New Members of ITU-T SG12 (testing co’s)

– Testing from various volunteers

• Key topics updated/added in 01:

– Measurement Considerations

– Reporting Formats

4

8.3 Meas. Considerations (new)Conditions which might be encountered during measurement, where packet losses may occur independently from send rate:

1. Congestion of an interconnection or backbone interface may appear as packet losses distributed over time in the test stream, due to much higher rate interfaces in the backbone.

2. Packet loss due to use of Random Early Detection (RED) or other active queue management.

3. There may be only small delay variation independent of sending rate under these conditions, too. THIS IS A “TELL”

4. Persistent competing traffic on measurement paths that include shared media may cause random packet losses in the test stream.

It is possible to mitigate these conditions… but try locating measurement points as close as possible, first! 5

8.3 Meas. Considerations (new)where packet losses occur independently from send rate:

Mitigate using parameters of search alg. described in Section 8.1 (tuning specific parameters, more flexibility than typical CCA).

6

START:

New Status

Report

Seq Errors = 0

and DelayVar

<= LowThresh

WAIT for

New Status

Report

YES

NO

(absence of

congestion)

(definite indication of

congestion)

Results Reporting Considerations• “Turbo-mode” concept (Matt Mathis’ testing)

• Report separate results for repeatable modes

7

• Other modes may be encountered (repeatable?)

• Radio constellations, Cellular modes, weather

Time =

dtn+1

TT + I

dtn

n=1n=m

IP-L

ayer

Cap

acit

y

9. Reporting Format Elements (Others?)

The Singleton IP-Layer Capacity results SHOULD be accompanied by the context under which they were measured.

o timestamps

(especially the time when the maximum was observed in dtn) o source and destination (by IP or other meaningful ID)

o other inner parameters of the measurement (Section 4)

o outer parameters, such as “performed in motion" or other factors belonging to the context of the measurement

o result validity (indicating cases where the process was somehow interrupted or the attempt failed)

o a field where unusual circumstances could be documented

o a field for "ignore/mask out" purposes in further processing8

9

Standards High-Level Status: IP-Layer Capacity Metric and Meas. • ITU-T Study Group 12 - Approved

• Question 17 on Packet Network Performance the Metric and Method of Measurement to Rec. Y.1540 - 2019 (Annexes A and B)• Considerable background (test results; research) in Appendices X thru XIII

• ETSI TC Speech and Multimedia Transmission Quality (STQ)• Approved the Metric in TS 103 222 Part 2 on High Speed Internet KPIs• Reference to Rec Y.1540 for all other material

• Broadband Forum (BBF) – Project Approved: WT-471• Standardize the identical Metric and Methods with additional details

on Measurement Points and Information Model for control and reporting. First Ballot in May, 2020, next meeting in June.

• IETF IP Performance Measurements (IPPM) Working Group • Internet Draft Adopted by WG, adding Metric details, Measurement

Considerations, and Results presentation formats

Next Steps• Post-WG Adoption work:

– Harmonization: Keep-up with parallel efforts to ensure IPPM’s expertise incorporated elsewhere

– Reach Consensus soon, start protocol support

• Additional Volunteers for Review

– Trigger more reviews with a WG Last Call?

10

REFERENCES• Hackfest 106 Slides: Test Results

• Hackfest 105 Slides: Test Results

• Liaisons from ITU-T SG 12 and ETSI TC STQ –see email for links, or

• https://datatracker.ietf.org/liaison/1645/

• https://datatracker.ietf.org/liaison/1643/

• https://datatracker.ietf.org/liaison/1634/

• https://datatracker.ietf.org/liaison/1632/

• More Test results in the Liaison attachments11

BACKUP

12

Define the Method• “PM” is short-hand for the performance

constraints on the Load Rate Adjustment Alg.:

13

Recent Test Results

1414

Udpst and Ookla Web Sockets Clients Udpst and Ookla Web Sockets Servers

1Gbps Access Service and IXP

IETF – Sept List Points raised (and addressed) -1

• @@@@ A clear take-away is that reporting must account for bimodal features, if/when measured.

• ++++ Covered in Section 6.6, reporting the Metric

• @@@@ Also, that wide-spread measurements will encounter wide-spread behaviors - testing should continue + expect some evolution.

• ++++ Covered in the Methods of Measurement Section

• @@@@ IMO, many of the above challenges fall on the measurement methodology: allow for traffic & time to initiate an on-demand access.

• @@@@ Also, results depend on the sending stream characteristics; we've known this for a long time, still need to keep it front of mind.

• ++++ both above covered in Methods of Measurement, Considerations.

• @@@@ Max IP-Layer Capacity and RFC 3148 BTC (goodput) are different metrics. Max IP-layer Capacity is like the theoretical goal for goodput.

• ++++ Section 1, Intro

• @@@@ This is a big one: when the path we measure is state-full based on many factors, the Parameter "Time of day" when a test starts is not enough info. We need to know the time from the beginning of a measured flow, and how the flow is constructed including how much traffic has already been sent on that flow, because state-change may be based on time or bytes sent or both. See RFC 7312.

• ++++ included in Measurement Considerations

IETF – Sept List Points raised (and addressed) -2• @@@@ The Singleton and Statistic formulations of IPPM's framework RFC 2330

are still valuable in this context, possibly combined with results criteria ("stable" for X singletons, non-arbitrary threshold needed to define "stable").

• ++++ The Singleton, Sample and Statistic for IP Capacity are implemented.

• ---- "stable" needs more discussion, or may be resolved by Qualification below.

• @@@@ Measurements depend on the access network and the use case. Here, the use case is to assess the maximum capacity of the access network, with specific performance criteria used in the measurement.

• ++++ Covered in the Intro.

• @@@@ Goals made clearer in the next draft, if possible.

• ++++ Covered in the Intro.

• @@@@ A qualification measurement for the search result is a subsequent measurement, sending at a fixed 99.x % of the Max IP-layer Capacity for I, or an indefinite period. The same Max Capacity Metric is applied, and the Qualification for the result is a sample without packet loss or a growing minimum delay trend in subsequent singletons (or each dt of the measurement interval, I). Samples exhibiting losses or increasing queue occupation require a repeated search and/or test at reduced fixed sender rate for qualification.

• Here, as with any Active Capacity test, the test duration must be kept short. 10 second tests for each direction of transmission are common today. In combination with a fast search method and user-network coordination, the concerns raised in [RFC 6815] are alleviated.

• +++ covered in the method of measurement section, subsection on Measurement Qualification and Verification

https://tools.ietf.org/html/draft-morton-ippm-capacity-metric-method-00

October List Discussion: Matt, Rüdiger, acm (1)

• Summary: Matt is saying (? Subject to confirmation)• @@@@@ RTT is a good singleton measurement interval (dt) to avoid

“bursts & silence”

• Use windowed Max of max_rate from BBR (but see our measurements)

• Rüdiger: “Len and acm meas. results show convergence to an LTE receiver bandwidth meas. with limited queuing and no drops.”• Defaults of dt = 1 second, Δt = 10 sec

• udpst tool sends feedback measurement at regular intervals = 50 ms

• acm thinking: sub-second rate meas. are more susceptible to the cases described by Matt, and by Joachim Fabini (time-slot service with full link capacity play-out of the queue: LTE, others).

• acm: But no assessment of loss with BBR, QUIC encrypt & aggregates

++++ We’ve added the defaults above with parameters when they appear, and more discussion in section 8.2

++++ Considerations for testing with parallel flows (sec 8).

++++ Default for the Sending rate measurement interval (sec 7, 0.05 s)

https://tools.ietf.org/html/draft-morton-ippm-capacity-metric-method-01

October List Discussion: Matt and Rüdiger• It is fairly normal to see packets arrive in back to back packet trains,

separated by periods of silence. Half- Duplex, Pkt Aggregation, …

• MM: simplistic meas. of LTE receive rates often see modes at 1Gb/s.

• BBRv2 uses rate measurement per RTT:• rtt_sample = delta(timestamp) # 1 RTT

• rate_sample = delta(total data ACKed)/rtt_sample # one RTT's worth of data

• Effectively: Capacity(t, Δt, n, <no PM>) = n0[dtn-1,dtn]/(dt = RTTn)

• min_rtt and max_rate (used by BBR congestion control) are the windowed (?) max and min of rtt_sample and rate_sample above

• MM: I predict that max of BBR's max_rate will be a more robust and more accurate measure of the short duration maximum rate than anything you can do with UDP (except perhaps QUIC, BBR over UDP).