Multiscale Traffic Processing Techniques for Network Inference and Control

Richard Baraniuk Edward Knightly Robert Nowak Rolf Riedi

Rice University INCITE ProjectApril 2001

Rice University | INCITE.rice.edu | April 2001

INCITEInterNet Control and Inference Tools at the Edge

• Overall Objective:

Scalable, edge-based tools for on-line network analysis, modeling, and measurement

• Theme for DARPA NMS Research:

Multiscale traffic analysis, modeling, and processing via multifractals

• Expertise:

Statistical signal processing, mathematics, network QoS

Technical Challenges

Poor understanding of origins of complex network dynamics

Lack of adequate modeling techniques for network dynamics

Internal network inaccessible

Need: Manageable, reduced-complexity models with characterizable accuracy

Multiscale modeling

Multiscale Analysis

Analysis: flow up the tree by adding

Start at bottom with trace itself

Multiscale statistics

Multiscale Synthesis

Synthesis: flow down via innovations

Start at top with total arrival

Signal: bottom nodes

Multiscale parameters

Multifractal Wavelet Model (MWM)

• Random multiplicativeinnovations Aj,k on [0,1]

eg: beta

• Parsimonious modeling(one parameter per scale)

• Strong ties with rich theory of multifractals

Multiscale Traffic Trace Matching

Auckland 2000 MWM matchscale

Multiscale Queuing

Probing the Network

Probing

• Ideally:

delay spread of packet pair spaced by T sec

correlates with

cross-traffic volume at time-scale T

Probing Uncertainty Principle

• Should not allow queue to empty between probe packets

• Small T for accurate measurements– but probe traffic would disturb

cross-traffic (and overflow bottleneck buffer!)

• Larger T leads to measurement uncertainties– queue could empty between probes

• To the rescue: model-based inference

Multifractal Cross-Traffic Inference

• Model bursty cross-traffic using MWM

Efficient Probing: Packet Chirps

• MWM tree inspires geometric chirp probe• MLE estimates of cross-traffic at multiple scales

Chirp Probe Cross-Traffic Inference

ns-2 Simulation

• Inference improves with increased utilization

Low utilization (39%) High utilization (65%)

ns-2 Simulation (Adaptivity)

• Inference improves as MWM parameters adapt

MWM parameters Inferred x-traffic

Adaptivity (MWM Cross-Traffic)

Eg: Route changes

Comparing Probing schemes

Comparing probing schemes

• `Classical’: Bandwidth estimation by packet pairs and trains

• Novel: Traffic estimation, probing best by Uniform? Poisson? Chirp?

Model based Probing

Chirp: model based, superior

Uniform: Uncertainty increases error

Impact of Probing on Performance

Heavy probing - reduces bandwidth - increases loss - inflicts time-outs

NS-simulation: Same `web-traffic’ with variable probing rates

Influence of probing rate on error

• Chirp probing performing uniformly good• Uniform requires higher rates to perform

Synergies

• SAIC (Warren): MWM code for real time simulator

• SLAC (Cottrell, Feng):Modify PingER for chirp-probingHigh performance networks

• Demo: C-code for real world chirp-probingusing NetDyn (TCP) + simple Daemon at receiver(INRIA France, UFMG Brazil, Michigan State)

INCITE: Near-term / Ongoing

Verification with real Internet experiments– Rice testbed (practical issues)

– SAIC (real time algorithms) – SLAC / ESNet (real world verification)

Enhancements: rigorous statistical error analysis deal with random losses multiple bottleneck queues (see demo)

passive monitoring (novel models)

closed loop paths/feedback (ns-simulation)

INCITE: Longer-Term Goals

• New traffic models, inference algorithms– theory, simulation, real implementation

• Applications to Control, QoS, Network Meltdown early warning

• Leverage from our other projects– ATR program (DARPA, ONR, ARO)

– RENE (Rice Everywhere Network:NSF)

– NSF ITR– DoE

Stationary multifractals

Stationary multiplicative models

j(s): stationary, indep., E[j(s)]=1

• A(t) = lim 0t 1(s) 2(s)… n(s) ds

– May degenerate (compare: MWM is conservative)– stationary increments

• Assume j(2j s) are i.i.d.; Renewal reward

– Compare MWM: j(2j s) constant over [k,k+1]

– If Var()<1: Convergence in L2 ; E[A(t)]=t

– Multifractal function: T(q)=q-log2E[q]

Simulation

• L2 criterion for convergence translates to

T(2)>0

• Conjecture: For q>1 converge in Lq if T(q)>0

Thus non-degenerate iff T’(1)>0, ie E[ log ( /2) ] >0

Parameter estimation

• No conservation: can’t isolate multipliers• Possible correlation within multipliers

• IID values:

– Z(s) = log [ 1(s) 2(s)… n(s) ]

– Cov(Z(t)Z(t+s))= i=1..n exp(-is)Var i(s)

• `LRD-scaling’ at medium scales, but SRD. Multifractal subordination -> true LRD.

INCITE.rice.edu

Multiscale Traffic Processing Techniques for Network Inference and Control

Documents

Coupling Methods for Multiscale Simulations and Adaptive · PDF file · 1980-01-01Calibration and validation of the coarse-grained models will be based on Bayesian inference.

Strategic Multiscale

Magnetospheric Multiscale MissionMagnetospheric Multiscale

INCITE: Edge-based Traffic Processing and Inference for ...€¦ · falls in proportion to the introduced cross-traffic. Our paper on pathChirp received the Best Student Paper Award

City-wide Traffic Volume Inference with Loop Detector Data ...lusu/papers/SIGSPATIAL2017.pdf · City-wide Traffic Volume Inference with Loop Detector Data and Taxi Trajectories Chuishi

Demographics Inference Through Wi-Fi Network Traffic Analysis · inference technique to predict user demographics. Our analytical results show that, for unencrypted trafﬁc, DIP

Multiscale Designer

Multiscale Modelling

Point-Based Multiscale Surface Representationfaculty.cs.tamu.edu/schaefer/teaching/689_Fall2006/MultiScale... · Point-Based Multiscale Surface Representation • 179 complex models

Multiscale Retinex

Cost-Sensitive Top-down/Bottom-up Inference for …sczhu/papers/Conf_2012/Scheduling...Cost-Sensitive Top-down/Bottom-up Inference for Multiscale Activity Recognition Mohamed R. Amer1,

Machine Learning for Generalized Multiscale Modeling · Machine Learning for Generalized Multiscale Modeling Oliver K. Ernst Multiscale Modeling ML for Model Reduction ML for Multiscale

Multiscale Modeling

Multiscale Composites

Automatic Inference of Anomalous Events from (California) Traffic Patterns

Real-time Traffic Pattern Analysis and Inference with ... · Real-time Trafﬁc Pattern Analysis and Inference with Sparse Video Surveillance Information Yang Wang1;2, Yiwei Xiao2,

Multiscale quantum/atomistic coupling using … · Multiscale quantum/atomistic coupling using constrained density functional ... Multiscale coupling ... The method is applied to

Deep-Q: Traffic-driven QoS Inference using Deep Generative ... · Deep-Q: Traffic-driven QoS Inference using Deep Generative Network Shihan Xiao, Dongdong He, Zhibo Gong Network Technology

Multiscale Traffic Processing Techniques for Network Inference and Control

Multiscale Damage