Software Defined Privacy-Preserving Measurement Instrument and Services

Yan Luo, Univ. of Massachusetts LowellCody Bumgardner, Univ. of KentuckyGabriel Ghinita, Univ. of Massachusetts BostonMichael McGarry, Univ. of Texas El Paso

Overview of IRNC AMIS ProjectDesign of IRNC AMIS Instrumentation Framework• Overview• AMIS measurement functions• Plugin management• Privacy preservation• Data analytics interface

Q&A

Outline

2IRNCAMIS,Internet2TechX,2016

Objectives• 40+Gbps flow-granularity network measurement

instrument• Software defined measurement• Preserving privacy of network flow info• In-depth flow analytics

NSF Funded Project Team: • Yan Luo, PI, University of Massachusetts Lowell• Cody Bumgardner, Co-PI, University of Kentucky• Gabriel Ghinita, Co-PI, Univ. of Massachusetts Boston• Michael McGarry, Co-PI, University of Texas El Paso

Overview of IRNC AMIS Project

3IRNCAMIS,Internet2TechX,2016

IRNC AMIS Deployment Plan

4IRNCAMIS,Internet2TechX,2016

Overview of IRNC AMIS Framework

5

PluginManagement

Web-basedNetworkData

VisualizationandAnalytics

PrivacyProtection

AMISMeasurementFunctions

UTEP1. Configureandmanage

measurementtasks2. Annotateinstrumentdata

withauxiliarydataforanalytics

3. Measurementdatavisualizationandanalytics

UKY1. Config management2. Dispactch mtask to AMIS nodes3. Query processing on netflow

records

UMB1. SyntacticPrivacy2. DifferentialPrivacywith

BigData tools

UML1. Netflow generation2. Packet tracing3. Packet validation4. Software defined

measurement5. Distributed

measurement

IRNCAMIS,Internet2TechX,2016

Current functions• Netflow

§ Generate NetFlow v5 record and push to NetFlow collector§ Analyze traffic characteristics from NetFlow records

• Packet Tracing§ Trace the occurrence of flows/packets on links monitored

by (distributed) AMIS instrument• Packet Validation

§ Validate the packets on the link (IP level validation) and report good/bad packets

Ongoing Work• Software Defined Measurement

§ P4 based protocol parser and matching table§ Measurement function composition

• Distributed and Collaborative Measurement

AMIS Measurement Functions

6IRNCAMIS,Internet2TechX,2016

AMIS Instrument Internal Architecture

7

1.NetFlow:OVS 2.PacketValidation:PCAP 3.PacketTracing:PCAP

veth0

veth1

PacketTracingCustomizedFunctions

pcap

Netflow

X86Server

Mellanox100GNIC

Packets

…

PacketValidation

veth2

veth3

OVSBridge

veth4

veth5

Controller

Policies&Rules

IRNCAMIS,Internet2TechX,2016

IRNC AMIS: Distributed Control System

Processing- Interaction DistributedControlSystem

-Project components managed through a distributed control system comprised of agents and plugins.

-Plugins implemented to manage control of data processing components.

-Agents allow for anonymous operation with centralized control.

8

Device View

9

mQueue_0

Probe

mQuery_0

Agent Controller

Central Services

Key:

-Control and Config Data

-Measurement Data

Service Module View:AMIS Device

mQuery_1mQueue_1

Measurements

Probe Agent

Local Processing

AMIS Device

lQueue_0

lQueue_1lQuery_1

Direct Client

eQueue_0

System View

10

lQueue_0

Probe Agent

Agent ControllerDashboard

/API

Query eQueue_1

Central Controller

Device 0

Key:

-Control and Config Data

-Measurement Data

lQueue_0

Probe Agent

Device 1

Remote Control and Aggregation

eQueue_0Direct Client

Privacy

OfflineModeOnlineMode

DifferentialPrivacyStrongProtectionModeratePerformanceStatisticalQueriesOnlyHighStorageRequirements

SyntacticPrivacyBest-effortProtectionFastPerformanceFlexibleQueryCapabilitiesLowStorageRequirements

AMIS: Supported Privacy Modes

11IRNCAMIS,Internet2TechX,2016

FlowWindow

FastIndexer

GeoMappingASMapping

k-anonymizerEngine

HilbertFractalsLinearRuntime

OnlinePrivacyModule

Sanitizedflows

Sensitive

Sanitized

Researcher

NetworkEngineer

Student

RabbitMQRawFlows

AMIS Privacy: Online Mode

12IRNCAMIS,Internet2TechX,2016

HBase Repository

DifferentialPrivacyEngine

HadoopAnalytics

OfflinePrivacyModule

Sensitive

Sanitized

QueryOptimizer

SanitizedQueryResults

RabbitMQRawFlows

Researcher

NetworkEngineer

Student

AMIS Privacy: Offline Mode

13IRNCAMIS,Internet2TechX,2016

Empower understanding of network exchange link utilization• who is communicating with who? what types of data are they communicating?• which institutions are utilizing the exchange links?

Empower network management (autonomous?)• FCAPS model of network management

• F – fault detection and correction– troubleshooting, monitoring, anomaly detection

• C – configuration and operation• A – accounting and billing• P – performance assessment and optimization

– performance monitoring, capacity planning

• S – security assurance and protection

Objectives for Visualization

14IRNCAMIS,Internet2TechX,2016

Most of our visualizations are driven by Netflowrecords

We annotate Netflow records with AS, geographic, and application data, Netflow+

We turn Netflow+ records into data viz objects to be rendered in the browser

Data Visualization Pipeline

15

AMISinstrument

AMISdatapipeline

URL:taskID

JSON:Ne1lowrecords

AMISNetFlowprocessing

RESTAPIURL:taskID

JSON:vizdata

RESTAPIWebserver

AMISNetFlowvisualiza<on

Webbrowser

AMISNetFlowannota<on

URL:taskID

JSON:NetFlowrecords

RESTAPIAMISdatapipeline

IRNCAMIS,Internet2TechX,2016

16IRNCAMIS,Internet2TechX,2016

17IRNCAMIS,Internet2TechX,2016

18

Thank You!

Q&A

IRNCAMIS,Internet2TechX,2016

Visualization of Anonymized Data

Hyper-RectangleView AnalyticsView

19IRNCAMIS,Internet2TechX,2016