Traffic Engineering in Cable-Data System

: Maximizing CMTS port utilization

Jun-seong Cho, Jongheon Park, Munhee Hong

NMS Development Part, Research and Development Center

Hanaro Telecom, Inc. Seoul Korea

Email: jjgoody@hanafos.comTel: +82-26266-4244

(2)NOMS 2004

Introduction• Broadband Internet Service using HFC network is widely available

• Regarding the traffic in HFC network– The advent of multimedia age has continuously increased the traffic.– Worse than that, P2P traffic has dramatically increased and the increased traffic

lasts all time while its users are on online.

• However, network interface has bandwidth limit – Network Interface(CMTS port) is shared by several users in HFC network– Possibility of some monopolizing a network resource– When monopolized, users per port decreases -> port utilization suffers– Need to control network resource monopolization

• Load balancing of Excessive User(EU) traffic can solve the problem– Mechanism finding excessive users has been introduced in my previous research

“Efficient Upstream Management in Cable-Data System“[1]– Compared to my previous research[1][2], this time it deals with not only upstream,

but also downstream, and traffic in general. – It introduces EU traffic load balancing to achieve the better port utilization, and

shows the field test result of the mechanism.

HFC > 2000 HOMESHeadend N COAXFiberC MTS

Cable Serving AreaCable Serving Area

(3)NOMS 2004

Intro on HFC Network’s Cable Data System

Core part of Cable Data System configuration are CMTS, Port, CM, Cell

※ CMTS: Cable Modem Termination System, CM: Cable Modem

C M

C M

C M

....

C MTS Devic e(ex: C isco uBR 7246)

Ports

HFCNetwork

Internet

Internet

CM

CM

CM

CELL

CMTS ONU

Node Element(tap,amplifier etc.)

Optical Fiber

Coaxial Cable

※ a cell can be provisioned to be handled by several port

(4)NOMS 2004

CMTS Port has limited bandwidth…

• Sharing the same port (bandwidth) leads to Bandwidth Contention

• Excessive user– Endangers CMTS performance (Lowers Port Utilization, leads to busy

port)– Make the line unavailable to the others who share the port

※ We define EU (Excessive User) as the user who generate more than 1 percent of the total traffic in the port where it belongs.

X

1) Let Upstream Port’s max bandwidth: 2) User A consumes the following

3) User B consumes the following

4) User C fails to use

All other users can not use!!

Upstream Port

Users sharing the same port

(5)NOMS 2004

Hot potato: EU's Monopolization

3/0 236 18,847 22 9.32% 78.30%4/0 210 15,677 17 8.10% 79.80%5/0 220 17,035 18 8.18% 76.90%6/0 192 19,229 19 9.90% 90.20%

Average 214.5 17697 19 9% 81%

EU’s trafficoccupying

ratio

downstream -port location

the number ofsubscribers

Sum of trafficgenerated by

EU(Kbps)

Number ofEUs

EU ratioamongUsers

• What’s the reality? – A CMTS has been chosen and the traffic has been monitored and the below

shows one of the outcome of the peak time– EUs who are less than 10 %(9 %) of total subscribers are taking up 81% of

total traffic. – May be fair for free competition environment to the EUs– However, if the monopolization especially in its peak time can lead to

harming resource utilization, bandwidth exhaustion, and make the other users suffer….

– What about the actual traffic trends => next slide shows a snapshot of a day

A CMTS from the HFC network in Hanaro Telecom Inc., Traffic has been monitored through ’03.3.1 14:50~’03. 3. 21 14:50’

(6)NOMS 2004

Hot potato: EU's Monopolization(2)

• 4/0 seems ok except 23:00~24:00

• 6/0 reaches the busy port level almost half the day

• The user

complaints in 6/0 have been piled up due to its slow network (Some are reported that they couldn’t use Internet)

• It would be nice if 4/0 can spare some of its free bandwidth to 6/0 when 6/0 becomes busy

* Traffic computed by five minute interval sampling

(7)NOMS 2004

What can we do about it?• Options

① Purchases enough network resources ② Lowers per-cell subscribers③ Engineers traffic, and minimize the

inconvenience until new resources are added.

• Option ➀ and ➁, every time when it reaches the bandwidth limit, – is not economical – makes service provider’s ROI suffer– Thus, is simply unrealistic

• ➂ Traffic Engineering sounds be nicer– if it solves afore mentioned problems.– The paper introduces a module called “ Load Control

Manager (LCM)” – LCM will monitor the traffic, and is responsible for the

traffic engineering

$$$

(8)NOMS 2004

LCM Basic Idea – Port Load Balancing

1. Detec ts busy port thru resourc e performanc e mgmt func tionality

C M

EU C M

C M

HFC Network

...

C MTS device(ex: C isco uBR 7246)

2. Monitor home- user traffic on the busy port, and filters out the EU(extreme user)s

3. C hec k wether there is a port to move the EU

4. Reloc ate the EU to another port that c an handle the traffic

Yes

Reloc ate!!

5. O ther traffic load c ontroling Managements

No

(9)NOMS 2004

LCM 1: Detection of Busy Port※ Same as the mechanism in my previous research [1]

• Everything starts from discovering busy port– If the port is not busy, there is no point of finding EUs. – Performance management module in NMS can easily find the busy port

because it has got the functionality monitoring every CMTS port performance– NMS notifies the detection of the busy port to LCM

• Defining Busy port– Well defining “busy port” condition is very important (ex: Port Utilization, Load)– The status that can endanger the service in no time.– Usually the traffic reaches near to its maximum bandwidth.

• NMS gives the following info to LCM– Busy Port Information : CMTS ID(CMTS IP), Port Index, Port Usge– CMTS Connection Info such as SNMP read community or telnet login id/password

Load C ontrol

Manager(LC M)

NMS

24 hour, global C MTS

Monitoring

NMS detec ts busy port, & notifies it to LC M

(C MTS ID, P ort, Busy rate, C onnection related Info .)

(10)NOMS 2004

LCM 2: Detection of EUs※ Same as the mechanism in my previous research [1]

• As soon as LCM are notified of the busy port from NMS…– LCM connect to the CMTS where the busy port belongs– Scan all the user upstream status on the busy port– After a monitoring interval, calculates all the users upstream speed– Finally draws EU’s CM Mac address, avg traffic that has been detected.

• How to query the traffic info from CMTS– Depends on the interface device offers– SNMP, as a standardized way of managing nw device, may be one candidate– However, using proprietary command interpreter works fast and simple if there

are only one vendor of devices.

• LCM uses the following info to detect EU– Busy Port Information : CMTS ID(CMTS IP), Port Index, Port Usage– CMTS Connection Info such as SNMP read community or telnet login id/password– EU detection rules and policies : EU upstream.downstream thresholds, Detection

period, Port busy status before and after the monitoring, etc.

LC M Monitors the Busy P ort. Detec ts EUs(C M mac , up/ down traffi c) based on EU detec tion po lic ies

EU detec tion rules & polic ies

(11)NOMS 2004

LCM 3: Port relocation• Up to now

– Busy port(port utilization, bandwidth utilization) is detected– EUs(id, traffic info, service category etc) were detected

• Who are going to give it a break?– Find the port that can handle the same cell. Find the ports which are

provisioned to handle the same cell. – Find the port amongst them, that can hold some of EUs. Ex) n EUs, helper port X For (i = 1; i < n; i++) { if (I-th EU traffic+ X’s port traffic) < busy port level traffic) transfer the I-th EU to the port X }– Relocates EUs from the previous port to the port X, Log and save the

record

• What if there’s no port to hole EUs? => Other traffic Load Controlling Methods– Still need to avoid the panic situation where a lot of ordinary users can

not use Internet – Service provider may opt to limit the EUs bandwidth rate only for the

busy time– Or, it might force many EUs to gather into a single port, so ordinary users

get no harm from them– Still in research on the other load controlling method

(12)NOMS 2004

Regarding Implementation• Setting the right thresholds makes all!

– Thresholds always becomes the bottom line of the mechanism. – Threshold can differ depending on the service provider’s

environment: Various combinations can be produced– The values have to be tested, analyzed, and carefully chosen.

(Due to confidentiality, I can not reveal the details of them, but will present its guidelines)

• Busy port related threshold– Everything starts from busy port– Port status, Port utilization, Port alarm record, Channel

Utilization

• EU threshold– Do not make everybody EU. If EUs are majority, then they are

not EU any more.– EU threshold: EU upstream / downstream thresholds,

sampling period

• Things to be solved further…– While port relocation, the EU will lose the connection

temporarily– EU traffic controlling mechanisms such as EU relocation, rate

limiting may violate the SLA.– EUs after relocated may experience slower speed than before

especially in the peak time

(13)NOMS 2004

Result – After applying LCM

6/0, After applying LCM

4/0, After applying LCM

* Traffic computed by five minute interval sampling

(14)NOMS 2004

Thanks for listening

Closing Remarks & Future Work

• However, there are things to solve…– While port relocation, the EU will lose the connection temporarily– EU traffic control mechanisms such as EU relocation, rate limiting may violate the

SLA.– EUs may experience slower speed than before esp. in the peak time

• Work to be done…– LCM and SLA relationship– Other EU traffic control mechanisms when there is no extra port to relocate– Making LCM’s EU detection rules & policies flexible (Rules are to change!)– LCM Scalability & Load Balancing– Relocation record management and linking it with QOS– Marketwise, service related traffic control mechanism

• EU’s monopolization of CMTS port can endanger the Internet service.

• Applying a Traffic Engineering scheme(LCM)…- Better Port Usage Rate: More subscribers to a CMTS por, more

revenue by resource saving- More satisfaction of home-users: Less complaint, better availability to

customers- Saving resources: Network resource + Operational human resources