1 INTRODUCTION1 Vegas’s performance in mobile ad hoc networks is

better than other TCP versions [1][2][3] at three aspects. Firstly, more precise RTT (round trip time) is prepared for the later prediction of throughput. Secondly, Vegas halves the congestion window (cwnd) size by identifying if the retransmission packets belong to the right now stage of congestion control. Thirdly, it emphasizes packet delay rather than packet loss as the criterion to determine transmission rate. However, it only calculates the expected throughput through the RTT of TCP layer. This scheme cannot reflect the real throughput of whole networks without consideration of parameters of lower layers. Besides, Vegas changes its cwnd right now based on network situation of previous time step. It means that this mechanism belongs to one of time lag predictions which are not accurate and adaptable.

How to cope with the potential problems which affect the performance of Vegas is currently attracting considerable interest from research community. Examples of such works include cwnd threshold[4], parameters of congestion avoidance phase such as alpha and beta[5], cwnd growth rate etc. But this kind of improved method has not shown good enough in performance without total consideration of dynamic characteristic of networks. In [6], the authors proposed that more potential data can be retrieved from lower layers for network condition estimation. In [7], the authors presented a novel model of TCP throughput in New Reno, but it did not use the parameters of lower layers to acquire the more accurate network conditions. In [8], the authors achieved cross layer model of TCP throughput which combined the classical throughput model and cross layer to estimate the bandwidth. In [9], the authors detected the idle time of

This work is supported by National Nature Science Foundation under

Grant of No. 61379005

channel to estimate available bandwidth. In [10], the authors applied the Q-Learning mechanism to congestion control of TCP Reno successfully.

To the best of our knowledge, the above paper gives little attention on how to improve the whole network performance by future prediction of throughput. Thus, ICATCP is proposed to deal with the problem of real achievable throughput of whole network and online congestion control. The remainder of the article is organized as follows. The main idea of the improved mechanism is shown in section 2. Section 3 formulates the cross layer model of throughput. Section 4 introduces the throughput prediction based on grey predictor. Section 5 describes the exploration of more reasonable cwnd changes based on Q-Learning. Section 6 numerically examines the ICATCP’s performance along with the other TCP protocols. Section 7 concludes the article. 2 ICATCP FORMULATION

Vegas uses the difference value (diff) by which (1)presented between expected (v_expect_) and actual (v_actual_) throughput to determine cwnd changes in the stage of congestion avoidance. Meanwhile, the congestion control mechanism is indicated by (2), where _v α and

_v β are initialed as 1 and 3 at the Vegas[2][3],respectively. WindowSize SentDatadiff

BaseRTT ActualRTT= − (1)

1, _1, _

,

cwnd diff vcwnd cwnd diff v

unchanged other

αβ

+ <= − > (2)

In order to utilize the available bandwidth of environment and make the right decision of congestion control, the ICATCP mechanism illustrated by Fig 1 which consists of three parts. They are throughput model, grey predictor and Q-Learning, respectively. The cross layer throughput model calculates the expected available throughput of environment. The grey predictor is used to

An improved congestion avoidance control model for TCP Vegas based on Ad Hoc networks

Ying Luo1, Minyong Yin2, Hong Jiang1, Shaoliang Ma2 1. College of Information Engineering, Southwest University of Science and Technology, Mianyang 621010, China

E-mail: YingLuo612@163.com, jianghong@swust.edu.cn

2. Institute of Computer Application, Academy of Engineering Physics, Mianyang 621900 China E-mail: yinmy@caep.cn, masl@caep.cn

Abstract: The congestion control of TCP Vegas is more stable and brilliant than other TCP versions such as Westwood Plus because of the congestion control before collision in Ad Hoc networks with frequent topology changes. Nevertheless, Vegas has less competitive ability when its data flow coexists with other protocols’. In addition, it cannot make full advantage of available bandwidth to transmit packets. The research aims to propose an improved TCP Vegas, ICATCP, which has three enhanced aspects in congestion avoidance phases. . The lower layers’ parameters will be considered in throughput model to improve the accuracy of theoretical throughput. . Forward throughput prediction mechanism with revision based on Grey Prediction is used to promote the online cwnd control. . The optimal exploring mechanism based on Q-Learning is applied to search the more reasonable changing size of congestion window. The simulation results show that the ICATCP has higher throughput, lower delay, more fair allocation of bandwidth than Vegas and Westwood Plus in multi-hops Ad Hoc scenarios. Key Words: ICATCP; Cross throughput model; Grey Prediction; Q-Learning; Ad Hoc network

2310978-1-4799-3708-0/14/$31.00 c©2014 IEEE

predict the expected throughput of next time step of environment. In addition, the difference between the output of throughput model and prediction can be corrected. The Q-Learning explores the more reasonable changing size of cwnd through the feedbacks.

Fig 1 The main idea of ICATCP

3 CROSS-LAYER MODEL OF TCP THROUGHPUT For simplicity of the model analysis, let us assume

that TL(Transport) layer uses TCP protocol and the MCU(Micro Control Unit) of every node of network is identical which means that the process time of every packet in APP(Application Layer) is the same as PHY(Physical Layer) and the LLC(Logical Link Control) layer. Meanwhile, the IP protocol is a best-effort service which has less useful feedbacks. Therefore, the main consideration of the throughput model is the TL(Transport Layer) and MAC(Media Access Control) sub-layer. 3.1 TCP throughput model

The throughput model is shown by (3)[7], Where N and other variables are explained by (4) and Tab 1.

( )

2

2

11

Th = 1 2 4 1 log

4

peak slr

peak slr

peakTO

CW pp CW p

CWN RTT p p p RTO RTT

×+

+ ×

× + + + × + + ×(3)

( ) ( )31 1

2peak

peak slr

CWN pTO CW p

+= + − ⋅ + ⋅ (4)

Tab 1 parameters of throughput model

Elements Definition P Loss event rate

CWpeak The average peak size of cwnd pslr Segment loss rate

RTT Average of Round Trip Timeout pTO Probability of Timeout state

RTO The first timeout of a series of timeouts

3.2 MAC sub-layer As mentioned by introduction of section 3,

Distributed Coordination Function-DCF of MAC layer will firstly be considered in lower layer protocols. DCF[11] scheme with RTS/CTS is shown by Fig 2. Meanwhile, ptDCF and dpDCF are assumed to be the process time and delivery probability of one packet, respectively.

Fig 2 The access mechanism of DCF with RTS/CTS

3DCF RTS CTS macACKpt DIFS SIFS t t t= + ⋅ + + + (5) The probability of dpDCF is given by (6)[8], where

ncdf indicates the number of competition channel data flow. _D τ denotes the probability of node requests to send data in a randomly chosen slot time. BER, dl, dh and

CWI represent the bit error rate, packet size, length of packet header and initial size of cwnd, respectively.

1(1 _ ) (1 )ncdf dl dhDCFdp D BERτ − += − ⋅ − (6)

2_1I

DCW

τ =+ (7)

The packet would be dropped immediately when the transmission loss reaches the retry attempt limit rl. So, the loss probability of one packet is defined by (8).

( )1 rldropDCF DCFlp dp= − (8)

3.3 Queue management model The other consideration of lower layer protocol is

queue management which is one of important ways of congestion control. Assume that RED (Random Early Discard) is used as queue management mechanism. The loss probability and the process time of successfully transmitted packet at RED stage are defined as lpqueue and ptqueue, respectively. ptqueue is available in real time via the feedbacks of RED protocol. lpqueue is shown in (9). The queuet

min and queuetmax are minimal and maximal size of

queue in every congestion control stage at t time. queuet+1

avg indicates the average queue length at t+1 time.

min1

max1

min1

max min

0 ,

1 ,

,

avgt tavg

queue t t

avgt t

t t

queue queuelp queue queue

queue queue otherwisequeue queue

+

+

+

≤

= ≥

−−

(9)

3.4 The full throughput model of ICATCP With consideration of MAC layer and queue

management, the full throughput model of ICATCP is shown by (10). The p* and RTT* are defined by (11) and (12). The lpTCP_ACK and ptTCP_ACK are the drop probability and process time of TCP ACK. Both of them can be detected in real time via feedbacks of TCP protocol.

( )

2

*

* * *2 *

11

= 1 2 4 1 log

4

peak slr

peak slr

peakTO

CW pp CW p

NV_tCW

N RTT p p p RTO RTT

×+

+ ×

× + + + × + + ×(10)

*_dropDCF queue TCP ACKP lp lp lp= + + (11)

( ) ( )( )

*

_ _

1 1

1

dropDCF DCF queue queue

TCP ACK TCP ACK

RTT lp pt lp pt

lp pt

= − ⋅ + − ⋅

+ − ⋅ (12)

4 GREY PREDICTION MODEL Vegas utilizes the parameters of t time to determine

the congestion control policy at t+1 stage of congestion avoidance which belongs to one of time-delay predictions. As a result, this mechanism cannot adapt to the dynamic topology networks with inaccurate online decisions. Therefore, a forward throughput prediction algorithm is proposed to cope with the real-time control and unpredictable matter of network under congestion avoidance phase based on grey prediction [12]. 4.1 The grey predictor of throughput

In grey theory, the color represents the known degree of data. The deeper the color, the bigger the data uncertainty[12]. The basic model of grey prediction (GM(1,1)) is shown in Fig 3, whose main purpose is to generate a differential function which could output the

2014 26th Chinese Control and Decision Conference (CCDC) 2311

prediction value of next time. The initial data are generated as regular data column by one time accumulated generating operation (AGO) which can eliminate the randomness and volatility of original data. And the MEAN is just the middle column for the purpose of generating the differential function of grey prediction.

The predicted throughput of t+1 time is used to determine what actions the cwnd could be taken, let us assume that the input of grey predictor is NV_GPto = {NV_t(1),NV_t(2),…,NV_t(gn)}(gn 5) which are outputted by TCP throughput model in section 3.

( )_ zNV GPt gn is white background value. (16) is the white function of GM(1,1) for NV_GPto, where

( )_ 1o

NV GP t gn∧

+ is the throughput of prediction, GPa is the evolution parameter, GPb represents the grey action. In order to satisfy the standards of grey prediction,

oNV_GPtσ , stepwise ratio of NV_GPt, is contained within

[exp(-2/gn+1),exp(2/gn+1)] which included within [0.1353,7.389] [12].

o

NV_GP t (k)∧

Fig 3 Basic model of Grey Predictor (GM(1,1))

( ) ( ) [ ]( )1

k=1_ _ 1,

moNV GPt NV GPt m gnm k= ∈ (13)

( )o

ogn-1oNV_GPt

gn

NV_GPtgn =

NV_GPt (14)

( ) ( ) ( )( ) [ ]( )1 1_ _ / 2 2,_ 1zNV GPt NV GPt m NVm m nGPt m g= + ∈− (15)

( ) ( )( ) ( )1_ _ 11 -o o GPa gn GPaGPb e eGPaNV GPt gn NV GPt∧

− ⋅ ⋅+ = − ⋅ (16)

( ) ( ) ( ) ( )( )

( ) ( )( ) ( )

( ) ( )( ) ( ) ( )

2

2 2 2

22

2 2

2

2 2 2

_ _ 1 _

1 _ _

_ _ _ _

z o z

z z

o z

gn gn gn

m m m

gn gn

m m

gn gz z

n gn

m m m

NV GPt m NV GPt m n NV GPt mGPa

n NV GPt m NV GPt m

NV GPt m NV GPt m NV GPt m NV GPt mGPb

= = =

= =

= = =

⋅ − − ⋅

=

− ⋅ −

⋅ − ⋅

=( )( )

( ) ( )( ) ( )

2

22

2 2

_

1 _ _

gn

m

gn

o

zg

zn

m m

NV GPt m

n NV GPt m NV GPt m

=

= =

⋅

− ⋅ −

(17) 4.2 Grey predictor with correction

In prediction theory, correction is one essential way to improve the accuracy of predictor. Therefore, the difference, which is defined by (18), between the output of predictor and throughput model needs to be predicted by the same way as section 4.1. The output of error correction is named as ( )_ 1NV GPt gnε

∧+ .

( ) ( )_

_ 11

_ __ _

o

NV GPt

NV GP t gn v actualgn

v actualε

∧

++

−= (18)

As mentioned by 4.1 and 4.2, the output of the Grey Predictor of throughput will be defined by (19). So, the diff will be rewritten as (20).

( ) ( ) ( )_1 _ 1 1oNV GPtGPt gn NV GP t gn gnε

∧ ∧= + ++ + (19)

( )1 SentDaGP tat gndiffActualRTT

+= − (20)

5 CONGESTION AVOIDANCE MODEL When application transmit quantitative data under

the limited bandwidth, the less the propagation delay, the

more packets can be sent. On the contrary, if the more the transmission time consumed, the less the bandwidth can be used. Therefore, we try to make full use of available bandwidth and follow Vegas’s conditions and principles synchronously by combining quantification with reinforcement learning in this section. 5.1 Vegas RTT quantization

As discussed above, the RTT will be quantified to different degree as the value of cwnd changes. The function is given by (21), where minRTT and maxRTT are the minimal and maximal RTT at each congestion control stage. The q_MRTT represents the interval of minRTT and maxRTT. b is the baseline which will lead the different horizontal baseline for (21). The is the slope of function. The detailed relationship of function members is graphically depicted in Fig 4.

( )

( )( )

( )

2

+1, min

minsin

2 _

2 minq(RTT) = ,min max

mincos

2 _-

2 min

, max

b RTT RTT

RTT RTTq MRTT

RTT RTTb RTT RTT RTT

RTT RTTq MRTT

RTT RTT

b RTT RTT

πα

πα

≤

−⋅ ⋅

⋅ −+ < ≤

−⋅ ⋅

⋅ −

<

(21)

Fig 4 Function relationship of q(RTT)

Fig 4 illustrates that the different b and α will lead the different baseline and slope of function which could be chosen to different environmental conditions. The , which contains in the range of [0.3,0.8]. In the article, α is set to be 0.5 which is the better choice for function. In order to improve the accuracy of quantization of RTT, the phase of congestion avoidance is formulated as Markov Decision Process(MDP) to explore the best horizontal baseline. 5.2 Q-learning exploration model

Q-Learning [13] is one of free-model Markov process which directly uses dynamic programming to estimate the utility function value of optimal strategy. The different stages of cwnd changes are set as the state Si. From the formula (21), the baseline of function will make cwnd to take different actions for environment based on RTT. Therefore, the baseline factor b is defined as the set of action Ai. The basis for selection of b must meet the conditions that the value of b+1 cannot exceed -1,1 and 0 too much. So, in this article, we set different baseline at the different congestion control status for the optimal action exploitation. In order to weigh the influence of action for the environment, the expected throughput is as to be the set of return value (Ri(s,ai)). In the end, the purpose of the Q-Learning exploration is to choose the

2312 2014 26th Chinese Control and Decision Conference (CCDC)

best action which could maximize the formula (22). ( ]0,1qγ ∈ is the discount factor which represents the

influence of the future return to the current. tα represents the learning rate at the pair of <st,at>. Q converges to Q* with probability 1[13] when the learning rate tα meets the

condition ( ) ( )( )2

, , , ,1 1,

t t t tt i s a t i s ai iα α∞ ∞

= == ∞ < ∞ , where t(i,st,at) is

assumed that the ith iteration of the at applied to st. The corresponding optimal strategy of st is defined by

( ) ( )* = argmaxt

*t t

ss Q s ,aπ .

( ) [ ] ( ) ( ) ( )1 1 1 1, 1 , + R max ,t t t t t t t t t t ta

Q s a Q s a q Q s aα α γ+ + + += − ⋅ ⋅ + ⋅ (22)

For the purpose of avoiding local optimal solution, greedyε − is utilized to balance the exploration and

utilization of the Q-learning. In this strategy, the optimal action will be selected with probability of ε . 6 SIMULATION

We implement ICATCP in NS2 and verify its performance along with other TCP by different scenarios. All of the scenarios have the same MAC layer parameters which are given by Tab 2. The learning factor qγ and learning rate tα are equal to 0.95 and 0.3, respectively. In addition, ε is set to be 0.8. Every flow is set between node 4 and 5 from the beginning to the end in simulation. The simulation results are obtained by averaging the results of 10 simulation runs.

Tab 2 MAC sub-layer parameters Parameters Value

Minimize of cwnd size 31 Maximize of cwnd size 1023

Slot time 20us SIFS 10us

Data length 144bit Header length 48bit

Short retry limit 7 Long retry limit 4

6.1 Multi-hop linear wireless network There are six nodes which have the identical

transmission range of 150 meters in this scenario. Positions of those nodes are indicated by Fig 5. The simulation seconds is set to be 270. During the simulation, all nodes’ velocity is randomly chosen in (0,10]m/s. Besides, all of them keep linear queue.

Fig 5 Positions information of nodes

Fig 6 Simulation results of multi-hop linear wireless network Simulation results are illustrated by Fig 6, we can

see that the performance of ICATCP is more excellent than Vegas and Westwood Plus in throughput and propagation delay. At the beginning of simulation, ICATCP is not stable because the perception of environment is not enough for Q-learning and Grey predictor. But that shortly afterwards, the property of throughput and delay are steady and perfect than Vegas and Westwood Plus through exploration of dozen of seconds. Because ICATCP has the ability of congestion prediction which means it foresees the possible environment conditions from the previous throughput data. Therefore, ICATCP make the TCP has the ability of self-learning to determine what actions should be taken when it faces different circumstances. Therefore, the average increase rate of throughput is about 59% and 64% by comparing with Vegas and Westwood. The average decrease rate of propagation delay is about 25% by comparing with Vegas and is far more than Westwood plus. 6.2 Random multi-hops ad hoc scenario

In this scenario, there are 100 nodes are setting in a square of 400*400 meters. Every node has the same propagation distance of 50 meters. The simulation seconds is set to be 600. During the simulation, all nodes moved around randomly at the rate of no more than 10m/s. The average results of simulation are shown in Fig 7.

Fig 7 Simulation results of multi-hop ad hoc wireless network From Fig 7, the simulation results show that the

performance of ICATCP is not stable because of incompleteness of environmental awareness. But, the performance of Vegas and ICATCP is obviously better than Westwood plus gradually both in the fields of average throughput as well as delay with the prediction of the trend of possible environment changes. Besides, the average throughput of ICATCP increases over one time than Vegas. And, its delay decreases 55% than Vegas on average. 6.3 Fairness analysis

Fairness is another crucial indicator of network performance. Thus, two TCP flows are set between node 4 and 5 based on the scenario of section 6.1. The first flow is the Vegas or ICATCP flow, and the other is the Westwood. The fairness mechanism of Jain’s index is indicated by (23)[14] , where n is the sum of data flows. The pfi, shown by Fig 8, represents the percentage of resource allocated to the i data flow. The percentage of fairness is shown in Fig 9.

( )2

2

1 1

n n

i im m

f X pf n pf= =

= ⋅ (23)

2014 26th Chinese Control and Decision Conference (CCDC) 2313

Fig 8 Percentage of bandwidth utilization

Fig 9 Fairness comparison of ICATCP and Vegas

Fig 8 shows the percentage of bandwidth utilization of two different TCP flows. From it, the results show that Westwood plus data flow always takes up Vegas’s bandwidth to transmit packets in whole simulation. On the contrary, the improved TCP version, ICATCP, is not showing obvious competiveness and coexistence with Westwood plus at the beginning because the perception of environment is not enough. However, ICATCP makes the fairness of environment more than 82% on average from Fig 9 because of the gradual circumstance cognizance of reinforcement learning and prediction mechanism, ICATCP can take proper actions before possible collision or congestion. As a result, the average fairness of environment in Jain’s index shows that ICATCP is better than Vegas about 23%. 7 CONCLUSIONS

The study and improvement of Vegas performance such as throughput, transmission delay, etc. are important in dynamic wireless networks. The article uses the cross layer throughput model to calculate maximum achievable throughput and predicts throughput of the next stage based on gray theory for real-time congestion control. The optimal action exploration under Q-Learning is to look for best quantization of RTT at congestion avoidance phases. Simulation results show that the performance of our proposed scheme highly outperforms than Vegas in terms of average throughput, delay and fairness under multi hop Ad Hoc environments. REFERENCE [1] F. U. Rashid, J. Singh, A. Panwar, M. Kumar, congestion

control Analysis over Wireless Ad Hoc Networks, International Journal of Engineering Research & Technology, Vol. 2, No. 5, 462- 465, 2013.

[2] M. Jehan, Dr. G. Radhamani, T. Kalakumari, VEGAS: Better Performance Than Other TCP congestion control Algorithms on MANETs, International Journal of Computer Networks(IJCN), Vol. 3, 155-158, 2011.

[3] K. Tsiknas, G. Stamatelos, Performance Evaluation of TCP in IEEE 802.16 Networks, IEEE Wireless Communications

and Networking Conference: Mobile and Wireless Networks, Vol. 2, No. 5, 2951-2955, 2012.

[4] R. S. Cheng, C. Y. Ke, An threshold-based congestion control mechanism for Vegas TCP over heterogeneous wireless networks, 2011 6th International Symposium on Wireless and Pervasive Computing conference, 1-6, 2011.

[5] G. A. Abed, M. Ismail, K. Jumari, Influence of Parameters Variation of TCP-Vegas in Performance of Congestion window over Large Bandwidth-Delay Networks, 2011 17th Asia-Pacific Conference on Communicaions,434-438,2011.

[6] H. H. Xie, R. W. Pazzi, A.e Boukerche, A Novel Cross Layer TCP Optimization Protocol over Wireless Networks by Markov Decision Process, Wireless Networking Symposium conference, 5723-5728, 2012.

[7] N. Parvez, A. Mahanti, Carey Williamson, An Analytic Throughput Model of TCP NewReno, IEEE/ACM Transactions on Networking, Vol. 18, No. 2, 448-461,2010.

[8] Z. H. Yuan, H. Venkataraman, GM. Muntean, A Novel Bandwidth Estimation Algorithm for IEEE 802.11 TCP Data Transmissions, WCNC 2012 Workshop on Wireless Vehicular Communications and Networks,377-382,2012.

[9] R. Belbachir, M. M. Zoulikha, A. Kies, B. Cousin, Bandwidth Reservation in Mobile Adhoc Networks, IEEE Wireless Communication and Networking Conference: Mobile and Wireless Networks, 2608-2613,2012.

[10] V. Badarla, C. S. R. Murthy. Learning-TCP: A Stochastic Approach for Efficient Update in TCP Congestion Window in Ad Hoc Wireless Networks, J. ParallelDistrib. Compute, Vol. 71, 863–878, 2011.

[11] B. Nithya, C. Mala*, V. K. B. Simulation and Performance Analysis of Various IEEE 802.11 Backoff Algorithms, The 2th International Conference on Communication, Computing & Security, 840 -847, 2012.

[12] J. Lin, RJ. Lian. Design of a Grey-Prediction Self Organizing Fuzzy Controller for Active Suspension Systems, Applied Soft Computing, Vol. 13, No. 10, 4162-4173, October 2013.

[13] T. M. Mitchell. Machine Learning, McGraw-Hill Science/Engineering/Math. March 1, 1997.

[14] H. SHI, R. V. Prasad, E. Onur, I.G.M.M. Niemegeers. Fairness in Wireless Networks: Issues, Measures and Challenges, IEEE Communications Surveys & Tutorials conference, Accepted for publication, 2013.

2314 2014 26th Chinese Control and Decision Conference (CCDC)