1

A Reduced Dimension MDP-based Call Admission Control Scheme for Next Generation

Telecommunications

Huan Chen1, Chih-Chuan Cheng2, Wei-Ho Chung*,2, and Hsi-Hsun Yeh3

1Department of Computer Science and Engineering, National Chung-Hsing University, Tai-Chung, Taiwan2Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan3Department of Electrical Engineering, National Chung Cheng University, Chia-Yi, Taiwan

2

Outline• Introduction

– Motivation– Reduced Dimension MDP (RD-MDP) Sub-optimal CAC scheme

• Problem Formulation– System Model and Design Consideration– Markov Decision Process (MDP)– Linear Programming (LP)

• Proposed RD-MDP Framework– Stage I: determining the Coarse Resolution of the CAC Policies– Stage II: determining the Finer Resolution of the CAC Policies

• Simulation• Conclusion

3

Motivation• The next generation telecommunication system is expected to

be capable of operating in the highly integrated heterogeneous environments, e.g., 3G cellular networks, LTE, WLAN and WiMax networks.

• The next generation telecommunication system is also expected to accommodate huge amount of users and data transmissions.

• One of the major challenge is to support services with different quality of service (QoS) requirements in such a heterogeneous system.

4

Call Admission Control• There are three types of calls, such as new

calls, horizontal handoff calls and vertical handoff calls, in a heterogeneous network.

• The call admission control (CAC) plays an important role for the radio access among systems.

• One of the major challenges in the design of CAC schemes is to provide differentiated treatment among users while efficiently utilizing the system resource to fulfill handoff requirements and subscriber satisfactions.

BS

BS

BS

CAC

CAC

CAC

Horizontal handoff

Vertical handoff

5

Related Works• The CAC schemes proposed for conventional cellular networks:– Traffic model and performance analysis for cellular mobile radio telephone systems

with prioritized and nonprioritized handoff procedures, IEEE Transactions on Vehicular Technology, 1986.

– On optimal call admission control in cellular networks, IEEE Infocom, 1996.– Drawback: a static priority treatment imposes starvations on the low-priority calls in a

heavily loaded access network.

• The Markov decision process (MDP) based CAC schemes proposed for heterogeneous wireless networks:– Optimal joint session admission control in integrated WLAN and CDMA cellular

networks with vertical handoff, IEEE Transactions on Mobile Computing, 2007.– Guard-channel-based incremental and dynamic optimization (guido) on call admission

control for next generation qos-aware heterogeneous systems, IEEE Transactions on Vehicular Technology, 2008.

– Drawback: MDP-based techniques suffer from the problem of the Curse of Dimensionality.

10 30 50 70 90100000000000

100000000000001000000000000000

1E+0171E+0191E+0211E+0231E+025

System Capacity C (channels)

Com

puta

tion

Com

plex

ity (i

n lo

g sc

ale)

Exponential growth

6

Reduced Dimension MDP (RD-MDP) sub-optimal CAC scheme

• RD-MDP sub-optimal CAC scheme is designed based on two-stage MDP structure.– Stage I: the aggregated system capacity is used as

the MDP state to determine the coarse resolution of the CAC policy (with Dimensionality=1).

– Stage II: the finer resolution of the CAC policy (with Dimensionality=3) is determined only at those critical range of the aggregated system capacity.

• The overall MDP dimension is thus greatly reduced.

7

Problem Formulation-System model and design consideration• A heterogeneous network architecture comprises a

core network and multiple heterogeneous access networks, with QoS mappers in between coping with the service classes mapping.

• Since the order of priorities of new calls (n), horizontal handoff calls (hh) and vertical handoff calls (vh) is: hh (highest priority) > vh > n (lowest priority), we have the following assumptions to reflect the importance of types of handoff calls:– , where is the maximum allowed blocking probability

for the call type i.– , where is the weighting factor for the call type i.– , are the arrival rate of horizontal handoff calls,

vertical handoff calls and new calls, which are assumed to follow the Poisson process.

– , and are the departure rate of horizontal handoff calls, vertical handoff calls and new calls, which are assumed to follow the Exponential distribution.

CoreNetwork

AccessNetworks

QoSMapper

𝐶𝑒𝑓𝑓 =⌊(𝑊 /𝑅 ) (1−𝜂 )

𝜀⌋

For CDMA systems

For MIMO systems𝐶𝑒𝑓𝑓 =(𝐵𝑅 ) ∙𝑙𝑜𝑔2𝑑𝑒𝑡 (𝐼 𝑟+ 𝜌

𝑡𝐻𝐻† )

8

Markov Decision Process• The MDP model for CAC schemes is

uniquely identified by the following five components:– Decision epochs: The CAC decision are made

at the occurrence of all arrival events.– State space: x = (xn,xvh,xhh), where xi is the

number of calls for call type i, and xn+xvh+xhhC.

– Action space: ax=(an,avh,ahh), where ai is the action for call type i (=0, reject; =1, accept).

– Reward function: r(x,ax)=wnan+wvhavh +whhahh.– Transition probabilities: The transition

probability is defined as the probability for the system to change from one state to another after a period of time. The sojourn time is denoted as the expected time elapsed for the system to stay in a specific state when the action is taken, and can be expressed by (x,ax)={nan+vhavh+hhahh+nxn+vhxvh+hhahh+}-1.

xn,xvh,xhh

xn+1,xvh,xhh xn,xvh+1,xhh xn,xvh,xhh+1

xn-1,xvh,xhh xn,xvh-1,xhh xn,xvh,xhh-1

𝑎𝑛 ⋅ 𝜆𝑛 ⋅𝜏 (𝐱 ,𝑎𝐱 ) 𝑎hh⋅ 𝜆hh ⋅𝜏 (𝐱 ,𝑎𝐱 )𝑎 h𝑣 ⋅ 𝜆 h𝑣 ⋅𝜏 (𝐱 ,𝑎𝐱 )

𝑥𝑛 ⋅𝜇𝑛 ⋅𝜏 (𝐱 ,𝑎𝐱 ) 𝑥hh⋅𝜇hh⋅𝜏 (𝐱 ,𝑎𝐱 )𝑥 h𝑣 ⋅𝜇 h𝑣 ⋅𝜏 (𝐱 ,𝑎𝐱 )

(1,0,0)(2,0,0)

Linear Programming Model and the Objective Function• The MDP problem can be formulated as a linear programming (LP)

problem, in which the objective is to maximize the system reward on pricing with QoS constraints on handoff blocking probabilities.– We can use simplex method to solve the LP problem.

9

QoS constraints onhandoff blocking probabilities

the system reward on pricing

is the decision variable

10

Proposed RD-MDP

Stage I: The aim is to determine acoarse resolution of the CAC policy(with dimensionality = 1)

Stage II: The aim is to determine afiner resolution of the CAC policyonly for x=(xn,xvh,xhh) on the thresholddecision planes of s = Th1 and Th2

(with dimensionality = 3)

The Key Concept at Stage I of RD-MDP

s=0 s=1 s=Th1 s=Cs=Th2

𝜆𝑛+𝜆 h𝑣 +𝜆hh

Avg()s=1

𝑎𝑠I=(1,1,1) 𝑎𝑠

I=(0 ,1,1) 𝑎𝑠I=(0 ,0 , 1)

Determined by 1-Dimension MDP model

12

The Key Concept at Stage II of RD-MDPs=Th2s=Th1

Determined by a simulation-based PolicyTunner()

Flow chart of PolicyTuner()

13

Input:

Th1=Th2?

Case1()

Th2=C-1? Th2=C-1?

Case2() Case3() Case4()

Yes No

Yes YesNo No

Output: CAC policy: x

Four Cases in PolicyTuner()

14

Case1()// Th1 = Th2 = C-11 2 output

Case2()// Th1 = Th2 < C-1

Case3()// Th1 Th2 & Th2 = C-1

Case4()// Th1 Th2 & Th2 C-1

(1,1,1) (0,1,1) (0,0,1)

(1,1,1) (0,1,1)

(1,1,1) (0,1,1)

(0,1,1) (0,0,1)

Flow Chart of Updating Actions for Each Case

15

Using exhausted search to choose one of all possible combinations of one state on Th1 and one state on Th2 , i.e., (0,xvh,xhh), (xvh+xhh ) S={Th1,Th2}

Adjust -Action at Th1: (1,1,1) -> (0,1,1)-Action at Th2: (0,1,1) -> (0,0,1)

Update

system rewardimproved?

Done

All statesvisited

Yes

No

16

Computation Complexity

𝐴∈𝑀 𝑙𝑟× 𝑙𝑐(𝓡 )

Computation Complexity of MDP (simplex method)

The overall cost of MDP is

Computation Complexity of RD-MDPStage I: (simplex method)

Stage II:

The overall cost of RD-MDP is

(LP)

17

Simulation• Simulator: event driven simulator written in C program• Environmental Setup:

– C=64• Metrics

– System reward on pricing, which can be approximated by

• Compared methods– Complete sharing scheme (CS)– Guard channel scheme (GC), e.g., GC(Th2,Th1)– Markov decision process (MDP) is NOT included in performance comparison

• the overall cost of MDP is 4.7E21, and thus the MDP model cannot be solved within reasonable time.

Performance Comparisonin terms of System Reward

18

• Intuitively, the system reward increases as the call arrivals increase.

• Since the CS scheme does not provide preferential treatment, it performs poor when traffic is heavy.

• The thresholds of GC(62,48) and GC(63,59) are obtained at the stage I of RD-MDP scheme under heavy and light traffic conditions respectively.

• Because RD-MDP has finer resolution of the CAC policies, it gains up to 3% more system reward than GC(62,48) and GC(63,59) in the best case.

QoS Satisfaction

19

• The CS scheme does not achieve the QoS requirements when the call arrival rates are above 26 calls per min.

• Due to the fact that GC(63,59) is determined based on light traffic condition, its vertical handoff call dropping rate is not satisfied with when the call arrival rates exceed 35 calls per min.

• Only GC(62,48) and RD-MDP can guarantee the satisfaction of the QoS constraints (in terms of call dropping rates).

20

Conclusion• The MDP-based schemes suffer the “Curse of Dimensionality” problem.• We proposed a computationally efficient sub-optimal MDP-based CAC

scheme, called RD-MDP, for the heterogeneous telecommunication systems with multiple service priority classes.– RD-MDP scheme is designed based on a two stage MDP framework.

• Stage I: A coarse resolution of the CAC policy is determined using the aggregated system capacity as the system state.

• Stage II: A finer resolution of the CAC policy is derived by a simulation-based policy tuner using comprehensive system information (for all three call types) as system states, i.e., the states on the critical threshold decision planes (such as s=Th1, Th2) found in stage I.

– RD-MDP can greatly reduce the overall computation complexity from the order of O(C12) to the order of O(C4).

• The simulation results show that the proposed RD-MDP scheme surpasses other techniques (including CS scheme and various GC schemes) in terms of the system reward under the QoS constraints.

• The study has confirmed that the proposed RD-MDP scheme can provide a practical MDP-based CAC solution, which is efficiently and cost effectively operable to the heterogeneous telecommunication systems.

21

Q&A

Thank you for your attendance