-
Spectrum Management of Cognitive Radio UsingMulti-agent
Reinforcement Learning
Cheng WuNortheastern University360 Huntington Avenue
Boston, MA, [email protected]
Kaushik ChowdhuryNortheastern University360 Huntington
Avenue
Boston, MA, [email protected]
Marco Di FeliceUniversity of Bologna
Via M. Anteo Zamboni, 7Bologna, Italy
[email protected] Meleis
Northeastern University360 Huntington Avenue
Boston, MA, [email protected]
ABSTRACTWireless cognitive radio (CR) is a newly emerging
paradigmthat attempts to opportunistically transmit in licensed
fre-quencies, without aecting the pre-assigned users of thesebands.
To enable this functionality, such a radio must pre-dict its
operational parameters, such as transmit power andspectrum. These
tasks, collectively called spectrum manage-ment, is dicult to
achieve in a dynamic distributed environ-ment, in which CR users
may only take local decisions, andreact to the environmental
changes. In this paper, we intro-duce a multi-agent reinforcement
learning approach basedspectrum management. Our approach uses value
functionsto evaluate the desirability of choosing dierent
transmis-sion parameters, and enables ecient assignment of
spec-trums and transmit powers by maximizing long-term re-ward. We
then investigate various real-world scenarios, andcompare the
communication performance using dierent setsof learning parameters.
We also apply Kanerva-based func-tion approximation to improve our
approachs ability to han-dle large cognitive radio networks and
evaluate its eect oncommunication performance. We conclude that our
rein-forcement learning based spectrum management can
signif-icantly reduce the interference to the licensed users,
whilemaintaining a high probability of successful transmissions ina
cognitive radio ad hoc network.
Categories and Subject DescriptorsH.4 [Information Systems
Applications]: Miscellaneous;D.2.8 [Software Engineering]:
Metricscomplexity mea-sures, performance measures
General TermsPerformance, ExperimentationCite as: Spectrum
Management of Cognitive Radio Using Multi-agentReinforcement
Learning, C. Wu, K. Chowdhury, M. D. Felice and W.Meleis, Proc. of
9th Int. Conf. on Autonomous Agents andMultiagent Systems (AAMAS
2010), van der Hoek, Kaminka,Lesprance, Luck and Sen (eds.), May,
1014, 2010, Toronto, Canada, pp.Copyright c 2010, International
Foundation for Autonomous Agents andMultiagent Systems
(www.ifaamas.org). All rights reserved.
KeywordsReinforcement learning, Cognitive Radio, Multi-agent
Sys-tem, Function Approximation, Spectrum Management
1. INTRODUCTIONWireless spectrum is a costly resource which is
licensed
by governmental agencies to operators for long periods ofuse,
often spanning several decades. However, a large por-tion of the
assigned spectrum is used sporadically leadingto under-utilization
of valuable frequency resources. To ad-dress this critical problem,
Federal Communications Com-mission (FCC) has recently approved the
use of unlicenseddevices in licensed bands [6]. Consequently,
dynamic spec-trum access techniques are proposed to solve these
currentspectrum ineciency problems [3, 8, 12]. This new area
ofresearch foresees the development of cognitive radio (CR)networks
to further improve spectrum eciency.
The basic idea of CR networks is that the unlicensed de-vices
(also called cognitive radio users) need to vacate theband once the
licensed devices (also known as primary users)are detected. CR
networks, however, impose a great chal-lenge due to the high
uctuation in the available spectrum aswell as diverse
quality-of-service (QoS) requirements. Specif-ically in cognitive
radio ad-hoc networks, the distributedmulti-hop architecture, the
dynamic network topology, andthe time and location varying spectrum
availability are someof the key distinguishing factors. As the CR
network mustappropriately choose its transmission parameters based
onlimited environmental information, it must be able to learnfrom
its experience, and adapt its functioning. The
challengenecessitates novel design techniques that simultaneously
in-tegrate theoretical research on reinforcement learning
andmulti-agent interaction with systems level network design.
Reinforcement learning [17] is a learning paradigm, whichwas
inspired by psychological learning theory from biology[18]. Within
an environment, a learning agent attemptsto perform optimal actions
to maximize long-term rewardsachieved by interacting with the
environment. The long-term reward is the expected accumulated
reward that theagent expects to receive in the future under the
policy, whichcan be specied by a value function. The value function
isoften a look-up table that directly stores values of states.
1705
1705-1712
-
Cognitive Radio Ad Hoc Networks
PU
Licensed Band 1
Unlicensed Band
Licensed Band 2
CR Users
Primary Networks
Figure 1: The CR ad hoc architecture
The study of multi-agent systems [20] allows us to buildcomplex
systems composed of multiple interacting intelligentagents. Each
agent in the system can sense the environmentand achieve its own
local knowledge and experience. Theagent can then take select
behaviors based on local infor-mation and attempt to maximize the
global performance ofthe system. A typical multi-agent system is
decentralized,without a designated controlling agent.
A challenge that CR networks face is that the performanceof CR
networks decreases sharply as the size of network in-creases.
Cognitive radio ad-hoc networks typically have avery large number
of unlicensed and licensed users, and therange of possible
transmission parameters is wide. There istherefore a need for
algorithms to apply function approx-imation techniques to scale up
reinforcement learning forlarge-scale CR problems.
Function approximation [16] is a generalization techniquewhich
has been widely used to solve large-scale learningproblems with
huge state and action spaces [9, 7, 14]. Ingeneral, function
approximation uses examples of the desiredvalue function to
reconstruct an approximation of this func-tion and compute an
estimate of the desired value from theapproximation function.
Within the context of reinforce-ment learning, instead of using a
look-up table, functionapproximation generalizes the function
values of the statesthat have not been previously visited from
known functionvalues of its neighboring states.
In this paper, we focus on CR ad hoc networks with
de-centralized control [1] [2]. The architecture of a CR ad
hocnetwork, shown in Figure 1, can be partitioned into twogroupsof
users: the primary network and the CR networkcomponents. The
primary network is composed of primaryusers (PUs) that have a
license to operate in a certain spec-trum band. The CR network is
composed of cognitive radiousers (CR users) that share wireless
channels with licensedusers that already have an assigned
spectrum.
Under this architecture, the CR users need to
continuouslymonitor spectrum for the presence of the primary users
andrecongure the radio front-end according to the demandsand
requirements of the higher layers. This capability canbe realized,
as shown in Figure 2, by the cognitive cycle com-posed of the
following spectrum functions: (1) determining
SpectrumDecision
SpectrumSharing
RadioEnvironment
TransmittedSignal
SpectrumCharacterization
PU Detection
RF Stimuli
Spectrum Hole
Decision Request
Spectrum
Channel Capacity
Mobility
SpectrumSensing
Figure 2: The cognitive radio cycle for the CR adhoc
architecture
the portions of the spectrum currently available
(Spectrumsensing), (2) selecting the best available channel
(Spectrumdecision), (3) coordinating access to this channel with
otherusers (Spectrum sharing), and (4) eectively vacating
thechannel when a licensed user is detected (Spectrum
mobil-ity).
By leveraging reinforcement learning, the tasks of spec-trum
sensing and sharing could be undertaken eectively,wherein specic
interferer transmission patterns may be dis-tinguished from each
other. Similarly, by considering eachCR user as an agent and
exploiting multi-agent interactionalgorithms, the network can
converge to the fair sharing ofthe spectrum and fast recovery if
the spectrum is reclaimedby the PU.
This paper is organized as follows. In Section 2, we formu-late
the cognitive radio problem. In Section 3, we describeour
reinforcement learning-based cognitive radio. The ex-perimental
results of our simulation are given in Section 4.In Section 5, we
discuss a Kanerva-based function approxi-mation technique for our
reinforcement learning based cogni-tive radio. The eect of function
approximation is evaluatedin Section 5.1. We conclude the paper in
Section 6.
2. PROBLEM FORMULATIONIn this paper, we assume that our network
consists of a
collection of PUs and CR users, each of which is paired
withanother user to form transmitter-receiver pairs. We also
as-sume perfect sensing that allows each CR user to correctlyinfer
the presence of the PU if the former lies within the
PUstransmission range. Each CR user must now undertake deci-sions
about spectrum and transmission power independentlyof the others
users in the neighborhood.
A choice of spectrum by CR user i is essentially the choiceof
the frequency represented by f i F , the set of
availablefrequencies. The CR users continuously monitor the
spec-trum that they choose in each time slot. The channels cho-sen
are discrete, and a jump from any channel to another ispossible in
consecutive time slots.
The transmit power chosen by the CR user i is given byP itx. The
transmission range and interference range are re-spectively
represented by Rt and Ri. Our simulator uses
1706
-
the free-space path loss equation to calculate the
attenuatedpower incident at the receiver, denoted P jrx. Thus,
P jrx = c P itxnDi
o,
where the path loss exponent = 2. The constant
c =c2
(4fc)2,
where the the speed of light c = 3 108m/s, and fc is
thefrequency at which the transmission occurs. The transmitpower
values are real numbers in a pre-decided range, and ajump from any
given value to another is possible in consec-utive time slots.
3. REINFORCEMENT LEARNING BASEDCOGNITIVE RADIO
Reinforcement learning enables learning from feedback re-ceived
through interactions with an external environment.The classic
reinforcement learning algorithm is implementedas follows. At each
time t, the agent perceives its currentstate st S and the set of
possible actions Ast . The agentchooses an action a Ast and
receives from the environ-ment a new state st+1 and a reward rt+1.
Based on theseinteractions, the reinforcement learning agent must
developa policy : S A which maximizes the long-term rewardR =
Pt rt for Markov Decision Processes (MDPs), where
0 1 is a discounting factor for subsequent rewards.One of the
most successful reinforcement learning algo-
rithm is Q-learning [19]. This approach uses a simple
valueiteration update process. At time t, for each state st andeach
action at, the algorithm calculates an update to itsexpected
discounted reward, Q(st, at) as follows:
Q(st, at) Q(st, at) +[rt + max
aQ(st+1, at+1)Q(st, at)]
where rt is an immediate reward at time t, t(s, a) is
thelearning rate such that 0 t(s, a) 1, and is the dis-count factor
such that 0 1. Q-learning stores thestate-action values in a
table.
3.1 Application to cognitive radioIn cognitive radio network, if
we consider each cognitive
user to be an agent and the wireless network to be the ex-ternal
environment, cognitive radio can be formulated as asystem in which
communicating agents sense their environ-ment, learn, and adjust
their transmission parameters tomaximize their communication
performance. This formula-tion ts well within the context of
reinforcement learning.
Figure 3 gives an overview of how we apply reinforcementlearning
to cognitive radio. Each cognitive user acts as anagent using
reinforcement learning. These agents do spec-trum sensing and
perceive their current states, i.e., spec-trums and transmission
powers. They then make spectrumdecisions and use spectrum mobility
to choose actions, i.e.switch channels or change their power value.
Finally, theagents use spectrum sharing to transmit signals.
Throughinteraction with the radio environment, these agents
receivetransmission rewards which are used as the inputs for
thenext sensing and transmission cycle.
A state in reinforcement learning is some information thatan
agent can perceive within the environment. In RL-based
Spectrum Decision
Spectrum Mobility
Sp
ectr
um
Sh
arin
g S
pe
ctru
m S
en
sin
g
trummmmmm M
um m m mm DeD
tru
m
um
ummmmmm
umm
umm
S
um
m
m
m
m
SSSSSSe
SS
Figure 3: Multi-agent reinforcement learning basedcognitive
radio.
cognitive radio, the state of an agent is the current
spectrumand power value of its transmission. The state of the
multi-agent system includes the state of every agent. We
thereforedene the state of the system at time t, denoted st, as
st = ( sp, pw)t,
where sp is a vector of spectrums and pw is a vector of
powervalues across all agents. Here spi and pwi are the spectrumand
power value of the ith agent and spi SP and pwj PW . Normally, if
there are M spectrums and N powervalues, we can using the index to
specify these spectrumsand power values. In this way, we have SP =
{1, 2, ...,m}and PW = {1, 2, ..., n}.
An action in reinforcement learning is the behavior of anagent
at a specic time at a specic state. In RL-basedcognitive radio, an
action a allows an agent to either switchfrom its current spectrum
to a new available spectrum inSP , or switch from its current power
value to a new availablepower value in PW . Here we dene action at
at time t as
at = (k)t,
where k is a vector of actions across all agents. Here kiis the
action of the ith agent and ki {jump spectrum,jump power}.
A reward in reinforcement learning is a measure of
thedesirability of an agents action at a specic state withinthe
environment. In RL-based cognitive radio, the rewardr is closely
related to the performance of the network. Therewards for the
dierent network conditions are as follows:
CR-PU interference: If primary user (PU) activity oc-curs in the
spectrum shared by the CR user, and inthe slot same selected for
transmission, then a highpenalty of 15 is assigned. The intuitive
meaning ofthis is as follows: We permit collision among the
CRusers, though that lowers the link throughput. How-ever, the
concurrent use of the spectrum with a PU
1707
-
*
*
* !"#
"
$
$
"%&
'
Figure 4: Block diagram of the implemented simulator tool for
reinforcement learning based cognitive radio.
goes against the principle of protection of the licenseddevices,
and hence, must be strictly avoided.
Intra-CR network Collision: If a packet suers a col-lision with
another concurrent CR user transmission,then a penalty of 5 is
imposed. Collisions among theCR users lowers the link throughput,
which must beavoided. The comparatively low penalty to the CRusers
arising from intra-network collisions is aims toforce fair sharing
of the available spectrum by encour-aging the users to choose
distinct spectrum bands, ifavailable.
Channel Induced Errors: The inherent unreliability inthe
wireless channel results in signal variations calledas fading
caused by multipath reections (several dif-ferent copies of the
same signal arriving within shortperiods of each other), relative
motion between sender-receiver pair (based on the Doppler eect),
amongothers. This phenomenon results in occasional bit-ips in the
received data stream, and in this work,we use standard expressions
of bit error rate derivedunder a Rayleigh fading environment [15].
Moreover,higher received signal strength reduces errors arisingout
of background noise. Interestingly, certain spec-trum bands are
more robust to channel errors owingto the lower signal attenuation
with distance.
By preferring the spectrum bands with the lowest packetbit error
rate (and hence, packet error rate PER),the CR users reduce
re-transmissions and associatednetwork delays. The carrier to noise
power ratio at
the receiver, say j, can be calculated byP jrxN
, where
the noise power N is assumed as 1010 mW. Con-sequently, the
energy per bit Eb/N0 is given by
EbN0
=P jrxN Bw
R. From this, the probability of bit error P jb at
the receiver j can be derived for QPSK modulation as
Qq
2 EbN0
[13], where the channel bandwidth Bw
is 22MHz, and the bit rate R is 2Mbps consideringa channel
structure similar to the one dened in theIEEE 802.11b specication
[10]. Q is the Q-functionthat can be expressed in terms of the
error function erf
as Q = 12
1 erf( x
2). Furthermore, assuming each
packet transmitted by the sender is bits in length, wecan
calculate the probability of unsuccessful transmis-sion by the
packet error rate or PER = 1 (1P jb ).
Link Disconnection: If the received power (P jrx) is lessthan
the threshold of the receiver Prth (here, assumedas 85 dBm), then
all the packets are dropped, andwe impose a steep penalty of 20.
Thus, the sendershould quickly increase its choice of transmit
power sothat the link can be re-established.
Successful Transmission: If none of the above condi-tions are
observed to be true in the given transmissionslot, then packet is
successfully transmitted from thesender to receiver, and a reward
of +5 is assigned.
In this way, we can apply multi-agent reinforcement learn-ing to
solve cognitive radio problem.
4. EXPERIMENTAL SIMULATIONIn this section, we describe
preliminary results from ap-
plying multi-agent reinforcement learning to our cognitiveradio
model. The overall aim of our proposed learning basedapproach is to
allow the CR users (hence, agents) to decideon an optimal choice of
transmission power and spectrumso that (i) PUs are not aected, and
(ii) CR users share thespectrum in a fair manner.
4.1 Experimental DesignA novel CR network simulator described in
Section 4.1 has
been designed to investigate the eect of the proposed
rein-forcement learning technique on the network operation. Asshown
in Figure 4, our implemented ns-2 model is composedof several
modications to the physical, link and networklayers in the form of
stand-alone C++ modules. The PUActivity Block describes the
activity of PUs based on theon-o model, including their
transmission range, location,and spectrum band of use. The Channel
Block contains achannel table with the background noise, capacity,
and occu-pancy status. The Spectrum Sensing Block implements
theenergy-based sensing functionalities, and if a PU is
detected,the Spectrum Management Block is notied. This, in
turncauses the device to switch to the next available channel,
andalso alert the upper layers of the change of frequency. The
1708
-
Spectrum Sharing Block coordinates the distributed chan-nel
access, and calculates the interference at any given nodedue to the
ongoing transmissions in the network. The CrossLayer Repository
facilitates the information sharing betweenthe dierent protocol
stack layers.
We have conducted a simulation study on two topologies:a 3 3
grid network with a total of 18 CR users (the smalltopology), and a
random deployment of 2000 CR users dis-tributed in a square area of
2000m side (the large topology).Half of the total deployed nodes
are senders, and the nearestneighbor to each of them becomes their
respective receiver.We consider the time to be slotted, and the
link layer at eachsender node attempts to transmit with a
probability p = 0.2in every slot. In the small topology, we assume
4 spectrumbands, given by the set F = {50MHz, 500MHz, 2GHz,
and5GHz}, and 4 transmit power values. There are a total of 2PUs.
In the large topology, we assume 400 spectrum bands,chosen as
default in the range from 50MHz to 5GHz, and20 transmit power
values. There are a total of 100 PUs.In both topologies, the
permissible power values are 20muniformly distributed between 0.5mW
to 4mW.
Each PU is randomly assigned one default channel inwhich it
stays with probability 0.5. It can also switch tothree other
pre-chosen successively placed channels with thedecreasing
probability {0.4, 0.3, 0.3}, respectively. Thus, thePU has an
underlying distribution with which it is active ona given channel,
but this is unknown to the CR user. Thetransmission in the CR
network occurs on multiple sets ofpre-decided node pairs, each such
pair forming a link rep-resented as {i, j}. The terms in the
parenthesis denote thedirectional transmission from the sender i to
the receiver j.The choice of spectrum is made by the sender node,
andis communicated to the receiver over the common controlchannel
or CCC. This CCC is also used to return feedbackto the sender
regarding possible collisions that may be expe-rienced by the
receiver. However, data transmission occursexclusively in the
spectrum chosen by the node pair formingthe link.
We compare the performance of our reinforcement learn-ing based
(RL-based) scheme with the other three schemes:(i) random
assignment, that selects a random combinationof spectrum and power
in each round; (ii) greedy assign-ment with history 1 (G-1), and
(iii) greedy assignment withhistory 30 (G-30). The G-1 algorithm
stores for every pos-sible spectrum and power combination the
reward receivedthe last time that combination was selected (if
any). Thealgorithm selects the combination with the highest
previousreward with probability and explores a randomly
chosencombination with probability (1 ). The G-30
algorithmmaintains a repository of the reward obtained in the 30
pastslots for every combination of power and spectrum, and se-lects
the best combination in the past 30 slots. Similar toG-1, G-30
selects the best known combination from the his-tory with = 0.8,
and explores a randomly chosen one withprobability (1 ) = 0.8. In
our RL-based scheme, the ex-ploration rate is set to 0.2, which we
found experimentallyto give the best results. Action exploration
stops after 25000epoches. The initial learning rate is set to 0.8,
and it isdecreased by a factor of 0.995 after each time slot.
Notethat G-1 uses the same amount of memory as the RL-basedscheme,
but the G-30 uses 30 times more memory.
4.2 Results
!
"
#
#
$
#
$
#
#
#
#
#
Figure 5: Average probability of successful trans-mission for
the small topology.
%
%
% %% %%% %% %%%% %% %%% %% %%%% %% %%% %% %%%%
%
Figure 6: Average reward of CR users for the smalltopology.
%
% %% %%% %% %%%%%%%%%%%%%%%
%%%%%%%%%%%
%
Figure 7: Average number of channel switches forthe small
topology.
1709
-
We applied the four schemes, i.e. random, G-1, G-30,RL-based, to
the small and large topologies. We collect theresults over 30000
time slots, and record the average proba-bilities of successful
transmission, the average rewards of CRusers, and the average
number of channel switches by CRusers. We then plot these values
over time. Each experimentis performed 5 times and we report the
means and standarddeviations of the recorded values. In our
experiments, allruns were found to converge within 30,000
epochs.
Figure 5 shows the average probability of successful
trans-mission when applying the four schemes to the small
topol-ogy. The results show that the RL-based scheme
transmitssuccessful packets with an average probability of
approx-imately 97.5%, while the G-30, G-1 and random
schemestransmit successful packets with average probabilities of
ap-proximately 88.2%, 79.4%, and 48.7%, respectively. Theresults
indicate that after learning, the RL-based approachcan eectively
guarantee successful transmissions, and itsperformance is much
better than the others, including theG-30 scheme which uses more
than an order of magnitudemore memory.
Figure 6 shows the corresponding average rewards receivedby CR
users when applying the four schemes to the smalltopology. The
results show that after learning, the RL-basedscheme receives the
largest positive reward of approximately+4.3, while the G-30 gets a
reward of approximately +1.7,G-1 gets an negative average reward of
approximately 0.8and the random scheme gets a negative average
reward ofapproximately 7.5. The results indicate that the
RL-basedapproach pushes CR users to gradually achieve higher
posi-tive rewards and choose more suitable spectrum and powervalues
for their transmission. The results also indicate thatthe reward
tends to be proportional to the probability ofsuccessful
transmission.
Figures 7 shows the corresponding average number of chan-nel
switches by CR users when applying the four schemes tosolve the
small topology. The results show that after learn-ing, the RL-based
scheme tends to decrease channel switch-ing to 0, while G-30 keeps
the channel switches to approxi-mately 1.5, G-1 keeps the channel
switches to approximately2.6, and the random scheme keeps the
channel switches toapproximately 6.0. The results indicate that our
RL-basedapproach is able to keep the channel switches very low.
Theresults also indicate that our approach can converge to
anoptimal solution for successful transmission after learning.
We further observe in the graphs of Figures 5, Figures 6and
Figures 7 that the behavior of the RL-based scheme issmoother and
more predicatable than the behavior of theother approaches. These
results suggest that our approachis more stable than the G-30, G-1,
and random approaches.
Figures 8 shows the average probabilities of
successfultransmissions when applying the RL-based, G-30, and
ran-dom schemes to the large topology. The results show thatthe
RL-based scheme transmits successful packets with theaverage
probability of approximately 78.2%, while the G-30 and random
scheme transmit successful packets with theaverage probabilities of
approximately 49.5% and 43.2%, re-spectively. Note that the average
probabilities of successfultransmissions increases sharply after
250000 epochs becauseaction exploration stops. The results indicate
that our pro-posed approach outperforms the G-30 and random
approachafter learning, even when G-30 uses more memory.
Figures 9 shows the corresponding average rewards of CR
!
"
#
#
$
##
#
#
Figure 8: Average probability of successful trans-mission for
large topology.
%
%
% %% %%% %% %%%% %% %%% %% %%%%
%% %%% %% %%%%
%
Figure 9: Average reward of CR users for largetopology.
%
%%
%%
%%
%%
%%%
%%
% %% %%% %% %%%%%%%%%%%%%%%
%%%%%%%%%%%
%
Figure 10: Average number of channel switches forlarge
topology.
1710
-
users when applying the same three schemes to the largetopology.
The results show that after learning, the RL-basedscheme receives
the largest average reward of approximately10.9, while the G-30
gets an average reward of approxi-mately 13.6 and the random scheme
gets a negative av-erage reward of approximately 14.0. The results
indicatethat our RL-based approach reduces the negative rewardsto
the largest extent when unsuccessful transmission mustoccur in a
cognitive radio network.
Figures 10 shows the corresponding average number ofchannel
switches of CR users when applying the three schemesto the large
topology. The results show that after learning,the RL-based scheme
tends to decrease channel switching to0, while G-30 keeps the
channel switches to approximately906 and the random scheme keeps
the channel switches toapproximately 991. The results indicate that
our proposedapproach can keep the channel switches very low and
con-verge to an optimal solution.
In the large topology, our reinforcement learning tech-nique
shows two key characteristics: (1) the learning curveinitially
underperforms in both the probability of successfultransmission and
the reward earned by the CR user, respec-tively , and (2) it
outperforms the competing schemes to-wards the later end of the
simulation duration, after about25, 000 time slots. In both these
cases, the learning tech-nique assumes a at slope that indicates
fewer uctuationsand a convergence-like behavior. Moreover, the
number ofchannel switches is signicantly lower than the others,
whichdirectly results in higher network throughput. The RL-based
scheme, as well as the competing schemes, are subjectto the complex
network behavior and exhibit dierent sub-optimal peaks before
converging on a static value. We notethat the authors are not aware
of simulation studies of sucha large-scale network being previously
performed.
5. FUNCTIONAPPROXIMATIONFORRL-BASED COGNITIVE RADIO
A key limitation on the eectiveness of reinforcement learn-ing
is the size of the table needed to store the state-actionvalues.
The requirement that an estimated value be storedfor every state
limits the size and complexity of the learn-ing problems that can
be solved. Problems with large statespaces, such as large-scale
cognitive radio problems, are typ-ically dicult to solve.
Function approximation [5], which stores an approxima-tion of
the entire table, is one way to solve this problem.Many function
approximation techniques exist, includingcoarse coding [9] and tile
coding [4] (also known as CMAC[19]), and there are guarantees on
their eectiveness in somecases. A limitation of these techniques is
that they cannothandle continuous state-action spaces with high
dimension-ality [17]. Sparse Distributed Memories (SDM) [11] can
alsobe used to reduce the amount of memory needed to store
thestate-action value table. This approach applied to
reinforce-ment learning, also called Kanerva Coding [17],
represents afunction approximation technique that is particularly
well-suited to problem domains with high dimensionality.
In order to implement Kanerva Coding to our reinforce-ment
learning based approach, a collection of k prototypestate
(prototypes) p = ( sp, pw) is randomly selected fromthe state space
of every CR user. A state s and a prototypep are said to be
adjacent if their Euclidean distance is no
&
& & & && & &
&
&
&
Figure 11: Average probability of successful trans-mission for
large topology.
more than a threshold. Normally, we set the threshold as
theratio of the number of states over the number of prototypes.A
state is represented as a collection of binary features, eachof
which equals 1 if and only if the corresponding prototypeis
adjacent. A value (i) is maintained for the ith feature,and an
approximation of the value of a state is then thesum of the values
of the adjacent prototypes. In this way,Kanerva Coding can greatly
reduce the size of the value ta-ble that needs to be stored in the
RL-based approach. Thecomputational complexity of the approach
depends entirelyon the number of prototypes, which is not a
function of thenumber of the dimensions of the state space.
5.1 ResultsWe evaluate our reinforcement learning-based
approach
with Kanerva-based function approximation (RL-K-based)by
applying it to the large topology. We compare the perfor-mance of
the RL-K-based scheme with the RL-based scheme.In the RL-K-based
approach, the learning parameters aresame as the RL-based approach.
The number of prototypesvaries over the following values: 2000,
4000, 6000. Note thatthe number of states in the large topology is
8000, that is,400 (channels) 20 (power values).
Figures 11 shows the average probability of successful
trans-mission when applying the RL-K-based scheme with vary-ing
numbers of prototypes to the large topology. The re-sults show that
after learning, the RL-K-based scheme with2000, 4000 or 6000
prototypes transmits successful packetswith an average probability
of approximately 41.5%, 65.2%,and 70.5%, respectively. The results
indicate that althoughthe performance of the RL-K-based scheme is
worse thanthe RL-based scheme, the RL-K-based scheme can use
lessmemory. For example, the RL-K-based scheme uses 2/3 ofmemory of
the pure RL scheme with a loss of only 7.9%of successful
transmissions with 6000 prototypes, and uses1/2 size of memory with
the loss of only 13.0% of success-ful transmissions with 4000
prototypes. The results alsoshow that if the number of prototypes
is too small, theperformance is similar to randomly choosing
channels andpower values. In our future work, we will focus on
usingprototype optimization techniques to improve the eciencyof
Kanerva-based function approximation for reinforcement
1711
-
learning based cognitive radio.
6. CONCLUSIONSCognitive radio is a new paradigm that attempts to
op-
portunistically transmit in licensed frequencies, without
af-fecting the existing primary users of these bands. To
realizethis capability, such a radio must predict specic
interferertransmission patterns and adaptively change its
operationalparameters, such as transmit power and spectrum.
Thesetasks, collectively referred to as spectrum management,
aredicult to achieve in a dynamic distributed environment, inwhich
CR users may only make local decisions, and react tochanges in the
environment. In this paper, we described anovel spectrum management
approach based on multi-agentreinforcement learning for CR ad hoc
networks with decen-tralized control. Our approach uses value
functions to mea-sure the desirability of choosing dierent
transmission pa-rameters, and enables ecient assignment of spectrum
andtransmit powers by maximizing long-term rewards.
We evaluated our approach by applying it to several real-world
scenarios. By comparing the communication perfor-mance with random
and greedy spectrum assignment, weshowed that our reinforcement
learning-based approach out-performs the other approaches. We also
employed Kanerva-based function approximation to improve our
approachsability to handle large cognitive radio networks. By
eval-uating its eect on communication performance, we showedthat
function approximation can eectively reduce the mem-ory used for
large networks with little loss of performance.We therefore
conclude that our reinforcement learning basedspectrum management
can signicantly reduce interferenceto licensed users, while
maintaining a high probability of suc-cessful transmissions in a
cognitive radio ad hoc network.
7. REFERENCES[1] I. F. Akyildiz, W.-Y. Lee, and K. R.
Chowdhury.
Crahns: Cognitive radio ad hoc networks. Ad HocNetworks Journal
(Elsevier), 7(5):810836, July 2009.
[2] I. F. Akyildiz, W.-Y. Lee, and K. R. Chowdhury.Spectrum
management in cognitive radio ad hocnetworks. IEEE Network,
23(4):612, July 2009.
[3] I. F. Akyildiz, W.-Y. Lee, M. C. Vuran, andS. Mohanty. Next
generation / dynamic spectrumaccess / cognitive radio wireless
networks: a survey.Computer Networks Journal (Elsevier),
50:21272159,September 2006.
[4] J. Albus. Brains, Behaviour, and Robotics.McGraw-Hill,
1981.
[5] L. Baird. Residual algorithms: Reinforcement learningwith
function approximation. In Proc. of the 12th Intl.Conf. on Machine
Learning. Morgan Kaufmann, 1995.
[6] FCC. Spectrum policy task force report. ET DocketNo. 02-155,
Nov. 2002.
[7] G. J. Gordon. Stable function approximation indynamic
programming. In Proc. of Intl. Conf. onMachine Learning, 1995.
[8] S. Haykin. Cognitive radio: Brain-empowered
wirelesscommunications. IEEE Journal on Selected Areas
inCommunications, 23(2):201220, February 2005.
[9] G. Hinton. Distributed representations. TechnicalReport,
Department of Computer Science,Carnegie-Mellon University,
Pittsburgh, 1984.
[10] IEEE Std 802.11b Specication. Ieee std802.11b-1999/cor
1-2001, 2001.
[11] P. Kanerva. Sparse Distributed Memory. MIT Press,1988.
[12] J. Mitola III. Cognitive radio for exible mobilemultimedia
communication. In Proc. IEEEInternational Workshop on Mobile
MultimediaCommunications (MoMuC) 1999, pages 310,November 1999.
[13] J. Proakis. Digital Communications.
McGraw-HillScience/Engineering/Math, August 2000.
[14] B. Ratitch and D. Precup. Sparse distributedmemories for
on-line value-based reinforcementlearning. In Proc. of the European
Conf. on MachineLearning, 2004.
[15] G. L. Stuber. Principles of mobile communication(2nd ed.).
Kluwer Academic Publishers, 2001.
[16] R. Sutton. Generalization in reinforcement
learning:Successful examples using sparse coarse coding. InProc. of
Conf. on Neural Information ProcessingSystems, 1995.
[17] R. Sutton and A. Barto. Reinforcement Learning:
AnIntroduction. Bradford Books, 1998.
[18] M. D. Waltz and K. S. Fu. A heuristic approach
toreinforcment learning control systems. In IEEETransactions on
Automatic Control, 10:390-398., 1965.
[19] C. Watkins. Learning from delayed rewards. Ph.Dthesis,
Cambridge Univeristy, Cambridge, England,1989.
[20] M. Wooldridge. An introduction to multiagentsystems. In
John Wiley Sons Ltd, ISBN0-471-49691-X, 2002.
1712
