An Architectural View of Game Theoretic Control

Raga Gopalakrishnan and Adam WiermanCalifornia Institute of Technology

Jason R. MardenUniversity of Colorado at Boulder

6/18/2010 Hotmetrics 2010

Distributed Resource AllocationSensor Coverage Wireless Access Point Selection

Wireless Channel Selection Power Control (sensor networks)

Resource Allocation Problem – A Simple Model

• Set of (distributed) agents, N = {1, 2, . . ., n}• Set of resources, R• Action sets, Ai µ 2R for agents i 2 N– Set of action profiles, A = A1 £ A2 £ . . . £ An

– Set of agents choosing resource r in action profile a, {a}r

• Objective function, W : A! R– Linearly separable, i.e., W(a) = r2R Wr ( {a}r )

Goal: Find an allocation a 2 A that maximizes W(a)

Distributed Approaches

Distributed Optimization

Lyapunov-based Control

Physics-inspired Control

Game-theoretic Control

Distributed Approaches

Distributed Optimization

Lyapunov-based Control

Physics-inspired Control

Game-theoretic Control

Promising new approach Model the agents as “self-interested”

players in a non-cooperative game

Still being explored The solution to the problem emerges

as the equilibrium of the game

Modeling the problem as a game

• Set of players, N = {1, 2, . . ., n}

• Action sets, Ai µ 2R for players i 2 N– Set of action profiles, A = A1 £ A2 £ £ An

– Set of players choosing resource r in action profile a, {a}r

• Utility functions, Ui : A! R for players i 2 N– Linearly separable, i.e., Ui(a) = r2R fr ( i,

{a}r )

• Welfare function W : A! R– Linearly separable, i.e., W(a) = r2R Wr

( {a}r )

Resource Allocation Problem Resource Allocation Game

• Set of agents, N = {1, 2, . . ., n}• Set of resources, R• Action sets, Ai µ 2R for agents i 2 N

– Set of action profiles, A = A1 £ A2 £ . . . £ An

– Set of agents choosing resource r in action profile a, {a}r

• Objective function, W : A! R– Linearly separable, i.e., W(a) = r2R Wr

( {a}r )

Game Theoretic Control (GTC)

Setup the game1

Design the players2

decision makers/players action sets

utility functions

agent decision rules(learning rules)

Desirable globalbehavior emergesas equilibrium ofthe game

Goal:

• A Nash equilibrium is an action profile a*2 A such that for each player i,

• Measures of efficiency for Nash equilibrium:

Game Theoretic Control (GTC)

Setup the game1

Design the players2

decision makers/players action sets

utility functions

agent decision rules(learning rules)

Desirable properties Existence of an equil. Efficiency of an equil. Tractability Locality of information Budget balance …

Desirable properties Locality of information Fast convergence Equilibrium selection Robust convergence …

Learning Design

Utility Design

Inherited

DesignedDesigned

Many other applications: [Akella et al. 2002, Kaumann et al. 2007, Marden et al. 2007, 2008, Mhatre et al. 2007, Komali and MacKenzie 2007, Zou and

Chakrabarty 2004, Campos-Nanez 2008, Marden & Effros 2009]

[Marden, Wierman 2008]

[Campos-Nanez, Garcia, Li 2008]

Applications of GTC

Utility Design

Learning Design

Sensor Coverage

Power Control (sensor networks)Is there a way to view Game Theoretic Control from an application-independent

perspective?

Architectural View for GTC

Utility Design

Learning Design

Class of Games

“Virtualization”layer

IP

NetworkApps

Networkhardware

OS

software

hardware

• Potential Games are games for which there exists a potential function F : A! R such that ∀ i 2 N, ∀a–i 2 A–i , ∀ ai, ai’ 2 Ai , it holds that

F (ai , a–i) – F (ai’ , a–i) = Ui (ai , a–i) – Ui (ai’ , a–i)

• Key Property: Local maxima of F are Nash equilibria

Potential Games-based Architecture

Utility Design

Learning Design

Potential Games

Unifying view of several existing designs:

[Akella et al. 2002][Kaumann et al. 2007]

[Marden et al. 2007, 2008][Mhatre et al. 2007]

[Komali and MacKenzie 2007][Zou and Chakrabarty 2004]

[Campos-Nanez 2008][Marden & W 2008]

[Marden & Effros 2009]and many others…

Utility Design (examples)

• Wonderful Life Utility (WLU) [Wolpert et al. 1999]

– Potential game with © = W (hence, price of stability = 1)– Price of anarchy = ½ for sub-modular games

• Shapley Value Utility (SVU) [Shapley 1953]– Potential game– Price of anarchy = Price of stability = ½ for sub-modular games

• Weighted SVU [Shapley 1953]– Similar properties as SVU

Adapted from cost-sharing literature in economic theory [Marden, Wierman]

Learning Design (examples)• Gradient Play [Ermoliev et al. 1997, Shamma et al. 2005]– Convergence to a Nash equilibrium

• Joint Strategy Fictitious Play (JSFP) [Marden et al. 2009]– Convergence to a Nash equilibrium

• Log-Linear Learning [Blume 1993, Marden et al.]– Convergence to the best Nash equilibrium

• Many others . . . [Ozdaglar et al. 2009, Shah et al. 2010]

Potential Games-based Architecture

Utility Design

Learning Design

Potential Games

SVU WonderfulLife

WSVU

GradientPlay

Log-Linear

Learning JSFP

+ Modularity / Decoupling

+ Flexibility

? Relationships to other approaches

? Limitations

+ Modularity / Decoupling

+ Flexibility

? Relationships to other approaches

? Limitations

Distributed Approaches

Distributed Optimization

Lyapunov-based Control

Physics-inspired Control

Potential Games

UtilityDesign

LearningDesign

Relationships to Other Approaches

Game-theoretic Control

• Distributed Constraint Optimization Problem (DCOP)

– Utility Design: WLU– Learning Design: Variety

Chapman, Rogers, Jennings – Benchmarking hybrid algorithms for distributed constraint optimization games [OptMAS ‘08]

Potential Games

WLU

Variety

Distributed Optimization

Distributed Approaches

Distributed Optimization

Lyapunov-based Control

Physics-inspired Control

Potential Games

UtilityDesign

LearningDesign

Game-theoretic Control

Relationships to Other Approaches

• Gibbs-sampler-based control―Utility Design: WLU―Learning Design: Log-Linear Learning

Access Point Selection Channel Selection

Kauffmann, Baccelli, Chaintreau, Mhatre, Papagiannaki, Diot – Measurement-based self organization of interfering 802.11 wireless access networks [INFOCOM ‘07]

Potential Games

WLU

Log-Linear Learning

Physics-inspired Control

We prove that

Distributed Approaches

Distributed Optimization

Lyapunov-based Control

Physics-inspired Control

Potential Games

UtilityDesign

LearningDesign

Game-theoretic Control

Relationships to Other Approaches

Distributed Approaches

Distributed Optimization

Lyapunov-based Control

Physics-inspired Control

Potential Games

UtilityDesign

LearningDesign

Game-theoretic Control

Relationships to Other Approaches

Potential Games-based Architecture

Utility Design

Learning Design

Potential Games

SVU WonderfulLife

WSVU

GradientPlay

Log-Linear

Learning JSFP

+ Modularity / Decoupling

+ Flexibility Relationships to

other approaches

? Limitations

+ Modularity / Decoupling

+ Flexibility Relationships to

other approaches

? Limitations

Do Potential Games Suffice?No utility design with all the

desirable properties

Utility Design

Learning Design

POTENTIAL GAMES

Desirable properties Existence of an equil. Efficiency of an equil.Budget balanceTractability Locality of information …

Not always!

Open Question: What other limitations are there?

Any linearly separable, budget-balanced utility design that guarantees equilibrium existence has PoS · ½

[Marden, Wierman 2009]

Summary

Utility Design

Learning Design

Potential Games

SVU WonderfulLife

WSVU

GradientPlay

Log-Linear

Learning JSFP

+ Modularity / Decoupling

+ Flexibility Relationships to

other approaches― Not all desirable

properties can be achieved

+ Modularity / Decoupling

+ Flexibility Relationships to

other approaches― Not all desirable

properties can be achieved

? Beyond Potential Games

Conclusion

Utility Design

Learning Design

Potential Games

SVU WonderfulLife

WSVU

GradientPlay

Log-Linear

Learning JSFP

+ Modularity / Decoupling

+ Flexibility Relationships to

other approaches― Not all desirable

properties can be achieved

?

Other choices for virtualization layer

[MW’09,AJWG’09,Sv’09]Strengths and Limitations

A library of architectures

Thank You