Business Intelligence: Optimization for Decision Making

Business Intelligence:Optimization for Decision Making

A short introduction to Particle Swarm Optimization

Michael G. Epitropakis

Computational Heuristics, Operational Research and Decision Support CHORDS,School of Natural Sciences,

Computing Science and Mathematics,University of Stirling, UK

[email protected]

Stirling, 24 March 2015

Michael G. Epitropakis Swarm Intelligence for Decision Making 1

Outline

1 Business IntelligenceMotivationDefinitionBusiness Intelligence Architecture

2 Global Optimization Problem

3 Swarm IntelligenceParticle Swarm Optimization (PSO)Background, Origins.The Original PSO modelPSO: Geometric Illustration

4 Applications

5 References


Business Intelligence Motivation

Business Intelligence: Motivation

Amazon, Barclays, Facebook, Google, Lloyds, Microsoft,Sainsbury’s, TESCO, ...

Data!The answer to my problem is hidden in my data... but I cannotdig it up!


Business Intelligence Definition

Business Intelligence

The enterprises that are capable of transforming data intoinformation and knowledge can use them to make quickerand more effective decisions and thus to achieve a competitiveadvantage.

Business Intelligence:

Business intelligence may be defined as a set ofmathematical models and analysis methodologies thatexploit the available data to generate information andknowledge useful for complex decision-making processes.




Main purpose of business intelligence systems:Is to provide decision makers with tools and methodologies thatallow them to make effective and timely decisions.

Effective: The application of rigorous analytical methodsallows decision makers to rely on information andknowledge from data which are more dependable.

Make better decisions and devise action plans that allowtheir objectives to be reached in a more effective way.

Timely: Enterprises operate in economic environmentscharacterized by growing levels of competition and highdynamism.

Rapidly react to the actions of competitors and to newmarket conditions is a critical factor in the success or eventhe survival of a company.






Data, information and knowledge

Data −→ Information −→ Knowledge

Data: are collected on a daily basisin the form of bits, numbers,symbols, and "objects".Information: is "organized data",which are preprocessed, cleaned,arranged into structures, andstripped of redundancy.Knowledge: is "integratedinformation", which includes factsand relationships that have beenperceived, discovered, or learned.


Business Intelligence Business Intelligence Architecture

Business Intelligence Architecture


Global Optimization Problem

Global Optimization

Single objective minimization problem:

Given a real-valued objective function f : Dn ⊆ Rn→ R, the aim is tofind an x? = (x?1,x

?2, . . . ,x

?n)> ∈Dn that

x? = arg minx∈Dn

f (x).

x? is a global minimizer and Dn is an n-dimensional scaled translationof the unit hypercube.

An objective function (f )

A solution representation of x (here x ∈Dn ⊆ Rn)

A search strategy – optimization algorithm.


Swarm Intelligence

Swarm Intelligence

Swarm intelligence (SI) is the collective behavior ofdecentralized, self-organized systems, natural or artificial.(Wikipedia)The system has abilities that are not present in theindividuals (is more intelligent)“The whole is more than the sum of its parts”Cooperation, co-evolution, competition, self-organisationand communicationExamples of systems can be found in nature: ant colonies,bird flocking, animal herding, bacteria molding and fishschooling

Beni, G., Wang, J. Swarm Intelligence in Cellular Robotic Systems, Proceed. NATO Advanced Workshop on Robotsand Biological Systems, Tuscany, Italy, June 26-30 (1989)


Swarm Intelligence

Swarm Intelligence


Swarm Intelligence

Swarm Intelligence Applications

Swarm-bots, an EU project led by Marco Dorigo, aimed tostudy new approaches to the design and implementationof self-organizing and self-assembling artifacts(http://www.swarm-bots.org/).

Swarmanoid: Towards Humanoid Robotic Swarms, Themain scientific objective of this research project is thedesign, implementation and control of a novel distributedrobotic system (http://www.swarmanoid.org)

Creation of complex interactive environments.

Disney’s The Lion King was the first movie to make useof swarm technology (the stampede of the bisons scene).

The movie “Lord of the Rings” has also made use ofsimilar technology during battle scenes


Swarm Intelligence Particle Swarm Optimization (PSO)

Particle Swarm OptimizationThe inventors:

James Kennedy(social psychologist)

Russell C. Eberhart(electrical engineer)

J. Kennedy, and R. Eberhart, Particle swarm optimization, inProc. IEEE. Int. Conf. on Neural Networks, Piscataway, NJ, pp.

1942–1948, 1995.


Swarm Intelligence Particle Swarm Optimization (PSO)

Particle Swarm Optimization (PSO)

What is PSO:a simple, computationally efficient optimization methodpopulation-based, stochastic search methoddirect search method, i.e. gradient freeindividuals follow very simple behaviors:

emulate the success of neighboring individuals,but also bias towards on experience of success

emergent behavior: discovery of optimal regions within ahigh dimensional search space

“Particle swarm algorithm imitates human (or insects) social behavior. Individualsinteract with one another while learning from their own experience, and gradually thepopulation members move into better regions of the problem space” Eberhart &Kennedy


Swarm Intelligence Background, Origins.

Induce complex behavior from simple rulesOrigins of PSO (precursors)

Reynolds (1987)’s simulation Boids: a simple flocking modelconsists of three simple local rules: http://www.red3d.com/cwr/boids/

Separation: Avoid Collision with neighboringagents (steer to avoid crowding localflockmates)

Alignment: Match the velocity of neighboringagents (steer towards the average heading oflocal flockmates)

Cohesion: Stay near neighboring agents(steer to move toward the average position oflocal flockmates)

The work of Heppner and Grenander on using a “roost” asattractor of all birds in the flock [HG90] (Seek roost)



Towards a computational principle

Evaluate your current positionCompare it to your own experience (previous best) and tothe experience of your society (neighborhood best)Imitate yourself and the others

Basic hypothesis:There are two major sources of cognition:

own experience andcommunication from others

Leon Festinger, 1954/1999, Social Communication and Cognition



The Original PSO model:

is a simplified social model of determining nearest neighborsand velocity matching

Initial objective: to simulate the graceful, unpredictablechoreography of collision-proof birds in a flock

Randomly initializes positions of birdsAt each iteration, each individual determines its nearestneighbor and replaces its velocity with that of its neighbor

This resulted in synchronous movement of the flock, butflock settled too quickly on the same, unchanging flyingdirection



Random adjustments to velocities (referred to ascraziness) prevented individuals to settle too quickly on anunchanging directionTo further expand the model, “roosts” were added asattractors:

personal experience (personal best)social experience (neighborhood best)

Introduction of the Particle Swarm Optimization method.


Swarm Intelligence The Original PSO model

Particle Swarm Optimization (PSO)Its main components:

What are the main components:

a swarm of particles (size usually fixed: NP)each particle represents a candidate solution of the problem athandthe elements of a particle represent parameters to be optimized

The search process:

Position updates:Xi(t+1) = Xi(t)+Vi(t+1), xi,j(0)∼ U(LBj,UBj)

Velocity updates:

denotes the amount of change (step size)drives the optimization processreflects the cognitive experience of a particle and thesocially exchanged information between particles.



Particle Swarm Optimization

The general PSO Algorithm1: Initialize particles in the swarm2: for each time step t do3: for each particle i in the swarm i ∈ {1,2, . . . ,NP} do4: Update cognitive knowledge/experience5: Update social knowledge/experience6: end for7: for each particle i in the swarm i ∈ {1,2, . . . ,NP} do8: Update Velocity of particle i9: Update Position of particle i

10: end for11: end for



Particle Swarm OptimizationThe global best (gbest) PSO

A simple PSO model: global best (gbest) PSO (Eberhart &Kennedy, 1995)

It uses a full neighborhood topology (star social network).

Velocity update rule per dimension:

vi,j(t+1) = vi,j(t)+ c1r1(t)(pi,j(t)− xi,j(t)

)+ c2r2(t)

(pbest,j(t)− xi,j(t)

),

vi,j(0) = 0 (preferred)c1,c2 are positive acceleration coefficientsr1(t),r2(t)∼ U(0,1)







vi,j(t+1) = vi,j(t)︸︷︷︸momentum

+ c1r1(t)(pi,j(t)−Xi,j(t)

)+ c2r2(t)


),

momentum:inertia componentprevious velocity term to carry the particle in the direction ithas traveled so farprevents particle from drastically changing direction







Vi(t+1) = vi,j(t)+ c1r1(t)(pi,j(t)− xi,j(t)

)︸︷︷︸cognitive component

+ c2r2(t)(pbest,j(t)− xi,j(t)

),

cognitive component:Pi(t): personal best position vectorquantifies performance relative to past performancestendency to return to the best position visited so far(memory)nostalgia








)+ c2r2(t)


)︸︷︷︸social component

,

social component:Pbest(t): neighborhood best position vector (here: globalbest position)quantifies performance relative to neighborstendency to be attracted towards the best position found inits neighborhood.envy



Particle Swarm OptimizationUpdate experience

Pi(t) is the personal best position calculated as (assumingminimization):

Pi(t+1) ={

Pi(t) if f (Xi(t+1))≥ f (Pi(t))Xi(t+1) if f (Xi(t+1))< f (Pi(t))

Pbest(t) is the global best position calculated as:

Pbest(t) = min{f (P0(t)), f (P1(t)), . . . , f (PNP(t))}

where NP is the number of particles in the swarm.



Particle Swarm Optimization

The gbest PSO Algorithm1: Initialize particles in the swarm2: for each time step t do3: for each particle i in the swarm i ∈ {1,2, . . . ,NP} do4: if f (Xi(t))< f (Pi(t)) then5: Pi(t) = Xi(t)6: end if7: if f (Pi(t))< f (Pbest(t)) then8: Pbest(t) = Pi(t)9: end if

10: end for11: for each particle i in the swarm i ∈ {1,2, . . . ,NP} do12:

Vi(t+1) = Vi(t)+c1r1(t)(Pi(t)−Xi(t)

)+c2r2(t)

(Pbest(t)−Xi(t)

),

13: Xi(t+1) = Xi(t)+Vi(t+1)14: end for15: end for


Swarm Intelligence PSO: Geometric Illustration

Particle Swarm OptimizationGeometric Illustration

Vi(t)

Xi(t)

Pi(t)

Pbest(t)

x1

x2




Vi(t)

Xi(t)

Pi(t)

Pbest(t)

Pbest(t)−Xi(t)

Pi(t)−Xi(t)

x1

x2




Vi(t)

Xi(t)

Pi(t)

Pbest(t)

Pbest(t)−Xi(t)

Pi(t)−Xi(t)

x1

x2




c1r1(t)(Pbest(t)−Xi(t))

Vi(t)

Xi(t)

Pi(t)

Pbest(t)

Pbest(t)−Xi(t)

Pi(t)−Xi(t)

x1

x2




c1r1(t)(Pi(t)−Xi(t))


Vi(t)

Xi(t)

Pi(t)

Pbest(t)

Pbest(t)−Xi(t)

Pi(t)−Xi(t)

x1

x2




Xi(t + 1)

c1r1(t)(Pi(t)−Xi(t))


Vi(t)

Xi(t)

Pi(t)

Pbest(t)

Pbest(t)−Xi(t)

Pi(t)−Xi(t)

x1

x2



Particle Swarm OptimizationThe local best (lbest) PSO

The local best (lbest) PSO uses a neighborhood topology (ringsocial network).



)+ c2r2(t)

(pnbest,j(t)− xi,j(t)

)︸︷︷︸social component

,

Pnbest(t): is the neighborhood best, defined as:

Pnbest(t+1) ∈ {x ∈ Ni|min{f (x),∀x ∈ Ni}}Ni = {pi−nNi

,pi−nNi+1, . . . ,pi−1,pi,pi+1, . . . ,pi+nNi}

where nNi is the neighborhood sizeneighborhoods are based on particle indices, not spatialinformationneighborhoods overlap to facilitate information exchange



Particle Swarm OptimizationSocial Neighborhood topologies: Social Network Structures

Two most common models:lbest: each particle is influenced only by particles in localneighborhoodgbest: each particle is influenced by the best found fromthe entire swarm

Ring Topology Star/full topology



Particle Swarm OptimizationSocial Neighborhood topologies: Social Network Structures (2)

Von NeumannTopology

Four Clusters Topology Wheel Topology



Particle Swarm OptimizationVelocity Clamping

Problem:The velocity has a tendency to

explode to large values.

Solution: Velocity Clamping

vi,j(t+1) ={

vi,j(t+1) if |vi,j(t+1)|< Vmax,jsgn(vi,j(t+1))Vmax,j if |vi,j(t+1)| ≥ Vmax,j

controlling the global exploration of the particlesit is problem-dependentdoes not necessarily prevent particles from leaving the searchspace, nor to converge.it confines the step sizes, therefore restricting particles fromfurther divergence




Xi(t)

x2

x1

Velocity Update

Position Update




Xi(t)

x2

x1

Velocity Update

Position Update

vi(t + 1)

v2(t + 1)




Xi(t)

Xi(t + 1)

x2

x1

Velocity Update

Position Update

vi(t + 1)

v2(t + 1)




Xi(t)

Xi(t + 1)

x2

x1

Velocity Update

Position Update

vi(t + 1)

v2(t + 1)

v̂2(t + 1)




Xi(t)

Xi(t + 1)

X̂i(t + 1)

x2

x1

Velocity Update

Position Update

vi(t + 1)

v2(t + 1)

v̂2(t + 1)



Some PSO Variants

Tribes (Clerc, 2006) – aims to adapt population size, so that it does not have tobe set by the users

FDR-PSO (Veeramachaneni, et al., 2003) – using nearest neighbour interactions

Cooperative PSO (van den Bergh and Engelbrecht, 2005) – a cooperativeapproach

CLPSO (Liang, et al., 2006) – incorporate learning from more previous bestparticles.

FIPS Fully Informed PSO (Mendes, Kennedy, 2004) – use several attractors inthe update rule

BBPSO Bare Bones PSO (Kennedy, 2003) – uses normal distribution aroundpersonal/global best

UPSO Unified PSO (Parsopoulos, Vrahatis, 2004) – a unification of gbest andlbest versions

PSODE (Epitropakis et al., 2012) – aims to combine various state-of-the-art DEand PSO variants

Standard PSO 2006, 2007, 2011 – aims to define a standard PSO version forcomparisons

http://www.particleswarm.info/


Applications

Applications (1)

Optimization Problems:Continuous, discrete, mixed search spaces, OptimizationProblemsCombinatorial Optimization ProblemsLarge Scale Optimization ProblemsMulti-modal Optimization ProblemsMulti-objective Optimization ProblemsProblems in Dynamic and Uncertain environmentsConstraint Optimization Problems


Applications

Applications (2)

Applications in:Machine Learning (clustering, classification, parametertuning, feature selection)Artificial Neural Networks (training, evolving structures)Robotics (path planning, localization)Bio-informatics and Medical Informatics (Medical diagnosisand decision making)Image processing (Image analysis, segmentation, patternrecognition)Industrial Applications (Job Scheduling, Vehicle RoutingProblem, Traveling Salesman Problem)


(-: Thank you very much for your attention :-)

Questions ???

Michael G. Epitropakis: [email protected]


References

References: Business Intelligence (Incomplete)

1 Carlo Vercellis, Business Intelligence: Data Mining and Optimization for DecisionMaking, John Wiley & Sons, 2009

2 Z. Michalewicz, M. Schmidt, M. Michalewicz, Adaptive Business Intelligence,Springer, 2006

3 Foster Provost, Tom Fawcett, Data Science for Business: What you need toknow about data mining and data-analytic thinking, O’Reilly Media, 2013

4 A. Brabazon, M. O’Neill, I. Dempsey, An Introduction to EvolutionaryComputation in Finance, IEEE Computational Intelligence Magazine, 2008

5 W. Pedrycz, N. Ichalkaranje, G.P. Wren and L. Jain, Introduction toComputational Intelligence for Decision Making, in Studies in ComputationalIntelligence, 97, 79-96, Springer-Verlag, 2008


References

References: PSO (Incomplete)1 Reynolds, C.W.: Flocks, herds and schools: a distributed behavioral model.

Computer Graphics, 21(4), p.25-34, 1987.2 Heppner, F. and Grenander, U.: A stochastic nonlinear model for coordinated

bird flocks. In S.Krasner, Ed., The Ubiquity of Chaos. AAAS Publications,Washington, DC, 1990.

3 Kennedy, J. and Eberhart, R.: Particle Swarm Optimization. In Proceedings ofthe Fourth IEEE International Conference on Neural Networks, Perth, Australia.IEEE Service Center(1995) 1942-1948.

4 Kennedy, J., Eberhart, R. C., and Shi, Y., Swarm intelligence, San Francisco:Morgan Kaufmann Publishers, 2001.

5 J. Kennedy, Small Worlds and Mega-Minds: Effects of Neighborhood Topologyon Particle Swarm Performance, Proceedings of the IEEE Congress onEvolutionary Computation, 1999, pp. 1931-1938.

6 Clerc, M.: Particle Swarm Optimization, ISTE Ltd, 2006.7 Engelbrecht, A.P., Fundamentals of Computational Swarm Intelligence, Wiley,

2006.8 Xiaodong Li, Advances in Particle Swarm Optimization, Tutorial, ACISS’09,

Melbourne9 Engelbrecht, A.P., Particle Swarm Optimization Tutorial, IEEE Congress on

Evolutionary Computation, 201310 KE Parsopoulos, MN Vrahatis, Particle swarm optimization and intelligence:

advances and applications, Information Science Reference, 2010


Documents

Business Intelligence: Optimization for Decision Making