PyBrain

An introduction

What's in PyBrain

Neural networks

Reinforcement learning

Black-box optimization

Supervised learning

...

General RL framework:

Agent interacts with an EnvironmentTask provides rewardsExperiment connects them

An agent generally chooses its actions according to a parametrized Policy .

(by default parameters is a continuous vector)

A Learner adapts the parameters of the policy in order to maximize reward.

Examples of Environments that work with Tasks:

FlexCubeEnvironmentRollingUpTask

WalkDirectionTaskJumpTask

CaptureGameGoMokuGame

MaximizeWinTaskHandicapTask

MazeEnvironment CheeseMazeTaskTMazeTask

A different perspective on the same scenario:Black-box optimization .

Class hierarchy for Learners

Policies use Modules :black box transform (vector-) input into (vector-) outputtransformation depends on parameters

Modules can be differentiable :

back-propagate output-error towards the inputcompute the partial derivatives w.r.t. each parameter

Example Modules:

ActionValueTable SigmoidLayerBernoulliLayerBiasUnitLSTMLayerGateLayerLinearLayer SoftmaxLayer

Modules can be combined in Networks :linked with Connections (which can have parameters)Networks are Modules themselvesNetwork-parameters are the concatenation of the component parameters.

Example Networks:

FeedForwardNetworkRecurrentNetworkConvolutionalNetworkBidirectionalNetwork

Example Connections:

FullConnectionIdentityConnectionPermutationConnectionSharedConnection

Example: A simple Recurrent Neural Network

4 Module components (1 in, 1 out) without parameters4 Connection components with parameters1 of which is a recurrent Connection (pink)

# initialize a new networkfrom pybrain import *n = RecurrentNetwork() # add all modules (in any order) n.addOutputModule(SigmoidLayer(2, name='out'))n.addModule(TanhLayer(10, name='h'))n.addModule(BiasUnit(name='bias'))n.addInputModule(LinearLayer(4, name='in')) # add all connections (in any order) n.addConnection(FullConnection(n['h'], n['out'])) n.addConnection(FullConnection(n['bias'], n['h'])) n.addConnection(FullConnection(n['in'], n['h'])) n.addRecurrentConnection(FullConnection(n['h'], n['h'])) # build the network structure n.sortModules() # use it to map an input to an outputoutput = n.activate([0,1,2,-2])

General supervised learning framework:

DataSet provides training examples: inputs with associated known targetsModule provides model that estimates targets from inputsTrainer tries to improve the model

Another view on the problem shows the analogy to reinforcement learning

A DataSet essentially consists of a collection of 2d-arrays (=fields ), holding the patterns to learn from.

Each ML problem solved by PyBrain has a custom DataSet .

Simplest case: SupervisedDataSet

DataSets can be saved/read assembled pattern by pattern constructed from arrays transformed, split, scaled extended with custom fields ...

The Trainer's function is to adjust parameters of the Module directly or (most of the time) iteratively, to produce the best model for the data. Schematically, this works as follows:

Module with initial parameters

Module is activated with an input pattern and produces output

Error is calculated by comparing with target, and back-propagated through the module

Parameters are adjusted by some gradient descend or other scheme

Further information

Web page: http://www.pybrain.org

Source: http://github.com/pybrain/pybrain

Mailing list: http://groups.google.com/group/pybrain/

Documentation: http://www.pybrain.org/docs/