Computational Intelligence Group Projects & Research Interestsfakegame.sourceforge.net/lib/exe/fetch.php?media=cig-pri.pdfComputational Intelligence Group, 19. 6. 2008 Projects & Research

Computational Intelligence GroupProjects & Research Interests

http://cig.felk.cvut.czDepartment of Computer Science and Engineering

Faculty of Electrical EngineeringCzech Technical University in Prague

EUROSIM 2007

Computational Intelligence Group, 19. 6. 2008Projects & Research Interests 2

Team & Scope

● 5 employees

● 6 PhD students

● Datamining, computational intelligence, artificial neural networks, evolutionary algorithms


Cooperation

● IBM Research Czech Republic

● Sun Microsystems

● Seznam (czech information portal)

● National museum

● 1st and 2nd Medical faculty, Charles University

● Faculty for Human Studies, Charles University


Upcoming Events


CIG (Software) Projects

● FAKE GAME – open source data mining tool

● CIV toolkit – advanced algorithms for Cell processor

● MathSC – Mathematica softcomputing toobox

● BlueCar – mobile robot for intelligent rooms

● SiMoNNe – simulator of modular Neural Nets

Being prepared:

● Java OPT – nature inspired optimization package

● PREPit – automated data preprocessing


FAKE GAME

● FAKE (Fully Automated Knowledge Extraction)

● by GAME (Group of Adaptive Models Evolution)

● Inductive modeling datamining tool

● Implemented in Java, opensourced in 2007 http://sourceforge.net/projects/fakegame/


CIV Toolkit

● Computational Intelligence and Voice Processing Toolkit on IBM Cell Broadband Engine

● 3x PlayStation3

● HMM, DTW, PSO, SOM, Neural Gas, Genetic Alg.

● http://axon.felk.cvut.cz/civtoolkit


Data Preprocessing

● Data preprocessing is a corner stone of successful data mining and modelling.

● It involves among others– Data transformation

– Outliers detection and treating

– Missing data imputation

– Data reduction

– Feature selection/Feature ranking


FAKE GAME Preprocessing Module

● Some of basic preprocessing methods are implemented in the FAKE GAME project.

● The wizard is implemented to guide user through basic preprocessing steps.


Automated Preprocessing

● Selection and setup of preprocessing methods is very complex.

● To automate selection of preprocessing methods the genetic approach is involved.– Simple Genetic Algorithm

– Linear Genetic Programming


Introduction to Feature Ranking and Selection

How important is each feature?

Feature Ranking

1. P-length2. P-width3. S-length4. S-width

Reduction

Knowledge

Feature Selection of

dimensionality

Ranks


Feature Ranking(FR) in FAKE-GAME

● FAKE-GAME tool creates GAME models using Niching Genetic Algorithm(NGA)

● Importance of each feature can be obtained as a side effect of NGA by computing utilization in model building process

● This approach also causes selection of important features by ignoring redundant and irrelevant.


Example of Feature Selection using FAKE-GAME

● Exapmle with Hypercube data set from UCI data repository

– UNC is a number of unique chromosomes used for feature ranking inside NGA (from 2 to All unique chromosomes)

– First row with bold numbers shows correct rank of features

– Gray background cells are unused features

UNC 1 2 3 4 5 6 7 8 9 102 1 2 3 4 5 6 7 8 9 103 1 2 3 4 5 6 7 8 9 10

1 / 4 1 2 3 4 5 6 7 8 9 101 / 3 1 2 3 4 5 6 7 8 9 101 / 2 1 2 3 4 5 6 7 8 9 102 / 3 1 2 3 4 5 6 7 8 9 10All 1 2 3 4 5 6 7 8 9 10

● Every feature has correct rank

● With growing number of UNC is Feature Selection fewer restrictive


Parallel Processing of Recurrent Neural Networks

● Recurrent neural networks– Fully connected

– Next state depends on previous state

● Proposed solutions– Reduce number of temporal connection to lower

communication overhead (Brain cortex architecture is similar)

– Introduce the data set parallelism (ensembles of networks)

yt1=d−∑i=1d

x jt−wij

t 2∑k=1

n

ykt mik

t


Motivation for Use of Multicore Parallel Systems

● Mainstream today

● Powerful, inexpensive hardware for consumer electronics and game consoles

● Effective use of resources? (most programs runs on single core)

● Highly available and with general purpose programming

● Specialized ASICs more powerful then FPGAs


Suitable Parallel Platforms

● Sony/IBM based (Cell)● 2 AltiVec CPUs, 8 SIMD SPU cores

● Intel based (x86)● 2-8 core CPUs, 80 VLIW cores in the future● SMP, cache coherent, SIMD

● nVidia based (GPU)● High performance computing initiative Tesla● nVidia CUDA C environment● 96-128 cores / chip, NUMA


Current Work

● Scalable parallel processing of recurrent neural networks (THSOM as the possible representative)

● Usage of multicore CPUs with care to specific architecture constraints– General purpose x86 CPUs (SMP)

– nVidia CUDA capable GPU (NUMA)

– Sony PS3 with Cell CPU


Algorithms

● GAME – evolution of hybrid inductive models

● THSOM – temporal data clustering

● CEA – continuous evolution of individuals

● DEANN – evolution of neural networks

● ANTCAST – ant colony with castes


Continual Evolution Algorithm

● Hybrid Genetic Algorithm– Combination of the genetic algorithm and

– gradient-based optimization method

– Variable population size

– Age parametr of the individual

– Sequential replacement of individuals

– Separated encoding of structure and behavior

– Evolution in the continual time dimension,


Two dimensional evolution in CEA


Neural Networks Construction


Evolution Control Process

● Probability functions– Death and Reproduction probabilities

● Balancing Functions


Evolution Control Process


TWEANNs

● TWEANN (Topology and Weight Evolving

Artificial Neural Network) algorithms.

● Topology and parameters (weights) are

optimized simultaneously,

– no need to „guess“ the right topology,

– optimal topology is likely to be found.

● Use of Evolutionary Algorithms (EAs).


(In)direct encodings

● With classic approaches (direct encodings) only relativly small neural networks are possible-> curse of dimensionality

● Indirect encodings allow the compression of information -> small genome encodes large (regular) neural nework.

● Inspiration in nature -> human genome consists of 30 000 genes which encode about 1011 neurons and 1014 synapses!


DEANN cellular encoding

● Cellular encoding – based on cellular (neural) growth, cell division etc., the program to build a neural network is encoded as a tree.

● Our algorithm DEANN (Developmental Evolution of Artificial Neural Networks)-> the cellular growth is controlled by a biology inspired model of a Gene Regulatory Network.

small tree

encodes large neural network


Ant Colony Continuous Optimization

● Optimizing parameters of one neuron in GAME

x1, x

2, ... , x

n є R

hybridizing existing ANT methods with gradient search


Ant Colony Optimization with Castes

Improving ant algorithms using

groups of ants with different

behaviour

● Solving: (A)TSP, SOP, phylogenetic trees

Spaeth, Cooper, Ferguson (2003)


Applied Computational Intelligence

● Colabroute – datamining from GPS tracks

● Robospace – shape reconstruction from laser scans

● Spiral – parkinson disease recognition

● BlueCar – mobile robot for intelligent rooms

● ParrotTalk – parrot speech analysis


Colabroute

● Datamining from GPS tracks

● Automated construction of routable road maps

● Automated extraction of points of interrest (Fuel stations, dangerous crossroads, ...)


Robospace

● Reconstruction from unstructured vector clouds

● Self-organizing Maps

● Application in mobile robotics


Parkinson Disease Recognition

● Analysis of spirals drawn by hand


Parrot Speech Analysis

● Speech Recognition Methods applied to analysis of voices of grey parrots.

● Clustering of samples by Self-organizing Maps


Mining Biological Signals

● Interest in data mining mainly but not limited to medical applications.– Sleep stages recognition based on EEG

– Heart contractions shapes classification based on ECG


Parallel Subsolutions for TSP

● Solving subproblems in parallel on Cell

● Updating pheromone on main CPU

clustering: k-means http://www.playstation2.cz


Anthropological data modelling

In this project, we focus on processing Anthropological data by means of several data mining methods. The goal is to predict an age of individuals described by a set of parameters measured on their skeletons. Data in this project are problematic due to very high noise. Methods are tuned and parameterized to give best possible performance on data. The performance of methods is compared and the recommendation, how to process noisy and partially inconsistent data will be one of the final conclusions of this project.


Estimation of Fetal weight

● Find accurate model of fetal weight prediction ● Based on sonography measured data during

pregnancy shortly before delivery

● EFW = 0,0504AC2*16,427AC + 38,867FL + 284,074

Documents

Computational Intelligence Group Projects & Research Interestsfakegame.sourceforge.net/lib/exe/fetch.php?media=cig-pri.pdfComputational Intelligence Group, 19. 6. 2008 Projects & Research