32
pypet PyData 2016 Robert Meyer Neural Information Processing, TU-Berlin

Robert Meyer- pypet

  • Upload
    pydata

  • View
    229

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Robert Meyer- pypet

pypet PyData 2016

Robert Meyer

Neural Information Processing, TU-Berlin

Page 2: Robert Meyer- pypet

Motivation

The cycle of simulations and proto-typing

Idea

CodingSimulations

Page 3: Robert Meyer- pypet

Motivation

The cycle of simulations (ground truth)

Idea

Spaghetticode

Parameter exploration bash scripts

Hacky I/Ofunctionalityto store data

Data Analysisthat only works with

my current I/O

Page 4: Robert Meyer- pypet

Motivation

The cycle of simulations (ground truth)

Uhm, maybe I shouldtry something else

Spaghetticode

Parameterexplorationbash scripts

Hacky I/Ofunctionalityto store data

Data Analysisthat only works with

my current I/O

Page 5: Robert Meyer- pypet

Motivation

The cycle of simulations (ground truth)

Spaghetticode

Parameterexplorationbash scripts

Hacky I/Ofunctionalityto store data

Data Analysisthat only works with

my current I/OLet's repeat an old simulation

Uhm, maybe I shouldtry something else

Page 6: Robert Meyer- pypet

Motivation

The cycle of simulations (ground truth)

Spaghetticode

Parameterexplorationbash scripts

Hacky I/Ofunctionalityto store data

Data Analysisthat only works with

my current I/OLet's repeat an old simulation

Uhm, maybe I shouldtry something else

Page 7: Robert Meyer- pypet

Motivation

The cycle of simulations (ground truth)

Uhm, maybe I shouldtry something else

Spaghetticode

Parameterexplorationbash scripts

Hacky I/Ofunctionalityto store data

Data Analysisthat only works with

my current I/O ???

Page 8: Robert Meyer- pypet

Motivation

The cycle of simulations (ground truth)

Uhm, maybe I shouldtry something else

Spaghetticode

Parameterexplorationbash scripts

Hacky I/Ofunctionalityto store data

Data Analysisthat only works with

my current I/O

pypet

Page 9: Robert Meyer- pypet

pypet

python parameter exploration toolbox Suitable for any numerical simulation in python

Page 10: Robert Meyer- pypet

pypet

python parameter exploration toolbox Suitable for any numerical simulation in python

Main objectives Allow easy exploration of the parameter space

Easy storage and retrieval of computed resultsand the underlying set of parameters

Page 11: Robert Meyer- pypet

pypet

Evolves around the trajectory container Tree Data Structure to manage parameters

and results Group your parameters according to meaningful categories

>>> traj.f_add_parameter('traffic.ncars', 100, comment = 'Number of vehicles')>>> traj.f_add_parameter('traffic.cyclists', 42, comment='Number of cyclists')>>> traj.f_add_parameter('simulation.dt', 0.01, comment='Timestep')

Page 12: Robert Meyer- pypet

pypet

Evolves around the trajectory container Tree Data Structure to manage parameters

and results Group your parameters according to meaningful categories

>>> traj.f_add_parameter('traffic.ncars', 100, comment = 'Number of vehicles')>>> traj.f_add_parameter('traffic.cyclists', 42, comment='Number of cyclists')>>> traj.f_add_parameter('simulation.dt', 0.01, comment='Timestep')

Access your data with Natural Naming >>> traj.traffic.ncars

100

Or cut short through the tree >>> traj.cyclists

42

Page 13: Robert Meyer- pypet

pypet

Evolves around the trajectory container Handles a variety of data formats

Python Natives (bool, int, float, complex, str) Python Containers (dict, list, tuple)

NumPy Arrays and Matrices, Scipy Sparse Matrices Pandas DataFrames BRIAN(2)_ Quantities and Monitors (Everything you can pickle)

_BRIAN(2): http://briansimulator.org/

Page 14: Robert Meyer- pypet

pypet

Evolves around the trajectory container Handles a variety of data formats

Python Natives (bool, int, float, complex, str) Python Containers (dict, list, tuple)

NumPy Arrays and Matrices, Scipy Sparse Matrices Pandas DataFrames BRIAN(2)_ Quantities and Monitors (Everything you can pickle)

Your work with something else? Easily extendible for other parameters and results

_BRIAN(2): http://briansimulator.org/

Page 15: Robert Meyer- pypet

pypet

Evolves around the trajectory container Tree Data Structure to manage parameters

and results

Allows arbitrary exploration of the parameterspace Beyond grids (Cartesian product space) and

straight lines

Page 16: Robert Meyer- pypet

pypet

Evolves around the trajectory container Tree Data Structure to manage parameters

and results

Allows arbitrary exploration of the parameterspace beyond grids (cartesian product lines

ncars

cyclists

Page 17: Robert Meyer- pypet

pypet

Evolves around the trajectory container Tree Data Structure to manage parameters

and results

Allows arbitrary exploration of the parameterspace beyond grids (cartesian pr lines

ncars

cyclists

Page 18: Robert Meyer- pypet

pypet

Evolves around the trajectory container Tree Data Structure to manage parameters

and results

Allows arbitrary exploration of the parameterspace beyond grids (cartesian produ lines

cyclists

ncars

Page 19: Robert Meyer- pypet

pypet

HDF5_ (Hierarchical Data Format) Portable file format for managing and storing data

Can handle large amounts of data Human readable data

You can browse within your data as in a regular file system

Your data is automatically stored using the PyTables_ interface

_HDF5: http://www.hdfgroup.org/HDF5_PyTables: http://www.pytables.org/

Page 20: Robert Meyer- pypet

pypet

Standing on the shoulder of giants HDF5_ (Hierarchical Data Format)

Portable file format for managing and storing data You can browse within your data as in a regular file system

Page 21: Robert Meyer- pypet

pypet Provides and environment to disentangle

your actual scientific simulation from I/O management and execution

Manages logging

Page 22: Robert Meyer- pypet

pypet Provides and environment to disentangle

your actual scientific simulation from I/O management and execution

Manages logging

Automatic integration with git version control

Page 23: Robert Meyer- pypet

pypet Provides and environment to disentangle

your actual scientific simulation from I/O management and execution

Manages logging

Automatic integration with git version control

Allows adaptive exploration (optimization with DEAP_)

_DEAP: http://deap.readthedocs.org

Page 24: Robert Meyer- pypet

pypet Provides and environment to disentangle

your actual scientific simulation from I/O management and execution

Manages logging

Automatic integration with git version control

Allows adaptive exploration (optimization with DEAP_)

Allows parallel simulations and multiprocessing(on multiple serves or clusters like SGE/qsub using SCOOP_)

_DEAP: http://deap.readthedocs.org _SCOOP: http://scoop.readthedocs.org

Page 25: Robert Meyer- pypet

pypet

Live Demo (although you shouldn't do any)!

>>> from pypet import Environment

Page 26: Robert Meyer- pypet

Even more important! Open source: project available on github_

Fully documented on readthedocs_

pypet

_github: https://github.com/SmokinCaterpillar/pypet

_readthedocs: https://pypet.readthedocs.org/en/latest/

Page 27: Robert Meyer- pypet

Even more important! Open source: project available on github_

Fully documented on readthedocs_

Thoroughly tested More than 1000 tests, 91% code coverage Continuous integration testing with Travis_ (Linux) and

Appveyor_ (Windows)

pypet

_github: https://github.com/SmokinCaterpillar/pypet

_readthedocs: https://pypet.readthedocs.io

_Travis: https://travis-ci.org/SmokinCaterpillar/pypet

_Appveyor:https://ci.appveyor.com/project/SmokinCaterpillar/pypet

Page 28: Robert Meyer- pypet

pypet

Where to get pypet (for python 2.6 till 3.5) ? https://pypi.python.org/pypi/pypet Or >>> pip install pypet Or https://github.com/SmokinCaterpillar/pypet

Documentation?! https://pypet.readthedocs.io Tutorial More than 20 comprehensive examples

Page 29: Robert Meyer- pypet

Bonus Track

I am looking for a job!

Soon to be PhD with lack of motivation/self-delusion to stay in academia

Page 30: Robert Meyer- pypet
Page 31: Robert Meyer- pypet

Bonus Track

I am looking for a job!

Soon to be PhD with lack of motivation/self-delusion to stay in academia

Will do (Python) Coding, Data Science, and Machine Learning for food!

Page 32: Robert Meyer- pypet

Thank You!

from xkcd.com

And thanks to the BCCN Berlin,my supervisor Prof. Klaus Obermayer,

Robert Pröpper, Phillip Meier, Owen Mackwood, and Mehmet Nevvaf Timur