Python Day5

8/13/2019 Python Day5

1/51

Scientific Computing in Computer Science, Technische Universitat Munchen

Part VII

Misc Modules, Functionality, and More

Tobias Neckel: Scripting with Python... and beyond

Compact Course @ GRS, June 03 - 07, 2013 112


2/51


Module random

Compute and use pseudo-random numbers

from random import *

print random()

print c h o i c e ( [ 1 , 4 , " H e l l o " ])

(Re)initialize random with seed([x])(def: time)




3/51


4/51


Module random (2)

Random integers

r an di nt ( a , b ) # a


5/51


Running External Programs

Module subprocess

Example: call echo in subshell

# ! / u s r / b i n / p y t h o n

import s u bp ro ce s s as sp

p = sp . Po pen ( " e c h o H e l l o W o r l d " , s he ll = T ru e ,

s t d o u t = s p . P I P E )

print p . s t d o u t . r e a d ( )

Opens new shell: shell=True Runs command echo HelloWorld Creates file object (read only) for stdout of shell




6/51


7/51


Module subprocess (2)

Example: Call Gnuplot

# ! / u s r / b i n / p y t h o n

import s u bp ro ce s s as sp

p = sp . Po pen ( [" g n u p l o t " , " - p e r s i s t " ] ,

s t di n = s p . P IP E , s t do u t = s p . PI PE , s t de r r = s p . S T DO U T )

p . s t d i n . w r i t e ( " p l o t s i n ( x ) \ n " )

p . s t d i n . f l u s h ( )

r a w _ i n p u t (" P r e s s E n t e r t o c o n t i n u e " )

p . s t d i n . w r i t e ( " p l o t s i n ( x ) , c o s ( x ) \ n " )

p . s t d i n . c l o s e ( )

File objects for stdin and stdout+stderr I/O bufferred (flush()if necessary)




8/51


9/51


Operating System Services (2)

Some methods related to files

o s . c h mo d ( p at h , m od e ) # c hm od

o s . c h m o d ( " t m p . p y " , 0 66 4) # f ou r d ig it s r eq ui re d

l i s t d i r ( p a t h ) # ls

mkdir ( path ) # m kd ir

makedirs ( path ) # m kdi r -pr e na m e ( sr c , d st ) # mv

r e m o v e ( p a t h ) # rm

r m d i r ( p a t h ) # r md ir

r e m o v e d i r s ( p a t h ) # rm - rf

s t a t ( p a t h ) # s ta ts ( atime , . .. )

w al k ( p at h [ , t op do wn ] ) # t ra ve rs e dir rec .

# ( d ir pa th , d ir na me s , f na me s ) f or e ac h d ir




10/51



Submoduleos.path To manipulate pathnames and check for file properties

a b s p a t h ( p a t h )

b a s e n a m e ( p a t h )

d i r n a m e ( p a t h )

e x i s t s ( p a t h )

i s fi l e ( p a th ) ; i sd i r ( p a th ) ; i s li n k ( p a th )

g e t s i z e ( p a t h )

g e t a t i m e ( p a t h )

j oi n ( pa th 1 , p at h2 , . .. )




11/51



Example traversing a directory

# ! / u s r / b i n / p y t h o n

import os

def p r i n t _ t r e e ( p a t h ) :

" "" P r in t t he c on te nt s of t he d ir ec to ry s pe c if ie d

in p ath a nd a ll i ts s ub di re ct or ie s . "" "for ( d i r pa t h , d i rn a me s , f n a me s ) in o s . w a l k ( p a t h ) :

# ge t th e d ir ec to ry l ev el

l ev e l = l en ( d i rp a th . s pl i t ( os . s e p ) ) -1

# p ri nt d i re ct or y

print " | " * l e v e l + " + " + o s . p a t h . b a s e n a m e ( d i r p a t h )

# pr int all file sfor f in f n a m e s :

print " | " * l ev el + " | " + f

p r i n t _ t r e e (" . " )




12/51


Regular ExpressionsPattern Syntax

Regular expressions should always be placed in a raw string E.g. r([^.]*.(.*))

. match any character but newline^ match start of the string match end of the string* match previous expression zero or more times (greedy)+ match previous expression one or more times (greedy)? match previous expression zero or one time (greedy)*?, +?, ?? non-greedy versions of*, +, ?{m} match previous expression m times{m,n} match previous expression m to n times (greedy)

{m,n}? non-greedy version of{m,n}[...] match any character from the enclosed set[^...] match any character not in the setA|B matches A or B (both regular expressions)(...) stores the contents of the match inside the backets




13/51


Character Escape Sequences

Characters with special meaning (., *, ...) can be used withtheir literal meaning by escaping them with a\, e.g. \., \*

Standard escape characters (\n, \t, ...) work as expected,e.g. r\n+matches one or more newlines

\number match the text matched by group number (starting from 1)\d same as[0-9]\D same as[^0-9]\s matches any whitespace (same as[\t\n\r\f\v]\S matches nonwhitespace\w matches any alphanumeric character\W matches nonalphanumeric characters\A matches the start of a string\Z matches the end of a string




14/51


Regular Expressions (cont.)

findall(patt, string) find all non-overlapping matches

sub(patt, repl, string) replaces matches by repl

from re import *

regstr = r \ d * \ . \ d * | \ d *

s = " 1 2 . 3 / 5 / 4 . 4 ; 5 . 7 ; 6 "

findall( regstr ,s)

regstr = r ( \ d * \ . \ d * | \ d * ) s ub ( r eg s tr , r \ 1 x x x , s )

greedy vs. non-greedy

r eg _g re ed y = r < . * >

s = " title "

m_g = search ( reg_greedy , s ) # w h ol e < H1 > t i tl e < / H1 > m a t c he d

r e g_ n on g re e dy = r < . * ? > # ma tc h as f ew c ha rs as p os si bl e

m_ng = search ( reg_nongreedy , s ) # o nl y m at ch ed

print m _ g . gr ou p ( ) + " , " + m _n g . g ro up ( )




15/51


Match Objects

Some re functions do not return strings, but match objects (m)

match(patt, string)returns MatchObject in case of match

search(patt, string)searches for first match

finditer(patt, string)like findall, but returns iterator

m.group([group1, group2]) returns subgroups of the match

import re

s = " d a t e s 0 5 / 0 2 / 2 0 1 0 - 0 5 / 1 4 / 2 0 1 0 "

patt = r ( \ d + ) / ( \ d + ) / ( \ d + )

for m in f i n d i t e r ( p a t t , s ) :print m . g r o u p ( )

print ( " % s - % s " % ( m . g ro up ( 2) , m . g ro up ( 3) ))




16/51

S i tifi C ti i C t S i T h i h U i it t M h


17/51


Exceptions: Dealing with Errors

Consider the user enters characters:

x = r aw _i np ut ( " P l e a s e e n t e r a n u m b e r : " )

print f l o a t ( x )

Failed conversion raises an ValueErrorexception

P le as e e nt er a n um be r : xy z

...

T ra ce b ac k ( m os t r ec en t c al l l as t ):

File " < s t d i n > " , line 1 , in

V a l ue E r ro r : i n va l id l i te r al for f lo a t ( ): x yz

To deal with erroneous behaviour catch exception and handle it



Scientific Computing in Computer Science Technische Universitat Munchen


18/51


Exceptions: Dealing with Errors (2)

Syntax is

try :

# do s om et hi ng

except V a l ue E r ro r , e : # Py > 3.0 : " E xc ep ti on as e "

# h an dl e V al ue E rr or except E x ce p t io n , e :

# h an dl e all o th er e rr or s

else :

# what to do if no er ror o c c ur e d

finally :

# code that sh ou ld be e xe cu te d in any case





19/51



Back to the previous example

g o t_ an s we r = F al se

w hi le n ot g o t _ a n s w e r :


try :


g o t_ an s we r = T ru eexcept V a l ue E r ro r , e :

print " T h a t w a s n t a n u m b e r "

# do s om et hi ng with e





20/51



Back to the previous example

g o t_ an s we r = F al se

w hi le n ot g o t _ a n s w e r :


try :


g o t_ an s we r = T ru eexcept V a l ue E r ro r , e :

print " T h a t w a s n t a n u m b e r "

# do s om et hi ng with e

To raise your own errors (use one of many default types or createyour own):

raise R u n t i m e E r r o r ( " O o o p s ! S o m e t h i n g w e n t w r o n g ! " )

raise I O E r r o r ( " F i l e n o t e x i s t e n t . . . " )





21/51


Built-in Exceptions

BaseException # root of all exc .

GeneratorExit

K e y b o a r d I n t e r r u p t # Ctrl +C , e .g .

SystemExit # P ro gr am E xi t ( s ys . e xi t ( ))

Exception # Base for non - e xit exc .

StopIteration

StandardError # only P yt ho n 2. x A r i t h m e t i c E r r o r

F l o a t i n g P o i n t E r r o r

Z e r o D e v i s i o n E r r o r

AssertionError

AttributeError

E n v i r o n m e n t E r r o rIOError # o pe n (" f i le . t xt " ) , e . g .

OSError

EOFError

ImportError # m odu l not found , e .g .





22/51

p g p ,

LookupError

IndexError # t up el s / l is ts , e . g .

KeyError # d i ct i on a ri e s , e . g .

MemoryErrorNameError # name not foun d

U n b o u n d e d L o c a l E r r o r

ReferenceError

RuntimeError

N o t I m p l e m e n t e d E r r o r

SyntaxErrorI n d e n t a t i o n E r r o r

TabError

SystemError

TypeError # type erro r (2 + "3")

ValueError # v al ue e rr or r ( f lo at ( " t h re e " ))

UnicodeError

U n i c o d e D e c o d e E r r o r

U n i c o d e E n c o d e E r r o r

U n i c o d e T r a n s l a t e E r r o r





23/51

Throw Built-in Exceptions

def r e a d f l o a t 1 ( ) :while T r u e :

try :

a = r aw _i np ut ( " N u m b e r b e t w e e n 0 . 0 a n d 1 . 0 : " )

a = fl oat ( a)

if (a 1 . 0 ) :

raise V a l u e E r r o r ( " N u m b e r n o t i n [ 0 . 0 ; 1 . 0 ] " )except V a lu eE r ro r a s e :

print " E r r o r : % s " % e

else :

return a

At a = float(a), an error is thrown automatically; the rest of thetry Block is not being executed

Ifaconvertable, the value range is checked and a new exceptionis possibly thrown





24/51

Module pickle

Serializing Python objects

Save objects as they are (current state) to file

import pickle

fd = open ( m y O b j e c t s , w b )

p i ck l e . d u mp ( x , f d )

p i ck l e . d u mp ( y , f d )

f d . c l o s e ( )

Most objects can be pickled Not valid for objects such as opened files





25/51

Module pickle

Serializing Python objects

Save objects as they are (current state) to file

import pickle

fd = open ( m y O b j e c t s , w b )

p i ck l e . d u mp ( x , f d )

p i ck l e . d u mp ( y , f d )

f d . c l o s e ( )

Most objects can be pickled Not valid for objects such as opened files Then load in same order

fd = open ( m y O b j e c t s , r b )x = p i c k l e . l o a d ( f d )

y = p i c k l e . l o a d ( f d )

f d . c l o s e ( )

Tobias Neckel: Scripting with Python... and beyondCompact Course @ GRS, June 03 - 07, 2013 131



26/51

Misc: Python 2 to 3

Which version to use? See

http://wiki.python.org/moin/Python2orPython3for hintsConverting from Python 2.x to Python 3.x

Convert your code to Python 2.6

To check, there is a Py3k warnings mode

p y th o n -3 s c ri p t na m e . p y




27/51

Misc: Python 2 to 3






If no warnings remain, use 2to3tool

Ships with Python Converts many things automatically Cant do everything

Rest manually




28/51

Misc: Python 2 to 3






If no warnings remain, use 2to3tool

Ships with Python Converts many things automatically Cant do everything

Rest manually

Further reading:http://docs.python.org/release/3.0.1/library/2to3.html




29/51

Misc: Python 2 to 3 (2)

Some examples

print " H e l l o " , " W o r l d " -> print ( " H e l l o " , " W o r l d " )

1 23456 7890 L -> 123456 7890

a b -> a != b

d ictio nary . has_key (x ) -> x in dictionary

try :

import somethingexcept ImportError , e : -> except I m po r tE r ro r a s e :

pass




30/51

Misc: Python 2 to 3 (2)

Some examples

print " H e l l o " , " W o r l d " -> print ( " H e l l o " , " W o r l d " )

1 23456 7890 L -> 123456 7890

a b -> a != b

d ictio nary . has_key (x ) -> x in dictionary

try :

import somethingexcept ImportError , e : -> except I m po r tE r ro r a s e :

pass

Some modules (urllib, httplib, . . . ) renamed or restructured More views, less lists

map ( lambda x : x +1 , l ) -> list ( map ( lambda x : x +1 , l ))

f i l t e r ( lambda x : x >3 , l ) - > l is t ( fi lt er ( .. .) )

xrange (10) -> range (10)

range (10) -> list ( range (10))




31/51

Misc: execfile/exec

Run commands

Run/execute commands in current namespace

exec ( " a = 3 + 7 " )

exec ( " p r i n t H e l l o " )

Run/execute commands from file

e x e c f i l e ( f i l e n a m e )

execfilecorresponds to runin ipython




32/51

Extending Python using C/C++

Performance critical parts can be written in hardware-dependentcode

Use existing functionality written in other languages

Use as module




33/51

Extending Python using C/C++

Performance critical parts can be written in hardware-dependentcode

Use existing functionality written in other languages

Use as module

Well built an example, using C++ and swig

Provides two functions, computing mod and factorial



34/51



35/51

Extending Python using C/C++ (3)

Additionally, we need to care about

Data conversion from C++ to Python Writing both wrapper and interface files




36/51


Additionally, we need to care about

Data conversion from C++ to Python Writing both wrapper and interface files

Fortunately, swig (Simplified Wrapper and Interface Generator)does everything for us

We only need a .i file:

/* t oo ls . i */

% m o d ul e t o ol s

%{

/* Put h ead er fil es here ... */

# i n c l u d e " e x a m p l e . h p p "

%}

/* P ar se the h ea de r f il e to g en er at e w ra pp er s */

% i n c l u d e " e x a m p l e . h p p "




37/51


Now, weve only got to run swig

s wi g - o t o ol s_ w ra p . c pp - c ++ - p yt ho n t oo ls . i

creatingtools.pyandtools_wrap.cpp. . .



38/51



39/51


Now, weve only got to run swig

s wi g - o t o ol s_ w ra p . c pp - c ++ - p yt ho n t oo ls . i

creatingtools.pyandtools_wrap.cpp. . .

. . . and compile everything as a shared library

g + + - o t oo l s_ wr a p . os - c - I / us r / i nc lu de / p yt ho n2 . 6 \t o ol s _ wr a p . c p p - f PI C

g ++ - o e xa mp le . o s - c e xa mp le . c pp - fP IC

g + + - o _ to ol s . so - s ha re d e xa mp le . o s t o ol s_ w ra p . o s

Then use in Python:

import tools

print t oo l s . m od ( 8 , 6 )

print t o o l s . f a c t ( 6 )




E di P h i C/C (5)


40/51


Coments

Using global variables is a little more tricky

Functions can be conveniently renamed in the .i file

% r e n am e ( _ _ s t r_ _ ) D a ta V e ct o r : : t o S t ri n g ;




Ti


41/51

TimeModuletime

Functions to access and display (the current) timestamp

Without parameter: use current time

from time import *

time() # s ec s s in ce 0 1/ 0 1/ 19 7 0

c t i m e ( [ s e c s ] ) # s tr in g ( l oc al t im e )

l o c a l t i m e ( [ s e c s ] ) # s tr uc t ( l oc al t im e )

g m t i m e ( [ s e c s ] ) # s tr uc t ( G MT )# Y , M , D , H , M , S , Wday , Yday , DST (1= y ,0= n , -1=?)

s l e e p ( s e c s ) # p au se c ur r . p ro ce ss




Ti


42/51

TimeModuletime

Functions to access and display (the current) timestamp

Without parameter: use current time

from time import *

time() # s ec s s in ce 0 1/ 0 1/ 19 7 0

c t i m e ( [ s e c s ] ) # s tr in g ( l oc al t im e )

l o c a l t i m e ( [ s e c s ] ) # s tr uc t ( l oc al t im e )

g m t i m e ( [ s e c s ] ) # s tr uc t ( G MT )# Y , M , D , H , M , S , Wday , Yday , DST (1= y ,0= n , -1=?)

s l e e p ( s e c s ) # p au se c ur r . p ro ce ss

Measure time (up to 1/100s, system clock dependent)

t = time ()for i in r a n g e ( 1 0 0 0 0 ) :

# do s om et hi ng

t = time () - t

print t , " s e c o n d s "




G hi l U I t f


43/51

Graphical User Interface

GUI

There are lots of GUI toolkits for Python Most common: TKinter






44/51


GUI

There are lots of GUI toolkits for Python Most common: TKinter

ModuleTkinter

Example: Simple window with title

# ! / u s r / b i n / p y t h o n

import T ki nt er as tk i

r oo t = tk i .Tk ()

r o o t . t i t l e (" H e l l o t o a l l ! " )

t ki . L a b e l ( ro ot , t ex t = " W e l c o m e h e r e " ,

font= " t i m e s 3 2 b o l d " ) . p a c k ( )r o o t . m a i n l o o p ( )

mainloop()runs until some event exits




Module Tkinter (2)


45/51

Module Tkinter (2)

A more useful example with a button, allowing to quit

# ! / u s r / b i n / p y t h o n

import T ki nt er as tk i

from tkMessageBox import askokcancel

import sys

def quit():

a oc = a sk o kc a nc e l (" E x i t " , " R e a l l y q u i t ? " )

if aoc:

s y s . e x i t ( )

r oo t = tk i .Tk ()

r o o t . t i t l e (" W h a t a n i c e w i n d o w . . . " )b = t ki . B ut to n ( root , t ex t =" P r e s s m e ! " ,

font= " t i m e s 2 4 n o r m a l " , c o mm a nd = q ui t )

b . p a c k ( )

r o o t . m a i n l o o p ( )




Animation with Tkinter


46/51

Animation with Tkinter

# ! / u s r / b in / p y th o nfrom Tkinter import *

import t ime , m at h

cv = C an va s ( w id th = 80 0 , h ei gh t = 6 00 )

c v . p a c k ( )

o va l = cv . c re at e_ ov al ( 300 , 50 , 400 , 1 50 )

c v . i t e mc o n fi g ( o va l , f il l = b l u e )

i = 0 . 0

while True:

c v . m ov e ( o va l , 1 0* m at h . c os ( i ) , 1 0* m at h . s in ( i ) )

c v . u p d a t e ( )

t i m e . s l e e p ( 0 . 0 2 )

i += 0.05




Compressing files


47/51

Compressing files

There are modules for tar, zip, bz2, gz, . . .




Compressing files


48/51

Compressing files

There are modules for tar, zip, bz2, gz, . . .

Example: module gzip

Simple to use: just replace standard call toopen

import gzip

fd = g zi p . op en ( " f i l e . t x t . g z " , " w " )

f d . w r i t e ( " "" F u nn y l i ne s

in g zip f il e """ )

f d . c l o s e ( )

fd = g zi p . op en ( " f i l e . t x t . g z " )

print f d . r e a d ( )f d . c l o s e ( )

...as easy as that

File modes as usual (rwa + b for binary)




More Python Fun :-)


49/51

More Python Fun :-)

Web server in 3 lines?

import S i m p l eH T T P Se r v e r , S o c k e t S e rv e r

h tt p d = S o c ke t S er v e r . T C P S er v er ( (" " , 8000) , \

S i m p l e H T T P S e r v e r . S i m p l e H T T P R e q u e s t H a n d l e r )

h t t p d . s e r v e _ f o r e v e r ( )




More Python Fun :-)


50/51

More Python Fun :-)

Web server in 3 lines?

import S i m p l eH T T P Se r v e r , S o c k e t S e rv e r

h tt p d = S o c ke t S er v e r . T C P S er v er ( (" " , 8000) , \

S i m p l e H T T P S e r v e r . S i m p l e H T T P R e q u e s t H a n d l e r )

h t t p d . s e r v e _ f o r e v e r ( )

What could this be good for?

Want to share quickly some files with colleagues in the samenetwork?

Goto directory, start python, run three lines, tell them your IPand the port (here: 8000)

Thats it!





51/51

import sys

sys.exit(0)



Documents

Python Day5