bash_day4

8/13/2019 bash_day4

http://slidepdf.com/reader/full/bashday4 1/27

Part V

Working RemoteAnd More

Tobias Neckel: Scripting with Bash and Python

Compact Course @ Max-Planck, October 07 - 18, 2013 61

8/13/2019 bash_day4


Working remote - ssh, scp and rsync

telnet - plain communication

• telnet hostname port

• Useful to find out whether some service works as it should

• Do NOT use it to work remotely, ssh is much more secure!

• Try telnet towel.blinkenlights.nl :-)

ssh and scp - security by encryption• Developed 1995 to allow secure connections between computers

• ssh username@computername log in onto computername asusername

• scp -r * username@computername:<path_rel_to_home>

• scp *.txt username@computername:/<absolute_path>

• scp "username@computername:path/*.*"./ – copy all remote filesin folder path to current directory

• Quotes necessary for remote wildcards



8/13/2019 bash_day4


8/13/2019 bash_day4


Working remote - ssh, scp and rsync (2)

rsync - backup and syncronisation

• Data transfer with scp for few files and unique transfer• rsync transmits only differences and compresses• General syntax rsync [OPTION...] SRC... [DEST]

• Both remote and local

Potentially useful examples:

• rsync sourcedir/ targetdir/ – sync sourcedir to targetdir• rsync sources targetdir/ – file/dir sources into targetdir• rsync sources user@host:targetdir/ – for remote access same

syntax as scp

• rsync -avuCn <src> <dest>

•

-a – archive: recursive; preserve permissions, times, . . .• -v – verbose• -u – update: skip files that are newer on the receiver• -C – exclude certain files (e.g. *~ #* .#* ,* _$* *$

*.old *.bak *.o *.a *.exe core .svn...)• -n – perform a trial run with no changes made



8/13/2019 bash_day4


Synchronising data - Unison

• very nice to sync 2 locations

• GUI and backup support

• 2-way sync (including merge etc.)• supports win and linux (MAC?)

• needs gtk and ssh

• download and infos:http://www.cis.upenn.edu/~bcpierce/unison/



8/13/2019 bash_day4


Working remote - screen

• during a typical session, many terminals are used• if working remote, this means opening many ssh connections

• for each login/logout, everything has to be closed/opened again

• screen solves all these problems by creating virtual terminals

• C-a c creates a new terminal• C-a " shows a selection-list of available terminals

• C-a d or screen -d detach the screen

• C-a D D detach the screen and log out

• screen -r reattach the screen

• C-a A give a name to a terminal

• C-a w show the currently opened terminals

• C-a k kill a terminal



8/13/2019 bash_day4


Process management

schedule jobs

at 4:01 pm < job . txt # use atq for list , atrm to de let e

c ro nt ab - e # -l to list

# m h dom mon dow command

12 14 * * * t ou ch / h om e / ne ck el / c ro n . txt

priority• nice -n <value> <command> starts command with lower priority• top shows processes with top computing time• within top, r can renice a process• nohup run a command immune to hangups

• ps -ef list all processes• pidof find out the process id for a program name• kill <ID> kills process with the given Id (-9 to force)• killall <name> kills processes with given name• bg and fg send programms to background/foreground



8/13/2019 bash_day4


Some more commands you should know

basename " t h i s / i s / t h e / p a t h / t o / a . f i l e "

dirname " t h i s / i s / t h e / p a t h / t o / a . f i l e "

t im e s le ep 3

time for i in

s eq 1 0 0 00 0

; do a =

[3*

i ]; done

users

groups

du - sh

df -h

ln - s < file > < link >

echo " 3 . 6 + 5 * 4 " | bc



8/13/2019 bash_day4


Part VI

Visualizing with Gnuplot



8/13/2019 bash_day4


Now for Something Completely Different?!

Why gnuplot?

• Fast way to visualize scientific data

• Vast functionality

• It’s free and available on Windows, Linux, and MAC

• Often bash-tools helpful to preprocess/use data

Next steps

• Shortly introduce basic functionality

• Visualization of 2d and 3d scientific data



8/13/2019 bash_day4


Interactive

$ g n up l otG N U P L O TV er si on 4 .2 p a t c hl e v e l 3l a s t m o d if i e d Mar 2008System : Linux 2.6.27−11−g e n e r i c

C o p y r i g ht ( C) 1986 − 1 99 3 , 1 99 8 , 2 00 4 , 2 00 7 , 2008

Thomas W i l l i a m s , C o l i n K e l l e y and many o t h e r s

Type ‘ h el p ‘ t o acc ess t h e on−l i n e r e f e re n c e manual .The g n u p l o t FAQ i s a v a i l a b l e f r om h t t p : / / www . g n u p l o t . i n f o / f a q /

Send b ug r e p o r t s and s u g g es t i o ns t o <h t tp : / / so u r c e fo r g e . n e t / . . . >

T er mi na l t yp e s et t o ’ w xt ’g n u p l o t> h e l p

‘ gnupl ot ‘ i s a command−d ri ve n i n t e r a c t i v e f u n c t i o n and datap l o t t i n g program .[ . . . ]



8/13/2019 bash_day4


Interactive Use

• Command interpreter

• Help functionality

• help – for general help• help commands – for overview• help plot – for help on command plot

• quit or <Ctrl+D> to exit



8/13/2019 bash_day4


A first example

Plotting sin(x )

gnupl ot> p l o t s i n ( x )gnupl ot>

. . . leads to



8/13/2019 bash_day4


Useful Commands

Comments

• Comments start with #

Plotting

• plot <function> for 2d data

• splot <function> for 3d data

• plot [xmin:xmax] sin(x) – specify range for x -axis• plot sin(x), cos(x) – multiple plots

• replot – repeat last plot command

• Escape linebreaks with \

• help plot – help on further optionsFunctions

• Variables x , y , z

• Usual operators; 2*x**2+sin(x)/3.2−4



8/13/2019 bash_day4


Useful Commands – Plotting

Some plot modifiers

• plot sin(x) title ”A nice plot” – specify title• plot sin(x) with <style> – specify style

Syntax :w i t h <s t y l e> { { l i n e s t y l e | l s < l i n e s t y l e >}

| {{ l i n e t y p e | l t < l i n e t y p e >}

{ l i n e w i d t h | lw < l i n e w i d t h >}{ l i n e c o l o r | l c <col orspec>}{ p o i n t t y p e | p t <p o i n t t y p e >}{ p o i n t s i z e | ps <p o i n t s i z e >}{ f i l l | f s < f i l l s t y l e >}{ nohidden3d}

{ p a l e t t e }}}

where <s t y l e> i s e i t h e r ’ l i n e s ’ , ’ p o i n t s ’ ,’ l i n e s p o i n t s ’ , ’ i m p ul s e s ’ , ’ dot s ’ , ’ ste ps ’ , . . .

• use test to see available style of your terminalTobias Neckel: Scripting with Bash and Python


8/13/2019 bash_day4


Setting Variables and Parameters

• Use set, e.g.

gnupl ot> s e t x ra ng e [ 0 : 3 * p i ]gnupl ot> s e t y ra ng e [ − 2 : 2 ]

gnupl ot> se t x l ab e l ” X a x i s ”gnupl ot> se t y l ab e l ” Y a x i s ”gnupl ot> p l o t s i n ( x )

• Use unset to reset settings, see help unset

• For xrange it’s restored with set xrange restore



8/13/2019 bash_day4


Setting Variables and Parameters

• Choose terminal (default: wxt)

gnupl ot> s et t e r mi n algnupl ot> s et t e r mi n al png t r u e c o l o r f o n t ” a r i a l ” 20 \

enhanced

• Save to file

gnupl ot> s e t ou t [ pu t ] ” f i l ena me ”gnupl ot> p l o t s i n ( x )



8/13/2019 bash_day4


Save and Load

Save and load settings• Save settings:

save ”filename. plt ”

• Load settings:load ”filename. plt ”

• load works also for plot commands⇒ use also for scripts or collection of commands

Batch mode

• Similar for batch mode:

gnuplot filename.gplt – run script filename.gplt

• Helpful hint: use parameter -persist

gnuplot -persist filename.gplt



8/13/2019 bash_day4


More Useful Commands

Functions, variables

• Use variables, calculations, functions

gnupl ot> a = s q r t (16)*10+2gnupl ot> p r i n t a42.0gnupl ot> p r i n t p i

3.14159265358979gnupl ot> f ( x ) = a * s i n ( x+ p i )gnupl ot> p l o t f ( x )

Shell access

• Leaving Gnuplot to list directory discards settings• Therefore shell-ecape:

• Single commands with ! command

• Spawn shell with shell , return with exit

• pwd and cd "folder" directly accessible (tab works)



8/13/2019 bash_day4


Plot Scientific Data

• Assume data file gaussian1.dat

# x f ( x ) error

- 2. 00 00 0 0 0 . 05 3 99 1 0 . 01 5 47 3

- 1. 80 00 0 0 0 . 07 8 95 0 0 . 02 5 09 6

- 1. 60 00 0 0 0 . 11 0 92 1 0 . 03 7 26 4

...

2 . 00 0 00 0 0 . 05 3 99 1 0 . 00 4 49 2

• Plot with

p l o t ’ gaussian1 . dat ’ w l i n e s p o i n t s lw 2 ps 2 p t 5p l o t ’ gaussian1 . dat ’ w e r r o r b a r s lw 2 p t 3

p l o t ’ gaussian1 . dat ’ u 1 :3 w i mp ul se s lw 2p l o t ’ gaussian1 . dat ’ w e r r o r b a r s l t 1 lw 2 , \

’ ’ w l i n e s l t 1 lw 2 , \’ gaussian2 . dat ’ w e r r o r b a r s l t 2 lw 2 p t 5 , \’ ’ w l i n e s l t 2 lw 2



8/13/2019 bash_day4


Data within the Gnuplot file

• Apart from data in files ( ’filename’ and ’ ’ ), data can be providedinline ( ’ −’)

• Easy way to pipe data to gnuplot without having to createintermediate files

s e t term pngs e t ou t ” t e s t . png ”p l o t ”−” w l i n e s0 01 12 4

3 9end

• replot then doesn’t work



8/13/2019 bash_day4


3d Data

Unordered data sets

s p l o t ” d i s t i l l e d . d at ” u 1 : 2 : 8



8/13/2019 bash_day4


3d Data

3d data on a grid

# x y f ( x , y )

0 0 40 1 40 2 5

1 0 51 1 51 2 7

3 0 63 1 7

3 2 1 0

• If grid has no missing entries, gnuplot creates surface

s p l o t ” 3 d f i l e . d at ” w l i n e s



8/13/2019 bash_day4


3d Grid Data

• There are much more options for plotting

• We don’t cover them all, just few examples

s p l o t s in ( x ) * y + 1 1 w l i n e ss e t h id de n3 d

r e p l o ts e t i sosample 40show views e t v ie w 60,15s e t c on to u r bases e t pm3d at bunset c o n t o u rs e t pm3d



8/13/2019 bash_day4


Modify values

• File one2ten.dat:

1 1. . .

10 10

• Use of built-in modifiers

p l o t ’ one2ten . dat ’ u 1 : ( $2 * $2 ) w l i n e s , \’ ’ u 1 : ( s q r t ( $2 ) ) w l i n e s , \

’ ’ u 1 : ( $1+$2 ) w l i n e s

Tobias Neckel: Scripting with Bash and PythonCompact Course @ Max-Planck, October 07 - 18, 2013 85

8/13/2019 bash_day4


Using Bash Commands

• Bash commands such as sed, awk, sort, tail, cat, paste, . . . canbe used as well

• Allows powerful data processing on the fly

• Examples:

p l o t ’<

cat one2ten . dat el even2twe l ve . dat ’ \u 1 : ( $2 * $ 2 ) w l i n e sp l o t ’ one2ten . dat ’ u 1 : ( $2 * $2 ) w l i n e s , \

’< sed ” s / 4 / 5 / g ” one2ten . d at ’ u 1 : ( $ 2 * $2 ) \w l i n e s

p l o t ”< awk ’ { p r i n t $1 , ( $2<4) ? 0 : $2 } ’ one2ten . dat ” \

w l i n e sp l o t ”< awk ’ { x=x+$2 ; p r i n t $1 , x /NR} ’ one2ten . dat ” \

w s t e p s


8/13/2019 bash_day4


Finally

More information. . .

• gnuplot website:http://www.gnuplot.info/

• Demo scripts for gnuplot:http://gnuplot.sourceforge.net/demo/

• “not so Frequently Asked Questions”:http://t16web.lanl.gov/Kawano/gnuplot/index-e.html


Documents

bash_day4