Embed Size (px)
DESCRIPTION
Source http://www.slideshare.net/SignisVavere Signis Vāvere - senior database analyst in the second biggest bank in Latvia Tēma: Oracle DBA utilītas - Standarta un nestandarta risinājumi izmantojot shell Valoda: Latviešu Tēmas apraksts: Apraksts: Kā tikt pie shell, shell kā tāds, biežāk izmantojamās konstrukcijas, nepieciešamākās komandas, servera informācija, monitorings, kļūdu meklēšana. Daži reālās dzīves piemēri, kā shell zināšana atvieglo un paātrina ikdienas DBA darbu.
Citation preview
UNIX SHELL IN DBA EVERYDAY
:~> ps -ef | grep pmon | grep -v grep
WHAT IS UNIX?Dance bar in Ashdod, Israel
DO NOT DANCE• examples is not tested on production
• one size does not fits all
• before copy and paste, check manual
• afterwards check manual
• how to get shell • echo $SHELL • how to roll shell • examples • real life • resources
UNIX SHELL IN DBA EVERYDAY
SHELL IS SEXY!
# who | grep -i blonde | date; cd ~; unzip; touch; strip; finger; mount; gasp; yes; uptime; umount; sleep
• who - show who is logged on • grep - print lines matching a pattern • mount - mount a filesystem • touch - change file timestamps • strip - discard symbols from object files • finger - user information lookup program • umount - unmount a filesystem • gasp - a preprocessor for assembly programs • yes - output a string repeatedly until killed
# who | grep -i blonde | date; cd ~; unzip; touch; strip; finger; mount; gasp; yes; uptime; umount; sleep
SHELL IS SEXY!
SHELL IS SEXY!
SHELL IS SEXY!
SHELL IS SEXY!
SHELL IS SEXY!
WHY ME?• read founded and existing scripts • automate systems or databases • manual or scheduled tasks from shell side • monitoring • backups • multiple databases tasks
HOW TO GET SHELL?
CLIENTS • PuTTY
• KiTTY
• SuperPutty
• SecureCRT
http://www.chiark.greenend.org.uk/~sgtatham/putty/
http://www.9bis.net/kitty/
https://code.google.com/p/superputty/
http://www.vandyke.com/products/securecrt/
http://en.wikipedia.org/wiki/Comparison_of_SSH_clients
$ echo $SHELL
SHELL• sh - Bourne shell • ksh - Korn shell • bash - Bourne-Again shell • csh - C shell • tcsh, zsh, rc, es (10+)
http://en.wikipedia.org/wiki/Unix_shell
SHELL sh csh ksh bash tcsh zsh rc es Job control N Y Y Y Y Y N N Aliases N Y Y Y Y Y N N Shell functions Y(1) N Y Y N Y Y Y Command history N Y Y Y Y Y L L Command line editing N N Y Y Y Y L L Vi Command line editing N N Y Y Y(3) Y L L Emacs Command line editing N N Y Y Y Y L L User name look up N Y Y Y Y Y L L Login/Logout watching N N N N Y Y F F Filename completion N Y(1) Y Y Y Y L L Username completion N Y(2) Y Y Y Y L L Hostname completion N Y(2) Y Y Y Y L L History completion N N N Y Y Y L L Builtin artithmetic evaluation N Y Y Y Y Y N N Can follow symbolic links invisibly N N Y Y Y Y N N Periodic command execution N N N N Y Y N N Custom Prompt (easily) N N Y Y Y Y Y Y Underlying Syntax sh csh sh sh csh sh rc rc Freely Available N N N(4) Y Y Y Y Y Can cope with large argument lists Y N Y Y Y Y Y Y Has non-interactive startup file N Y Y(5) Y(5) Y Y N N Has non-login startup file N Y Y(5) Y Y Y N N Has anonymous functions N N N N N N Y Y List Variables N Y Y N Y Y Y Y Full signal trap handling Y N Y Y N Y Y Y Local variables N N Y Y N Y Y Y Exceptions N N N N N N N Y
KSH vs. BASH
$ typeset -l lowercase_only $ lowercase_only="WiBbLe" $ echo $lowercase_only wibble
lowercase_only=$(echo "WiBbLe" | tr "[:upper:]" "[:lower:]")
KSH vs. BASH
a=$(( b+c ))
let a=b+c
KSH vs. BASH
bash-3.1$ echo wibble | read variable bash-3.1$ echo $variable !
bash-3.1$
$ echo wibble | read variable $ echo $variable wibble $
KSH vs. BASH• bash is much easier to set a prompt that displays the current directory • ksh has associative arrays and bash doesn’t • ksh handles loop syntax a bit better • bash handles getting exit codes from pipes in a cleaner way • ksh has the print command which is way better than the echo command • bash has tab completions • ksh has the syntax cd old new which replaces old with new in your
directory and cd over there !
pwd - /foo/bar/barfoo/one/bar/bar/foo/bar cd to /foo/bar/barfoo/two/bar/bar/foo/bar ksh - cd one two bash - cd ../../../../../two/bar/bar/foo/bar
HOW TO ROLL SHELL?
SHEBANG
• #!/bin/ksh • special comment for interpreter • no bang, script run in current shell • shell may be on different hosts in different
locations
PROCESS RUN• cmd1 && cmd2 - run cmd1; if it returns 0 (success), run cmd2
• cmd1 || cmd2 - run cmd1; if it returns non-zero, run cmd2
• cmd1 & cmd2 - run cmd1 and also cmd2
• (ls -1) - run the command "ls -1" in a sub shell
• cmd1 | cmd2 - run cmd1 and output as input to cmd2
PIPEReal UNIX pipe
REDIRECTION
• command > outfile - redirect output to file
• command >> outfile - redirect output and append to file
• command 2> outfile - redirect STDERR
• command &> outfile - redirect STDOUT and STDERR
• tee - redirect STDOUT to file and STDOUT
REDIRECTION# ls # ls > file # ls | tee file # ls | tee –a file # ls | tee file1 file2 file3
mailx –s ‘alert.log errors’ [email protected] < \ tail -10 alert_$ORACLE_SID.log | grep ORA-|\ tee error.log !
FLUSH
# > alert_ORCL.log
IF• [ -a FILE ] True if FILE exists. • [ -d FILE ] True if FILE exists and is a directory. • [ -h FILE ] True if FILE exists and is a symbolic link. • [ -s FILE ] True if FILE exists and has a size greater than zero. • [ -r FILE ] True if FILE exists and is readable. • [ -w FILE ] True if FILE exists and is writable. • [ -x FILE ] True if FILE exists and is executable. • [ -O FILE ] True if FILE exists and is owned by the effective user ID. • [ -G FILE ] True if FILE exists and is owned by the effective group ID. • [ -L FILE ] True if FILE exists and is a symbolic link.
IF• [ -z STRING ] True of the length if "STRING" is zero. • [ -n STRING ] True if the length of "STRING" is non-zero. • [ STRING1 == STRING2 ] True if the strings are equal. • [ STRING1 != STRING2 ] True if the strings are not equal. • [ STRING1 < STRING2 ] True if "STRING1" sorts before
"STRING2" lexicographically in the current locale. • [ STRING1 > STRING2 ] True if "STRING1" sorts after
"STRING2" lexicographically in the current locale. • [ ARG1 OP ARG2 ] "OP" is one of -eq, -ne, -lt, -le, -gt or -ge.
IF
• [ ! EXPR ] True if EXPR is false.
• [ EXPR1 -a EXPR2 ] True if both EXPR1 and EXPR2 are true.
• [ EXPR1 -o EXPR2 ] True if either EXPR1 or EXPR2 is true.
IFif [ -f alert_ORCL.log ]; then echo "alert_ORCL.log exists." fi
if [ "$STATUS" == "PRIMARY" ]; then echo "Database status - ${STATUS}" else echo "Database not in correct mode (${STATUS})" exit 0; fi
if [ "$num_of_beer" -gt "3" -a "$num_of_beer" -lt "5" ]; then echo "You've worked hard enough for today." fi
FOR WHILE• for loop
for FILE in udump/*.trc do tkprof $FILE $FILE.out sort=exeela,fchela,prsela done
• while loop
COUNTER=0 while [ $COUNTER -lt 10 ]; do echo The counter is $COUNTER let COUNTER=COUNTER+1 done
INPUT• get data from user
stty -echo echo -n “Please enter sys password: " read pw stty echo
cont=n echo -n "Do you really want to continue? (y/n) " read cont if [ "$cont" != "y" ]; then echo "Quitting..." exit fi
OPEN PORTSecho "Scanning TCP ports..." for p in {1..1023} do (echo >/dev/tcp/localhost/$p) >/dev/null 2>&1 && echo "$p open" done
TOOLSET• grep - searching for text pattern • awk - script language for text-processing • ps - information on running processes • find - searching for files • sed - stream editor for filtering and transforming text • watch - execute a program periodically • wc - counting lines/chars • cat - join files
http://en.wikipedia.org/wiki/List_of_Unix_programs
GREP
• grep -f badORA.txt alert_PROD.log
badORA.txt ORA-00600 ORA-1653 ORA-00257
• more alert_PROD.log | grep ORA-• grep -r ORA- admin/*
AWK
• more /etc/passwd | grep oracle | awk -F: '{print $3}'
http://www.catonmat.net/blog/awk-one-liners-explained-part-one/
• AWK one liners
awk '1; { print "" }'
awk 'BEGIN { ORS="\n\n" }; 1'
awk '{ print } { print "" }'
• awk '{pattern + action}' {filenames}
• cat /etc/oratab | awk -F: '{if ($1=="ORCL") print $2 }'
ONE LINERS• http://www.pement.org/awk/awk1line.txt
!
!
• http://sed.sourceforge.net/sed1line.txt
# delete trailing whitespace (spaces, tabs) from end of each line awk '{sub(/[ \t]+$/, "")};1' ! # change "scarlet" or "ruby" or "puce" to "red" awk '{gsub(/scarlet|ruby|puce/, "red")}; 1' ! # print any line where field #5 is equal to "abc123" awk '$5 == "abc123"'
http://www.cheat-sheets.org/saved-copy/awk_quickref.pdf
PROCESS KILLING
!
• ps -ef | grep LOCAL=NO | awk '{print "kill -9 " $2}' | sh !
• ps -ef | grep LOCAL=NO | awk '{print $2}' | xargs kill -9 !
• killall oracle
WATCH• watch -tn 0.2 'ps -o cmd -C zabbix_server -C zabbix_agentd'
ALIAS
# alias name=value # alias name='command' # alias name='command arg1 arg2' # alias name='/path/to/script'
ALIAS• alias ..='cd ..' • alias ...='cd ../..' • alias rm='rm -i’ • alias mv="mv -iv" • alias grep="grep -i” • alias mkdir='mkdir -pv’ • alias df='df -H' • alias du='du -ch'
ALIAS• alias mount='mount |column -t’
!
!
• alias path='echo -e ${PATH//:/\\n}'
/dev/sda1 on / type ext4 (rw,errors=remount-ro)proc on /proc type proc (rw,noexec,nosuid,nodev)sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)none on /sys/fs/fuse/connections type fusectl (rw)none on /sys/kernel/debug type debugfs (rw)none on /sys/kernel/security type securityfs (rw)udev on /dev type devtmpfs (rw,mode=0755)devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
/usr/local/sbin/usr/local/bin/usr/sbin/usr/bin/sbin/bin/usr/games
ALIAS• alias ports='netstat -tulanp’
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:55255 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN - tcp 0 0 136.169.15.197:80 210.97.192.220:55544 TIME_WAIT - tcp 0 0 136.169.15.197:80 180.76.6.213:35700 TIME_WAIT - tcp 0 0 136.169.15.197:80 180.76.5.189:29922 TIME_WAIT - tcp 0 416 136.169.15.197:55255 87.110.183.173:24426 ESTABLISHED - tcp6 0 0 :::4949 :::* LISTEN - tcp6 0 0 :::55255 :::* LISTEN - tcp6 0 0 :::25565 :::* LISTEN 15500/java udp 0 0 127.0.0.1:11211 0.0.0.0:* - udp 0 0 0.0.0.0:68 0.0.0.0:* -
ALIAS• alias last="find . -type f -print0 | xargs -0 stat -c'%Y
:%y %12s %n' | sort -nr | cut -d: -f2- | head"
2013-12-10 16:53:07.000000000 +0200 31750 ./.bash_history2013-12-08 13:02:26.000000000 +0200 35 ./.lesshst2013-12-05 18:23:23.000000000 +0200 17962 ./.viminfo2013-10-31 16:51:20.000000000 +0200 6630 ./.ssh/known_hosts2013-08-21 12:58:51.000000000 +0300 273 ./pwd2013-08-20 16:01:42.000000000 +0300 432 ./.vim/.netrwhist2013-08-04 12:47:09.000000000 +0300 111891750 ./vegetarisms.sql.zip2013-01-14 08:18:13.000000000 +0200 163 ./.mysql_history2013-01-13 12:46:18.000000000 +0200 69632 ./skriesim_2013.xls2012-11-14 10:13:49.000000000 +0200 105 ./Scripts/ezermala.sh
ALIAS• alias cpp=“rsync --progress -ravz”
• alias most='du -hsx * | sort -rh | head -10’
• alias usage='du -ch 2> /dev/null |tail -1’
• alias untar='tar -zxvf ’
• alias nocomment='grep -Ev '\''^(#|$)'\'''
ALIASbu() { cp $@ [email protected]`date +%y%m%d`; echo "`date +%Y-%m-%d` backed up $PWD/$@" >> ~/.backups.log; } alias backup=bu
ALIAS
• alias instances='ps -ef | grep -v grep | grep dbw | cut -d _ -f 3‘
• alias sid=‘env | grep ORACLE_SID’
• alias alert='tail -100 $ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_$ORACLE_SID.log | more’
ALIAS
# alias
# unalias name # unalias -a
# \name
BONUS ALIAS
doskey /macrofile=\doskey.mac
doskey.mac
ls=dir $* /o/w cat=type $* rm=del $* lsl=dir $* /o/p quit=exit
BRACE• cp admin/ORCL/pfile/initORCL.ora{,.bak}
!
• cp admin/ORCL/pfile/initORCL.ora admin/ORCL/pfile/initORCL.ora.bak
• ls *.{trc,log}
HISTORY
SUDO !!Run last command from history
SPYhistory | awk '{print $2}' | sort | uniq -c | sort -n | tail
7 ps 9 exit 9 rm 12 more 19 sqlplus 20 vi 22 pwd 85 cd 109 df 185 ls
SPACE USAGE
SPACE USAGE• du
• find . -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
• find . -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
• find . -type d -exec du -sk {} \; | sort -n -k 1
SPACE USAGE# sh get_largest_files.sh /root 10 ![SIZE (BYTES)] [% OF DISK] [OWNER] [LAST MODIFIED ON] [FILE] !913608 0% root 2012-08-25 17:33:33 /root/test.h.gch 73164 0% root 2012-09-18 11:06:02 /root/mem.log 38772 0% root 2012-08-02 15:17:39 /root/install.log 22455 0% root 2012-09-17 21:09:15 /root/.bash_history 4890 0% root 2012-08-25 17:33:33 /root/a.out 170 0% root 2012-09-03 20:10:44 /root/userinfo.sh 385 0% root 2012-09-07 01:02:02 /root/test.sh 23 0% root 2012-08-25 16:58:19 /root/test.h 175 0% root 2012-08-25 17:04:46 /root/test.c.good 134 0% root 2012-08-25 17:32:57 /root/test.c !Total disk size: 23792652288 Bytes Total size occupied by these files: 1053776 Bytes [ 0% of Total Disc Space ] !*** Note: 0% represents less than 1% ***
http://www.thegeekscope.com/linux-script-to-find-largest-files/
LOAD GENERATION
LOAD GENERATION• cat /dev/urandom > md5sum &
• dd if=/dev/zero of=/dev/null
• while true; do /bin/true; done
• yes > /dev/null &
!
• dd if=/dev/random of=/tmp/largefile bs=1024 count=4000000
fulload() { dd if=/dev/zero of=/dev/null | dd if=/dev/zero of=/dev/null & }; fulload; read; killall dd
LOAD GENERATION• stress --cpu 2 --timeout 60
#!/bin/bash duration=120 # seconds instances=4 # cpus endtime=$(($(date +%s) + $duration)) for ((i=0; i<instances; i++)) do while (($(date +%s) < $endtime)); do :; done & done
http://linuxdrops.com/how-to-produce-high-cpu-load-memory-io-or-stress-test-a-linux-server/
FORK BOMB
:(){ :|:& };:
http://en.wikipedia.org/wiki/Fork_bomb
bomb() { bomb | bomb & }; bomb
REAL LIFE
say() { echo $1 | tee -a oraup_stat.txt } !up_db() { say ">>>>>>>> ORACLE_SID=$ORACLE_SID, celju augshaa..." sqlplus "/ as sysdba" <<EOF | tee -a oraup_stat.txt startup exit EOF } !if [ ! -f $ORACLE_HOME/bin/oracle ]; then say "Nav uzstadita ORACLE_HOME." exit else say "ORACLE_HOME=$ORACLE_HOME" fi !if [ $# -eq 0 ]; then say ">>> Nav noraditi parametri" die() fi !while [ $# -ne 0 ]; do export ORACLE_SID=$1 up_db done
TEST ENVIr paceltas sekojoshas datubaazes: ====================================================================================================================== DB SID Memory ORACLE_HOME and Product Version Actual Database Version ---------------------------------------------------------------------------------------------------------------------- APVDIG 1029 M /export/home/oracle/product/10.2.0 10.2.0.4.0 64-bit 10.2.0.4.0 64-bit ARHTEST 545 M /export/home/oracle/product/8.1.7 8.1.7.4.0 32-bit 8.1.7.4.0 32-bit EBRTEST 1895 M /export/home/oracle/product/11.2.0. 11.2.0.2.0 64-bit 11.2.0.2.0 64-bit ECRMUAT 1496 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit IDM 856 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit MBPTEST 1736 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit PRO11 515 M /export/home/oracle/product/11.2.0. 11.2.0.2.0 64-bit 11.2.0.2.0 64-bit RMSTEST 429 M /export/home/oracle/product/10.2.0 10.2.0.4.0 64-bit 10.2.0.4.0 64-bit ROSMET11 856 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit SMTEST 183 M /export/home/oracle/product/9.2.0.1 9.2.0.8.0 64-bit 9.2.0.8.0 64-bit SYMDW 1511 M /export/home/oracle/product/9.2.0.1 9.2.0.8.0 64-bit 9.2.0.8.0 64-bit SYMIBM 4903 M /export/home/oracle/product/9.2.0.1 9.2.0.8.0 64-bit 9.2.0.8.0 64-bit SYMUAT 6503 M /export/home/oracle/product/9.2.0.1 9.2.0.8.0 64-bit 9.2.0.8.0 64-bit TPENS11 828 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit TPENSNEW 588 M /export/home/oracle/product/11.1.0 11.1.0.7.0 64-bit 11.1.0.7.0 64-bit TRISD 960 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit UFTEST 381 M /export/home/oracle/product/10.2.0 10.2.0.4.0 64-bit 10.2.0.4.0 64-bit VOIS11T 884 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit ====================================================================================================================== Kopaa DB: 19 26472 M
_run_sql() { selekts=`sqlplus "/ as sysdba"<<endsql set head off; set serveroutput on; declare x number; i number; n number; comp varchar2(255); vers varchar2(255); c varchar2(1000); begin select count(*) into x from dba_views where view_name = 'DBA_REGISTRY' and rownum = 1; if x = 0 then -- versijas zem 9.0.0.0.0 c := 'select product, version from product_component_version where product like ''Oracle%Enterprise Edition%'' or product like ''Oracle%Server%'''; else -- versijas virs 9.0.0.0.0 c := 'select comp_name, version from dba_registry where comp_name like ''Oracle%Catalog Views%'''; end if; i := dbms_sql.open_cursor; dbms_sql.parse(i, c, dbms_sql.native); dbms_sql.define_column(i, 1, comp, 255); dbms_sql.define_column(i, 2, vers, 255); n := dbms_sql.execute(i); if dbms_sql.fetch_rows(i) > 0 then dbms_sql.column_value(i, 1, comp); dbms_sql.column_value(i, 2, vers); dbms_output.put_line('>>>'||comp||' '||vers||'<<<'); end if; dbms_sql.close_cursor(i); end; / select '>>>'||length(addr)*4||'<<<' from v\\$process where ROWNUM = 1; exit endsql` echo $selekts }
for pmonproc in `ps -ef -o pid,args | grep pmon | grep -v grep | grep -v sed | \ sed 's/\(.*\) ora_pmon_\(.*\)/\1 \2/' | sort -k 2`; do ! orastr=`_run_sql` orver=`echo $orastr | sed 's/.*Release \([^ ]*\) .*/\1/1'` orabinv=`file $env_orahome/bin/oracle | sed 's/.*ELF \(.*\)-bit .*/\1/'` typeset -L35 env_orahome typeset -R11 orver typeset -R11 prodshortver echo "$dbsid $mem2 M $env_orahome $orver $orabinv-bit\c" ! prodver=`echo $orastr | perl -ne 's/.*?>>>(.*?)<<<.*/\1/ && print'` prodshortver=`echo $prodver | sed 's/.* \([^ ]*\)/\1/'` prodbytes=`echo $orastr | perl -ne 's/.*>>>(.*?)<<<.*/\1/ && print'` vermat=`echo $prodshortver | awk '{if ($0 == $orver) {print 0} else {print 1}}'` echo " $prodshortver $prodbytes-bit" if [ $vermat -ne 0 -o $prodbytes -ne $orabinv ]; the echo " ^ ^ ^ ^ ^ ^ !!!!!!! NESAKRIIT VERSIJAS !!!!!!! ^ ^ ^ ^ ^ ^" fi !!done
env_orahome=`pargs -e "$pid" | grep "ORACLE_HOME=" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_ldlib=`pargs -e "$pid" | grep "LD_LIBRARY_PATH" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_path=`pargs -e "$pid" | grep ^PATH= | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_orabase=`pargs -e "$pid" | grep "ORACLE_BASE=" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_orasid=`pargs -e "$pid" | grep "ORACLE_SID=" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_nlsl=`pargs -e "$pid" | grep "NLS_LANG=" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` ! mem=`pmap "$pid" 2>/dev/null | tail -1 | awk '{ print $2 }'` mem2=`echo $mem | awk '{split($0,a,"K"); print a[1]}'` mem2=`expr $mem2 / 1024` memtot=`expr $memtot + $mem2` ! if [ "$env_path" -eq "" ]; then env_path="/usr/bin:/etc:/usr/ccs/bin/:/usr/local/bin:/usr/usb:.:"$env_orahome/bin fi PATH="$env_path" LD_LIBRARY_PATH="$env_ldlib" ORACLE_BASE="$env_orabase" ORACLE_HOME="$env_orahome" ORACLE_SID="$env_orasid" NLS_LANG="$env_nlsl" export PATH LD_LIBRARY_PATH ORACLE_BASE ORACLE_HOME ORACLE_SID NLS_LANG
oracle@test-dbs>pargs -e 9298 9298: ora_pmon_IDM envp[0]: SKGP_SPAWN_DIAG_PRE_EXEC_TS= envp[1]: SKGP_HIDDEN_ARGS= envp[2]: SKGP_SPAWN_DIAG_POST_FORK_TS= envp[3]: SKGP_SPAWN_DIAG_PRE_FORK_TS= envp[4]: ORACLE_SPAWNED_PROCESS=1 envp[5]: MANPATH=/usr/share/man:/usr/local/share/man/:/opt/local/share/man/ envp[6]: TERM=vt100 envp[7]: SHELL=/bin/bash envp[8]: NLS_LANG=AMERICAN_AMERICA.AL32UTF8 envp[9]: ODBC_HOME=/export/home/oracle/odbc64 envp[10]: USER=oracle envp[11]: LD_LIBRARY_PATH=/export/home/oracle/odbc64/lib:/export/home/oracle/product/11.2.0.3/lib:/usr/lib:/usr/openwin/lib:/usr/local/lib:/usr/dt/lib envp[12]: ORACLE_SID=IDM envp[13]: ORACLE_BASE=/export/home/oracle envp[14]: TNS_ADMIN=/export/home/oracle/product/11.2.0.3/network/admin envp[15]: PATH= envp[16]: PWD=/export/home/oracle/admin/IDM/pfile envp[17]: EDITOR=vi envp[18]: PS1=\\u@\\h> envp[19]: SHLVL=1 envp[20]: HOME=/export/home/oracle envp[21]: ODBCINI=/export/home/oracle/odbc64/odbc.ini envp[22]: DISPLAY=test-dbs:10.0 envp[23]: ORACLE_HOME=/export/home/oracle/product/11.2.0.3 envp[24]: _=/export/home/oracle/product/11.2.0.3/bin/sqlplus envp[25]: OLDPWD=/export/home/oracle/admin/IDM envp[26]: ORA_NET2_DESC=8,1
TOPP PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 3587 oracle 6522M 6518M cpu6 1 0 0:00:14 4.9% oracle/1 3587 :15 ora_j002_SYMUAT 2011 oracle 495M 479M sleep 59 0 0:01:42 1.7% oracle/71 2011 :42 oracleRMSTEST (LOCAL=NO) 15441 oracle 1501M 1483M sleep 49 0 0:16:26 1.3% oracle/1 15441 :26 oracleECRMUAT (LOCAL=NO) 18128 oracle 6504M 6500M sleep 5 0 1:51:00 1.1% oracle/1 18128 :00 oracleSYMUAT (LOCAL=NO) 27174 oracle 1500M 1483M sleep 48 0 0:01:21 0.5% oracle/1 27174 :22 oracleECRMUAT (LOCAL=NO) 3255 oracle 6510M 6507M sleep 32 0 0:00:02 0.2% oracle/1 23255 :42 ora_dia0_PRO11 20861 oracle 399M 392M sleep 57 0 34:05:31 0.1% tnslsnr/1 20861 :31 /export/home/oracle/product/9.2.0.1/bin/tnslsnr NET9 -inherit 2953 oracle 1500M 1482M sleep 26 0 0:00:13 0.1% oracle/1 2953 :13 oracleECRMUAT (LOCAL=NO) 3591 oracle 6504M 6500M sleep 52 0 0:00:00 0.1% oracle/1 3591 :00 ora_j004_SYMUAT 2014 oracle 446M 417M sleep 59 0 0:00:44 0.1% oracle/15 2014 :44 ora_lgwr_RMSTEST 2010 oracle 444M 426M sleep 1 0 0:01:44 0.1% oracle/258 2010 :44 ora_dbw0_RMSTEST 29354 oracle 1502M 1480M sleep 1 0 8:17:49 0.1% oracle/1 29354 :49 ora_dia0_ECRMUAT 2012 oracle 441M 425M sleep 1 0 0:01:11 0.0% oracle/258 2012 :11 ora_dbw1_RMSTEST 3522 oracle 4904M 4900M sleep 55 0 0:00:00 0.0% oracle/1 3522 :01 ora_j002_SYMIBM 3821 oracle 855M 625M sleep 59 0 0:00:00 0.0% oracle/1 3821 :00 ora_j001_IDM 8178 oracle 1898M 1879M sleep 56 0 0:23:22 0.0% oracle/1 8178 :22 ora_psp0_EBRTEST 8222 oracle 1900M 1881M sleep 59 0 3:50:20 0.0% oracle/1 8222 :21 ora_dia0_EBRTEST 3819 oracle 856M 626M sleep 59 0 0:00:00 0.0% oracle/1 3819 :00 ora_j000_IDM 3777 oracle 1899M 1883M sleep 53 0 0:00:00 0.0% oracle/1 3777 :00 ora_p019_EBRTEST 10501 oracle 6508M 6503M sleep 1 0 0:06:01 0.0% oracle/1 10501 :02 oracleSYMUAT (LOCAL=NO) 3769 oracle 1899M 1883M sleep 52 0 0:00:00 0.0% oracle/1 3769 :00 ora_p015_EBRTEST 3767 oracle 1899M 1883M sleep 52 0 0:00:00 0.0% oracle/1 3767 :00 ora_p014_EBRTEST 3761 oracle 1899M 1883M sleep 53 0 0:00:00 0.0% oracle/1 3761 :00 ora_p011_EBRTEST 3743 oracle 1899M 1883M sleep 54 0 0:00:00 0.0% oracle/1 3743 :00 ora_p002_EBRTEST 3771 oracle 1899M 1883M sleep 53 0 0:00:00 0.0% oracle/1 3771 :00 ora_p016_EBRTEST 3787 oracle 1899M 1883M sleep 54 0 0:00:00 0.0% oracle/1 3787 :00 ora_p024_EBRTEST 3589 oracle 6503M 6500M sleep 59 0 0:00:00 0.0% oracle/1 3589 :00 ora_j003_SYMUAT 3789 oracle 1899M 1883M sleep 54 0 0:00:00 0.0% oracle/1 3789 :00 ora_p025_EBRTEST 3813 oracle 1899M 1883M sleep 55 0 0:00:00 0.0% oracle/1 3813 :00 ora_p037_EBRTEST 3824 oracle 432M 416M sleep 58 0 0:00:00 0.0% oracle/1 3824 :00 ora_j000_RMSTEST Total: 689 processes, 9160 lwps, load averages: 1.63, 1.37, 1.29
#!/usr/bin/perl !$bg = $ARGV[0]; !$pid_list = ''; if ($bg ne '') { foreach (split(/\n/,`ps -ef | grep $bg | grep -v grep`)) { @top=split(/\s+/," ".$_); $pid_list = $pid_list.$top[2].","; print $_."\n"; } } if ($pid_list ne '') {$pid_list = " -p $pid_list";} !while (1 eq 1) { system("clear"); $pid=""; foreach (split(/\n/,`prstat -n 30 $pid_list 1 1`)) { @top=split(/\s+/," ".$_); $ps_line ="";$ps=""; if ($pid eq "PID") { @ps = split(/\n/,`ps -ef | grep $top[1] | grep -v grep`); $ps_line = substr($ps[0],9,6)." ".substr($ps[0],47,length($ps[0])); } else { $pid=$top[1]; } print $_." ".$ps_line."\n"; } `sleep 10`; }
ALERT.LOG• alert.log smart error checkMon Aug 19 17:36:13 2013 Errors in file /export/home/oracle/admin/ORCL/bdump/symuat_j003_20936.trc: ORA-12012: error on auto execute of job 4469813 ORA-20777: Kïûda -20777 ORA-20777: Kïûda 100 ORA-01403: no data foundprocedûrâ 0 KK_FG_RMSACCT_UNprocedûrâ 0 RP_AB_RMSACCT_UN ORA-06512: at "MASTER.ERROR_REG", line 70 ORA-06512: at "MASTER.RP_AB_RMSACCT_UN", line 292 ORA-06512: at line 1
• adrciADR Home = /u01/app/oracle/product/11.1.0/db_1/log/diag/rdbms/orclbi/orclbi: ***************************************************************************** INCIDENT_ID PROBLEM_KEY CREATE_TIME ----------------- ------------------------- --------------------------------- 3808 ORA 603 2007-06-18 21:35:49.322161 -07:00 3807 ORA 600 [4137] 2007-06-18 21:35:47.862114 -07:00 3805 ORA 600 [4136] 2007-06-18 21:35:25.012579 -07:00 3804 ORA 1578 2007-06-18 21:35:08.483156 -07:00 4 rows fetched
function get_errors ($cmd) { foreach (split("\n",$cmd) as $row) { preg_match( "/^(\S+\s+\S+\s+\d+)\s+\d+:\d+:\d+\s+(\d+)/x", $row, $match ); if (isset($match[0])) { $starting = 1; $grouping[] = $group; $group=$row."\n"; } else { if ($starting == 1) $group.=$row."\n"; } } ! $grouping_ora = array(); foreach ($grouping as $row) { $ja=0; if (check_group($row, "ORA-")) { ! if (check_group($row,"ORA-01554")) $ja=1; if (check_group($row,"ORA-3136")) $ja=1; if (check_group($row,"SYSTOR")) $ja=1; if ((check_group($row,"12333"))&&(check_group($row, "ORA-00600"))) $ja=1; if ($ja == 0) {$grouping_ora[]=trim($row);} } $ja=0; if (check_group($row, "Failure to extend rollback segment")) { $grouping_ora[]=trim($row); } $ja=0; if (check_group($row, "WARNING")) { if (check_group($row,"Starting ORACLE instance")) $ja=1; if ($ja == 0) {$grouping_ora[]=trim($row);} } } return $grouping_ora; }
DB BACKUP
• RMAN backup • one script • flexible configuration
# datu bazes statusa noskaidrosana STATUS_SQL="select DATABASE_ROLE FROM v\$database;\nexit\n" STATUS=`echo $STATUS_SQL | sqlplus -S / as sysdba | tail -2| head -1` !echo $STATUS !if [ "$STATUS" == "PRIMARY" ]; then echo "Database status - ${STATUS}" else echo "Database not in correct mode (${STATUS})" exit 0; fi
# # backup konfiguracija # !# --------------------------------------------------------------------------- # Ikdienas backup # FULL - pilnais backup (incremental 0) # INCR - inkrementals backup (incremental 1) !IKDIENAS_BACKUP.1=INCR IKDIENAS_BACKUP.2=INCR IKDIENAS_BACKUP.3=INCR IKDIENAS_BACKUP.4=INCR IKDIENAS_BACKUP.5=INCR IKDIENAS_BACKUP.6=INCR IKDIENAS_BACKUP.0=FULL !!# --------------------------------------------------------------------------- # Papildus backup. Tiek noteikts pec datuma (yyyy.mm.dd) # MONTH - izpildit backup menesea lentaas # QUART - izpildit backup kvartala lentaas # YEAR - izpildit backup gada lentaas !PAPILDUS_BACKUP.2012.05.31=MONTH PAPILDUS_BACKUP.2012.06.30=QUART PAPILDUS_BACKUP.2012.07.31=MONTH PAPILDUS_BACKUP.2012.08.31=MONTH PAPILDUS_BACKUP.2012.09.29=QUART PAPILDUS_BACKUP.2012.10.18=FULL PAPILDUS_BACKUP.2012.10.19=FULL
# check for business backup required PAPILDUS_BACKUP=`cat ${BACKUP_CFG} | grep -v "#" | grep "PAPILDUS_BACKUP.${D_DATE}" | awk 'BEGIN {FS="="} {print $2} '` IKDIENAS_BACKUP=`cat ${BACKUP_CFG} | grep -v "#" | grep "IKDIENAS_BACKUP.${D_WEEK_DAY}" | awk 'BEGIN {FS="="} {print $2} '` !BACKUP_TIME=28; !if [ "${PAPILDUS_BACKUP}" = "MONTH" ] ; then # start business backup BACKUP_TIME=120; BACKUP_TAPE=tdpo_monthly.opt BACKUP_LEVEL=0; BACKUP_TAG=${PAPILDUS_BACKUP}` date '+%Y%m' ` # elif [ "${IKDIENAS_BACKUP}" = "INCR" ] ; then BACKUP_TIME=28; BACKUP_TAPE=tdpo.opt PAPILDUS_BACKUP=INCR BACKUP_LEVEL=1; BACKUP_TAG=${PAPILDUS_BACKUP}` date '+%Y%m%d' ` # elif [ "${IKDIENAS_BACKUP}" = "FULL" ] ; then BACKUP_TIME=28; BACKUP_TAPE=tdpo.opt PAPILDUS_BACKUP=FULL BACKUP_LEVEL=0; BACKUP_TAG=${PAPILDUS_BACKUP}` date '+%Y%m%d' ` # fi
cat > $RUNCMD <<! connect target /; !# Parameters set section CONFIGURE ENCRYPTION FOR DATABASE OFF; CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 28 DAYS; CONFIGURE CONTROLFILE AUTOBACKUP ON; CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE 'sbt_tape' TO 'online_ctl_%F.ctl'; CONFIGURE CHANNEL DEVICE TYPE 'sbt_tape' PARMS 'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/${BACKUP_TAPE})' FORMAT 'backup_df_%t_%s_%p_%c' MAXPIECESIZE 100G; CONFIGURE DEVICE TYPE 'sbt_tape' PARALLELISM 2 BACKUP TYPE TO BACKUPSET; CONFIGURE DEFAULT DEVICE TYPE TO SBT; CONFIGURE ARCHIVELOG DELETION POLICY TO SHIPPED TO ALL STANDBY; !run { set duplex 1; backup incremental level ${BACKUP_LEVEL} database tag '${BACKUP_TAG}' KEEP UNTIL TIME 'sysdate + ${BACKUP_TIME}' LOGS; backup current controlfile; backup spfile format 'spfile_%d_%s_%T_dbid%I.ora' tag 'spfile ${BACKUP_TAG}'; backup as compressed backupset archivelog all delete input filesperset=20; } ! !# ${ORACLE_HOME}/bin/rman msgno cmdfile=${RUNCMD} log=${RMANLOG} 1>>${RUNLOG} 2>&1
# log apvienosans un parsesana cat ${RMANLOG} | sed 's/^RMAN-[0-9][0-9][0-9][0-9][0-9]\:.//g' >> ${RUNLOG} cat ${RMANLOG_DEL} | sed 's/^RMAN-[0-9][0-9][0-9][0-9][0-9]\:.//g' >> ${RUNLOG} cat ${RMANLOG_CTRL} | sed 's/^RMAN-[0-9][0-9][0-9][0-9][0-9]\:.//g' >> ${RUNLOG} !!# ORA- kludu pievienosana log faila sakumaa grep ORA- ${RUNLOG} > /tmp/ora_errors.log cat /tmp/ora_errors.log ${RUNLOG} > /tmp/tempfile ; mv /tmp/tempfile ${RUNLOG} !!cat ${RUNLOG} | ${MAILER} -s "BACKUP at `date` on ${ORACLE_SID} DONE `head -1 /tmp/ora_errors.log`" $MAILTO
HA WORKAROUND• AIX hardware • Oracle Database on ASM • Oracle DataGuard + Observer • Full HA tests • Service Request + AIX support
HA WORKAROUND!
• database is PHYSICAL STANDBY (RULE_1) • if dgmgrl show configuration hang (RULE_2) • has errors "DMON: Database ora is still
working on the task." (RULE_3) • manual failover
#setup home . /home/oracle/oracleDB.sh !RUN_NODE=secondary RUN_CHECK_NODE=primary RUN_SLEEP=10 LOG_SLEEP=60 LOG_TAIL=10 RUN_HOME=/home/oracle/local/dg
!INSTANCE_COUNT=`ps -ef | grep master.sh | grep -v grep | grep -v $$ | wc -l` if [ $INSTANCE_COUNT != 0 ]; then date echo "Instance $$" ps -ef | grep master.sh exit 0 fi
echo "RULE 1 check" !STATUS_SQL="select DATABASE_ROLE FROM v\$database;\nexit\n" STATUS=`echo $STATUS_SQL | sqlplus -S / as sysdba | tail -2| head -1` !if [ "$STATUS" == "PHYSICAL STANDBY" ]; then echo "\tOK" echo "\tDatabase status - ${STATUS}" else echo "\tDatabase not correct mode (${STATUS})" exit 0; fi
#!/bin/sh !#setup dgmgrl home . /home/oracle/oracleDB.sh !# start check configuration dgmgrl <<EOL connect sys/master; show configuration verbose; EOL
DG_STATUS.SH
echo "RULE 2 check" !NOW=`date` echo "\tCheck time - ${NOW}" echo "\tSleep time - ${RUN_SLEEP} sec" echo "\tSleeping..." sh $RUN_HOME/DG_status.sh > /dev/null 2>&1 & sleep $RUN_SLEEP !NOW=`date` echo "\tCheck time - ${NOW}" !PROCESS=`ps -ef | grep DG_status.sh | grep -v grep` PROCESS_COUNT=`ps -ef | grep DG_status.sh | grep -v grep | wc -l` !echo "\tProcess count - ${PROCESS_COUNT}" !if [ $PROCESS_COUNT != 0 ]; then echo "\tOK" echo "\t${PROCESS}" else echo "\tDG configuration check run normaly"; exit 0 fi
echo "RULE 3 check" !sleep $LOG_SLEEP !NOW=`date` echo "\tCheck time - ${NOW}" !# DG log faila atrasanas vietas noskaidrosana DIAG_DEST_SQL="select value from v\$parameter where name='diagnostic_dest';\nexit\n" DIAG_DEST=`echo $DIAG_DEST_SQL | sqlplus -S / as sysdba | tail -2| head -1` DGLOG_FILE="${DIAG_DEST}/diag/rdbms/${RUN_NODE}/${RUN_NODE}/trace/drc${RUN_NODE}.log" DGLOG_ERROR="DMON: Database ${RUN_CHECK_NODE} is still working on the task.” !if [ -f $DGLOG_FILE ]; then echo "\tDGLOG_FILE=${DGLOG_FILE}" echo "\tDGLOG_ERROR='${DGLOG_ERROR}'" echo "\tLOG_TAIL=${LOG_TAIL}" else echo "DG log file don't exists - ${DGLOG_FILE} (diagnostic_dest=${DIAG_DEST})" exit 0 fi !LINE=`tail -${LOG_TAIL} ${DGLOG_FILE} | grep "${DGLOG_ERROR}"` LINE_COUNT=`tail -${LOG_TAIL} ${DGLOG_FILE} | grep "${DGLOG_ERROR}" | wc -l` !if [ $LINE_COUNT != 0 ]; then echo "\tOK" echo "\tLINE_COUNT=${LINE_COUNT}" echo "\t${LINE}" else echo "\tNO errors in log file"; exit 0 fi
# te jataisa ir failover dgmgrl <<EOL connect sys/master; failover to ${RUN_NODE} EOL
FAILOVER
RESOURCES
MYSELF
• Signis Vāvere
• AS SEB banka
• database analyst
• http://lv.linkedin.com/in/signis
SQL> set pagesize 0 SQL> set head off SQL> with Mx as (select 60 as MaxWidth from dual) select decode ( sign(floor(MaxWidth /2)-rownum) , 1 , lpad( ' ', floor(MaxWidth /2)-(rownum-1)) || rpad( '*', 2*(rownum-1)+1, ' *') , lpad( '* * *', floor(MaxWidth/2)+3) ) from all_tables ,Mx where rownum < floor(MaxWidth /2) + 6;
HAVE YOURSELF A MERRY LITTLE CHRISTMAS
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * !35 rows selected.
