with Exadata / GI / ASM / opatch Must-have free scripts

Franky Weber Faust -- May 4th 2019

Must-have free scripts when working with Exadata / GI / ASM / opatch

Franky Weber Faust -- May 4th 2019

Must-have free scripts when working with Exadata / GI / ASM / opatch

4© The Pythian Group Inc., 2018 4

FRANKY WEBER FAUST

● Senior Oracle Database Consultant at Pythian and PagSeguro

● 28 years old● Writer at OTNLA and Lore Data Blog● Speaker at conferences around the

world● High Availability specialist● Performance researcher● Exadata, RAC, DataGuard,

GoldenGate● AcroYoga practicioner● Guitar player

loredata.com.br

© 2017 Pythian. Confidential 5

Years in Business

20Pythian Experts in 35 Countries

400+Current Clients

Globally

350+

Why this talk ?

● We sometimes expect things to be easy -- but they actually are not or they just don’t exist

● I am lazy and do not like doing repetitive and/or useless non added value tasks

● Fred developed all these scripts for our team to use and then shared with the community

● They have been improved a lot and also for OS different than Linux, a GI version we do not have handy, a new feature idea, etc ...

● All the scripts are free, simply bash made

asmdu.sh

ASM diskgroups sizes and usage

ASM disk usage : the old fashioned way

Non mirrored sizes Mirrored sizes

ASM disk usage : a new & better way with asmdu.sh

asmdu.sh : option -v for a verbose mode

Total non mirrored size

Non mirrored free size

Amount of free space required to be available in order to restore redundancy after one or more disk failures without affecting data availability

asmdu.sh : external redundancy

asmdu.sh : using different units

asmdu.sh : handling subdirectories

for D in `asmcmd ls DATA_DG`do asmcmd du +DATA_DG/$Ddone

asmdu.sh : a subdirectories example

asmdu.sh : another subdirectories example

asmdu.sh : option -h for the syntax

rac-status.shRAC / GI 11g,12c,18c,19c, Oracle Restart

resources status in a glimpse

© 2017 Pythian. Confidential 18

rac-status.sh : the old fashioned way

What is up ? What is supposed to be up ? Where is it up ? etc...

/home/oracle/pythian > ps -ef | grep pmon | wc -l37/home/oracle/pythian > olsnodesanexadatadb01anexadatadb02anexadatadb03anexadatadb04anexadatadb05anexadatadb06anexadatadb07anexadatadb08/home/oracle/pythian > crsctl stat res -p -w "TYPE = ora.database.type" | wc -l18096/home/oracle/pythian > crsctl stat res -v -w "TYPE = ora.database.type" | wc -l3410/home/oracle/pythian >

© 2017 Pythian. Confidential 19

rac-status.sh : a new & better way

© 2017 Pythian. Confidential 20

A La

rge

Impl

emen

tatio

n

© 2017 Pythian. Confidential 21

rac-status.sh : identifying issues

© 2017 Pythian. Confidential 22

rac-status.sh : different DB Types

© 2017 Pythian. Confidential 23

rac-status.sh : also shows the listeners

© 2017 Pythian. Confidential 24

rac-status.sh : also shows the listeners

© 2017 Pythian. Confidential 25

rac-status.sh : also shows the services

© 2017 Pythian. Confidential 26

rac-status.sh : see everything

© 2017 Pythian. Confidential 27

rac-status.sh : Oracle Restart (single instance)

© 2017 Pythian. Confidential 28

rac-status.sh : output customisation (with variables)

● Showing the databases, listeners and services may not be wanted at each execution

● Default output can be customised by modifying variables within the script

SHOW_DB="YES" # Databases #SHOW_DB="NO" SHOW_LSNR="YES" # Listeners#SHOW_LSNR="NO" SHOW_SVC="YES" # Services SHOW_SVC="NO"

● The last uncommented value wins

● The default shipped with the script shows the Databases and the Listeners

© 2017 Pythian. Confidential 29

rac-status.sh : output customisation (from command line)

● Using the command line

-d Revert the behavior defined by SHOW_DB-l Revert the behavior defined by SHOW_LSNR-s Revert the behavior defined by SHOW_SVC

● Show / hide everything

-a Show everything-n Show nothing

© 2017 Pythian. Confidential 30

rac-status.sh : make your maintenances idempotent

● Very useful to ensure being idempotent during a maintenance

● Save the status of the resources before the maintenance

$ ./rac-status.sh -a > status_before_maintenance

● Check the status of the resources after the maintenance

$ ./rac-status.sh -a > status_after_maintenance

● Any difference means something is not back to normal

$ diff status_before_maintenance status_after_maintenance

© 2017 Pythian. Confidential 31

rac-status.sh : output customisation (grep and ungrep)

● You can grep and ungrep from the output

-g Act as “grep” to grep a pattern from the output (key sensitive)-v Act as "grep -v" to ungrep from the output (key sensitive)

● Examples

$ ./rac-status.sh -g 12 # Only grep 12c databases

$ ./rac-status.sh -a -g Shut # Only the lines containing "Shut"

$ ./rac-status.sh -n -l -v Online # Only the listeners that are not Online

© 2017 Pythian. Confidential 32

rac-status.sh : output customisation (colors)

● Option -u for an Uncolored output

© 2017 Pythian. Confidential 33

rac-status.sh : output customisation (colors)

● Option -r to Revert the colors

© 2017 Pythian. Confidential 34

rac-status.sh : environment

● rac-status.sh uses oraenv to set the ASM environment and the uses crsctl

● Option -e if you do not want to use oraenv but the current environment

● You can set USE_ORAENV="NO" in the script to always use the current environment and never use oraenv

© 2017 Pythian. Confidential 35

rac-status.sh : option -h for help

© 2017 Pythian. Confidential 36

rac-status.sh : patch progression

rac-mon.sha GI 11g,12c,18c,19c and Oracle Restart

monitoring tool based on rac-status.sh

rac-mon.sh : how it works

● A monitoring tool based on rac-status.sh

● First, it saves a reference status of your cluster (when everything works as expected)

$ ./rac-status.sh -a > rac-status_reference

● At every execution, it takes the current status of your cluster

$ ./rac-status.sh -a > current_status

● Compare it with the reference

$ diff rac-status_reference current_status

rac-mon.sh : pre requisites

● The rac-status.sh script

RACSTATUS=~/rac-status.sh

● A file to save the reference status of your cluster

REFERENCE=~/rac-status_reference

rac-mon.sh : a first execution

rac-mon.sh : when errros are detected

rac-mon.sh : sending emails on error

rac-mon.sh : sending emails on success

EMAIL_ON_FAILURE="No" # Yes to always send emailsEMAIL_ON_SUCCESS="No" # Yes to always send emails

● Always send emails with no option

rac-mon.sh : email configuration

EMAILTO="youremail@company.com" # The email to send the alert to

● Email address(es)

● Succes / Failure Emails Subjects

FAILURE_SUBJECT="Error : Cluster status at "`date` # Subject of the email sentSUCCESS_SUBJECT="OK : Cluster status at "`date` # Subject of the email sent

rac-mon.sh : recreate the reference file

● You need to recreate the reference file if you change something on your cluster on purpose

$ ./rac-status.sh -a > ./rac-status_reference

● A quick test

$ ./rac-mon.sh No change has been identified across the cluster, all good !$

rac-mon.sh : cron it

● You may want to cron it to monitor your cluster resources on a regular basis

*/5 * * * * /home/oracle/rac-mon.sh -e >> /var/log/rac-mon.log 2>&1

● It needs to be scheduled on only 1 node of the cluster

rac-mon.sh : option -h for help

cell-status.shExadata Storage Server (Cell)

Bare Metal, OVM, ExaCCStorage overview made simple

cell-status.sh : the old fashioned way

cell-status.sh : a new & better way

cell-status.sh : a new & better way

cell-status.sh : a new & better way

lspatches.shAn Oracle patch reporting tool

© 2017 Pythian. Confidential 54

lspatches.sh : an Oracle patch reporting tool

● Opatch is a nice and easy tool to install and rollback patches

● But it lacks of reporting capabilities

● opatch -lspatches does not provide remote patch information

© 2017 Pythian. Confidential 55

● lsinventory -all_nodes report many lines

lspatches.sh : questions a DBA has to answer

● 472 lines for 1 Home on 2 nodes then almost 2000 lines for 8 nodes

● And you may have dozen Homes then thousands of lines

Is patch 123456 installed on all the nodes ?

Is there a missing patch on a Home ?

© 2017 Pythian. Confidential 56

lspatches.sh : a sample output

© 2017 Pythian. Confidential 57

lspatches.sh : how that works ?

© 2017 Pythian. Confidential 58

lspatches.sh : analyze the Home(s) you want

© 2017 Pythian. Confidential 59

lspatches.sh : show before proceeding (1)

© 2017 Pythian. Confidential 60

lspatches.sh : show before proceeding (2)

© 2017 Pythian. Confidential 61

lspatches.sh : input and output files are supported (1)

© 2017 Pythian. Confidential 62

lspatches.sh : input and output files are supported (2)

© 2017 Pythian. Confidential 63

lspatches.sh : input and output files are supported (3)

© 2017 Pythian. Confidential 64

lspatches.sh : local and remote

● On RAC system, default is remote patch information (-all_nodes or opatchauto -remote)

● Option -l can be used to gather local patch information only

● On non-RAC system, it goes local automatically

© 2017 Pythian. Confidential 65

lspatches.sh : option -h for help

exa-versions.shExadata components versions in a glimpse

exa-versions.sh : the old fashioned way

[root@myclusterdb01 ~]# dcli -g ~/dbs_group -l root imageinfo -ver

myclusterdb01: 12.1.2.2.1.160119

myclusterdb02: 12.1.2.2.1.160119

myclusterdb03: 12.1.2.2.1.160119

myclusterdb04: 12.1.2.2.1.160119

[root@myclusterdb01 ~]#

● dcli is a very good tool to launch many commands in parallel across all the nodes of a cluster

● How to deal with 8 DB nodes, 14 storage cells and 2 IB Switches ?

● exa-versions goal is to clearly show all the versions of each Exadata component

● Syntax for IB Switches is different than DB Servers and Cells

● Are your dbs_group , cell_group and ib_group files up to date ?

exa-versions.sh : a new & better way

exa-versions.sh : detect a status = failure

[root@ahalfrack_db02 ~]# imageinfo

Kernel version: 2.6.39-400.284.1.el6uek.x86_64 #1 SMP Mon Sep 12 19:26:25 PDT 2016 x86_64Image kernel version: 4.1.12-61.57.2.el6uek

Image version: 12.2.1.1.3.171017

Image activated: 2017-12-28 11:42:32 -0600

Image status: failure

System partition on device: /dev/mapper/VGExaDb-LVDbSys1

[root@ahalfrack_db02 ~]#

exa-versions.sh : options

exa-versions.sh : option -h for the syntax

exa-racklayout.sha whole Exadata Rack Layout overview in a glimpse

exa-racklayout.sh : the old fashioned way

● What is the IP of my database server ILOM ?

● What is the Cisco switch IP ?

● What are the cells IP ?

● How many IB Switches do I have in my Rack ?

$ cat /etc/hosts

$ vi or grep /opt/oracle.SupportTools/onecommand/databasemachine.xml

exa-racklayout.sh : a new & better way

exa-racklayout.sh : a new & better way

● -s option to hide the empty U

many-exa-racklayout.shexa-racklayout.sh on many hosts

● executes exa-racklayout.sh on many hosts

LIST="oracle@AN_IP

oracle@ANOTHER_IP"

● SSH keys needed

rac-on_all_db.shExecute a SQL on all databases running on a cluster

rac-on_all_db.sh : a real life example

rac-on_all_db.sh : the old fashioned way

/home/oracle/pythian> ssh anexadata_dbXX

/home/oracle/pythian> . oraenv <<< dbYY

/home/oracle/pythian> sqlplus / as sysdba

/home/oracle/pythian> select name, value . . .

/home/oracle/pythian>

for every database from rac-status.sh do

done

rac-on_all_db.sh : a new & better way

rac-on_all_db.sh : how to customize the queries

exa-howsmart.shHow much your databases use the Exadata features ?

exa-howsmart.sh

exa-howsmart.sh

Where to find the scripts?Fred Denis on Git hub

Thank You Questions ?

© 2018 Pythian. Confidential 87

© 2017 Pythian. Confidential 88

STAY IN TOUCH!

● E-mail: faust@pythian.com● Blog: http://loredata.com.br/blog● Facebook: https://facebook.com/08Franky.Weber● Instagram: https://www.instagram.com/frankyweber/● Twitter: https://twitter.com/frankyweber● LinkedIn: https://linkedin.com/in/frankyweber/en● Oracle ACE profile: https://apex.oracle.com/pls/apex/f?p=19297:4:::NO:4:P4_ID:17002