95
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | EXAchk Health Checks & Best Practices for Exadata

EXAchk for Exadata Presentation

Embed Size (px)

Citation preview

Page 1: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Health Checks amp Best Practices for Exadata

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The following is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

2

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

3

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

4

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Oracle Engineered Systems

Configuration specific Health Checks

Saves Time Saves Money

Why EXAchk

5

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

ndashAutomated risk identification and proactive notification before business is impacted

ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base

ndashRuns in your environment ndash no need to send anything to Oracle

ndashScheduled email Health Check reports

ndashFindings can be integrated into other tools of choice

Why EXAchk

Engineered Systems

Non Engineered Systems

EXAchk

ORAchk

Common Framework

Lightweight amp non intrusive Health Check framework for the Oracle stack

6

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Engineered Systems

bull Exadata DBM is engineered for best fastest and stable database performance and availability

bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles

bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk

7

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Configuration specific Health Checks

bull EXAchk provides configuration specific up to date health check across the entire DBM stack

o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it

o Exadata DB GI ASM critical issues list specific for the environment

o Exadata full stack software planner

o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices

o Automated scheduling ability and automatic difference identification between runs with email notification

8

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 2: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The following is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

2

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

3

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

4

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Oracle Engineered Systems

Configuration specific Health Checks

Saves Time Saves Money

Why EXAchk

5

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

ndashAutomated risk identification and proactive notification before business is impacted

ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base

ndashRuns in your environment ndash no need to send anything to Oracle

ndashScheduled email Health Check reports

ndashFindings can be integrated into other tools of choice

Why EXAchk

Engineered Systems

Non Engineered Systems

EXAchk

ORAchk

Common Framework

Lightweight amp non intrusive Health Check framework for the Oracle stack

6

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Engineered Systems

bull Exadata DBM is engineered for best fastest and stable database performance and availability

bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles

bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk

7

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Configuration specific Health Checks

bull EXAchk provides configuration specific up to date health check across the entire DBM stack

o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it

o Exadata DB GI ASM critical issues list specific for the environment

o Exadata full stack software planner

o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices

o Automated scheduling ability and automatic difference identification between runs with email notification

8

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 3: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

3

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

4

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Oracle Engineered Systems

Configuration specific Health Checks

Saves Time Saves Money

Why EXAchk

5

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

ndashAutomated risk identification and proactive notification before business is impacted

ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base

ndashRuns in your environment ndash no need to send anything to Oracle

ndashScheduled email Health Check reports

ndashFindings can be integrated into other tools of choice

Why EXAchk

Engineered Systems

Non Engineered Systems

EXAchk

ORAchk

Common Framework

Lightweight amp non intrusive Health Check framework for the Oracle stack

6

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Engineered Systems

bull Exadata DBM is engineered for best fastest and stable database performance and availability

bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles

bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk

7

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Configuration specific Health Checks

bull EXAchk provides configuration specific up to date health check across the entire DBM stack

o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it

o Exadata DB GI ASM critical issues list specific for the environment

o Exadata full stack software planner

o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices

o Automated scheduling ability and automatic difference identification between runs with email notification

8

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 4: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

4

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Oracle Engineered Systems

Configuration specific Health Checks

Saves Time Saves Money

Why EXAchk

5

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

ndashAutomated risk identification and proactive notification before business is impacted

ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base

ndashRuns in your environment ndash no need to send anything to Oracle

ndashScheduled email Health Check reports

ndashFindings can be integrated into other tools of choice

Why EXAchk

Engineered Systems

Non Engineered Systems

EXAchk

ORAchk

Common Framework

Lightweight amp non intrusive Health Check framework for the Oracle stack

6

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Engineered Systems

bull Exadata DBM is engineered for best fastest and stable database performance and availability

bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles

bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk

7

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Configuration specific Health Checks

bull EXAchk provides configuration specific up to date health check across the entire DBM stack

o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it

o Exadata DB GI ASM critical issues list specific for the environment

o Exadata full stack software planner

o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices

o Automated scheduling ability and automatic difference identification between runs with email notification

8

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 5: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Oracle Engineered Systems

Configuration specific Health Checks

Saves Time Saves Money

Why EXAchk

5

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

ndashAutomated risk identification and proactive notification before business is impacted

ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base

ndashRuns in your environment ndash no need to send anything to Oracle

ndashScheduled email Health Check reports

ndashFindings can be integrated into other tools of choice

Why EXAchk

Engineered Systems

Non Engineered Systems

EXAchk

ORAchk

Common Framework

Lightweight amp non intrusive Health Check framework for the Oracle stack

6

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Engineered Systems

bull Exadata DBM is engineered for best fastest and stable database performance and availability

bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles

bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk

7

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Configuration specific Health Checks

bull EXAchk provides configuration specific up to date health check across the entire DBM stack

o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it

o Exadata DB GI ASM critical issues list specific for the environment

o Exadata full stack software planner

o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices

o Automated scheduling ability and automatic difference identification between runs with email notification

8

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 6: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

ndashAutomated risk identification and proactive notification before business is impacted

ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base

ndashRuns in your environment ndash no need to send anything to Oracle

ndashScheduled email Health Check reports

ndashFindings can be integrated into other tools of choice

Why EXAchk

Engineered Systems

Non Engineered Systems

EXAchk

ORAchk

Common Framework

Lightweight amp non intrusive Health Check framework for the Oracle stack

6

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Engineered Systems

bull Exadata DBM is engineered for best fastest and stable database performance and availability

bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles

bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk

7

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Configuration specific Health Checks

bull EXAchk provides configuration specific up to date health check across the entire DBM stack

o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it

o Exadata DB GI ASM critical issues list specific for the environment

o Exadata full stack software planner

o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices

o Automated scheduling ability and automatic difference identification between runs with email notification

8

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 7: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Engineered to work together with Engineered Systems

bull Exadata DBM is engineered for best fastest and stable database performance and availability

bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles

bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk

7

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Configuration specific Health Checks

bull EXAchk provides configuration specific up to date health check across the entire DBM stack

o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it

o Exadata DB GI ASM critical issues list specific for the environment

o Exadata full stack software planner

o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices

o Automated scheduling ability and automatic difference identification between runs with email notification

8

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 8: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Configuration specific Health Checks

bull EXAchk provides configuration specific up to date health check across the entire DBM stack

o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it

o Exadata DB GI ASM critical issues list specific for the environment

o Exadata full stack software planner

o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices

o Automated scheduling ability and automatic difference identification between runs with email notification

8

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 9: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Saves Time

bull No time required on your part to develop checks (unless you want tohellipcovered later)

o Hundreds of checks for best practices are provided

bull Configuration is minimal

bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand

bull Executes quickly

o Parallelized execution keeps typical wall clock time low

bull Provides prioritized categorized report of findings focused on high priority issues first

bull Provides self service guidance for each check

bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications

9

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 10: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Saves Money

bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract

oNo extra cost for EXAchk included within your support fee

oAvailable to download from My Oracle Support (10709541)

bull No development costs required by your organization

oHealth Checks developed and provided for free by Oracle

oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks

10

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 11: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

11

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 12: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM

bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production

o Interim releases may be released sooner if significant Critical Issues are discovered

o Also timed to coincide with major software releases and or hardware changes

bull Maintains backward compatibility

Development Methodology

12

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 13: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Continuous improvement from

o Enhancement requests feedback from customers

o Exadata MAA Best Practices team

o Critical Issues discovery

o Internal testing Proofs of Concepts amp Support Partners

Development Methodology

13

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 14: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Development Methodology

bull EXAchk does not duplicate the function of other established tools

o Enterprise Manager for continuous monitoring

o AWR reports

o SQL tuning tools

o Advanced diagnostic gatherers

o Automated Service Request

o Storage server alert mechanisms

bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)

14

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 15: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

15

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 16: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Recommended Usage ndash Automated Risk Notification

1) Schedule EXAchk to run in daemon mode weekly or daily and email report

2) Identify actions easily by viewing automated comparison of previous runs

3) Act on recommendations

16

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 17: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Schedule EXAchk

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000

AUTORUN_SCHEDULE =

17 17

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 18: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull First email will contain the HTML report

Email Notification

18

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 19: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Health score

bull Summary of EXAchk run

bull Table of content

bull Controls for report features

bull Findings

bull Recommendations

19

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 20: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Report

bull Table of content

bull Controls for report features bull Checks which passed are not shown by default

bull Quickly show or hide checks based on their status

bull Show or hide major sections of the report

bull Collapse or expand check findings

bull Show check Ids

bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis

bull Display a printable view

20

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 21: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View EXAchk Findings

bull Check status

bull Type of Check

bull Check Message

bull Where the check was run

bull Link to expand details

21

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 22: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull What to do to solve the problem

bull Links to relevant Knowledge docs

bull Where recommendation applies

bull Where problem doesnrsquot apply

bull Example of data the recommendation is based on

View Recommendations

22

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 23: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Review MAA Score Card

bull Critical Issues in MAA Scorecard

o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo

bull Software version mapping table

bull Installed software versions checked for noncurrent or incompatible feature usage

(Many other areas of report shown in demo sample later)

23

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 24: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Subsequent emails compare results to previous run

bull Easily see if something has changed

bull Email attachment has

o Latest report

o Previous report

o Diff Report

Email Notification

24

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 25: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Diff overview

bull Summary of this run vs previous

Diff Output

25

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 26: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Differences between each run

Diff Output

26

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 27: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Upgrade or Patching

o Detects all databases registered in the clusterware automatically

o Presents list of databases to check

o Pre Upgrade Run during upgrade planning phase

Asks which version you plan to upgrade to

o Post Upgrade Run after upgrade

bull Other typical times to run EXAchk

o Machine moves

o Hardware failure repair

o Problem troubleshooting

o In addition to go-live testing

bull Other Key My Oracle Support notes

o MOS 7575521 publishes the MAA Exadata Best Practices

o MOS 8888281 is the master Exadata version reference

o Exadata Critical Issues (Doc ID 12700941)

Other Recommended Usage

exachk -u ndasho pre

exachk -u ndasho post

27

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 28: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

28

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 29: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Report Demo

29

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 30: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

30

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 31: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Supported Environments

bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems

o Big Data Appliance

o Exadata (V2 and above HP hardware based V1 systems are not covered)

o Exalogic

o Exalytics

o SuperCluster

o Zero Data Loss Recovery Appliance

31

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 32: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Included in base image and latest OEDA

2 Download latest version from My Oracle Support (install lt 1 min)

3 Auto update when later version available

Roll Out amp Maintain EXAchk

32

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 33: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

1 Download the exachk bundle from MOS note 10709541 to your local machine

2 Unzip it and transfer exachkzip to a directory on the target Engineered System

3 Unzip exackzip

o As owner of oracle database or grid home

Installation

33

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 34: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Run as root (recommended)

o EXAchk will su to lower privileged owners of RDBMS or grid homes

o To specify a user other that root for these situations

bull Run as RDBMS or GRID Home Owner

o User must be able to switch to root for root level checks ndash several options

1 Provide the root userid password at prompts

or

2 Set up sudo

or

3 Pre-configure passwordless SSH connectivity

Which User to Run EXAchk as

Connect via SSH amp

Run Checks on Default User

Change User By exporting user id in this Environment Variable

Exadata Storage Server

root RAT_CELL_SSH_USER

InfiniBand switches

root (when run as root)

RAT_IBSWITCH_USER nm2user (when run as other user)

34

Note You may only choose from the provided lower privileged accounts

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 35: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Option 1 With internet connection

o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support

o Can also be specifically triggered with

Option 2 No internet connection

a) Download the latest exachkzip to a shared network staging location

b) Set environment variable RAT_UPGRADE_LOC

c) Next time exachk script is started it will prompt to allow it to upgrade itself

Maintaining EXAchk

exachk -download

export RAT_UPGRADE_LOC=ltstaging dirgt

35

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 36: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Automated Usage with the Daemon

36

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 37: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options When What amp Who to Tell

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

AUTORUN_SCHEDULE

bull Schedule when exachk will be run

bull Hour day of month month of year amp day of week

bull Comma separate multiple values for same timeframe

bull Wildcard

Hour (0 ndash 23)

Day of month (1 ndash 31)

Month (1 ndash 12)

Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)

exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo

AUTORUN_FLAGS

bull Command line options to be passed through to exachk run

exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo

NOTIFICATION_EMAIL

bull Comma separated list of emails to send daemon notifications to

exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo

37

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 38: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Maintenance

exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo

COLLECTION_RETENTION

bull Number of days to keep files created by scheduled run files older than this will be deleted

exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo

PASSWORD_CHECK_INTERVAL

bull Frequency in hours of password validation

bull When found invalid daemon stops amp notifies via log amp email

exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo

38

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 39: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Set Daemon Options Multiple IDs

bull Only one Daemon process should be used across a database cluster

o Multiple daemon profiles can be configured using IDs

o Allows for multiple different types of exachk runs

exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo

39

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 40: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get Daemon Option

bull Find a specific daemon option which has been set

bull Use with or without ID

exachk ndashd ltIDgt ndashget ltoptiongt

Get value of particular option for specific id

exachk ndashget ltoptiongt

Get value of particular option for all IDs

40

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 41: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Get All Daemon Options

bull Find any Daemon options which have been set

bull Used with or without ID

exachk ndashd ltIDgt ndashget all

Get value of all options for specific id

exachk ndashget all

Get value of all options for all IDs

41

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 42: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Start Daemon

exachk ndashd start

1 Start the daemon

2 Follow the prompts

42

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 43: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Information

exachk ndashd status

Check if the daemon is running

exachk ndashd info

Get more detailed information about the daemon

43

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 44: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Next Autorun

exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun

Find when the next auto run will happen Find when the next auto run will happen for a specific ID

44

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 45: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Daemon Stopping

exachk ndashd stop_client exachk ndashd stop

Stop an exachk run mid-flow Stop the daemon

45

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 46: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Auto Restart Daemon

bull Restart daemon if machine restarts o Uses passwordless ssh

user equivalence to root

exachk ndashinitsetup

46

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 47: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Restart status of daemon can be queried If you want to remove restart settings

Auto Restart Daemon Check Status amp Remove Auto Restart

exachk ndashinitcheck exachk ndashinitrmsetup

47

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 48: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

With or Without the Daemon

bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values

o To avoid connecting to the daemon use ndashnodaemon

o To only run exachk when the daemon is running use ndashdaemon

exachk ndashnodaemon

exachk ndashdaemon

48

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 49: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Email Notifications bull Run in ad hoc mode and receive html report via email

bull Verify email configuration function

bull Use ndashtestemail all to use email addresses stored in daemon configuration

exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom

exachk -testemail all

49

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 50: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Additional EXAchk Capabilities amp Features

50

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 51: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Health Check Catalog bull Bundled within zip download

bull Audit_Checks_Report_Exachkhtml

bull Contains all published checks

bull Filterable amp searchable bull Engineered System

bull Profiles

bull Alert Level

bull Release Check Authored

bull Platforms

bull Look up check id without running report

51

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 52: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Database Checks

bull EXAchk runs checks against all database nodes in the cluster by default

o To specify only a subset of nodes use

o Only local node

bull Automatically discovers all databases and prompts for which should be checked

o Do not prompt but run all checks on all discovered database

o Do not prompt and skip all database related checks

o Only run checks against a subset of databases

exachk ndashclusternodes ltnode_1gtltnode_2gt

exachk ndashlocalonly

exachk ndashdball

exachk ndashdbnone

exachk ndashdbnames ltdb_1gtltdb_2gt

52

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 53: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Checks for Cells amp Switches

bull Run on a subset of cells using ndashcells

bull Run on a subset of switches using -ibswitches

exachk ndashcells ltcell_1gtltcell_2gt

exachk ndashibswitches ltswitch_1gtltswitch_2gt

53

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 54: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Virtualization

1 Run from dom0 on first compute to cover all dom0s cells and switches

o Install EXAchk into the management domain (DOM0) amp run as root

o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all

o If a single machine has been separated into multiple management domains

bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains

2 Run once on each virtual cluster

54

exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 55: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

EXAchk Output

bull EXAchk will output the collection results to the directory it is run from

o Output can be directed to a different directory with ndashoutput

bull Output will be directory and a zip of the same name

exachk ndashoutput ltOUTPUT_DIRgt

o log various log files

o outfiles collection results checks are based on

o Reports subreports used to build the main report

o Scripts scripts used during collection

o Upload files for upload of collection into database or integration into other tools

o exachk_html Main HTML report output

55

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 56: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Temporary Working Directory

bull EXAchk will create temporary files during execution

bull Default location is $HOME of the user EXAchk is running as

bull Location can be changed by setting RAT_TMPDIR

bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file

export RAT_TMPDIR=ltTEMP_DIRgt

exachk

ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh

56

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 57: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Parallel Execution

bull Database collections are executed in parallel

bull The default number of slave processes used is calculated automatically

bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax

bull Parallel execution can be disabled altogether if required with -dbserial

exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax

exachk ndashdbserial

57

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 58: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Tagging Merging amp Comparing Reports

bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml

bull Tag collections so output contains another word to help differentiate it

exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml

bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files

bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce

bull Compare collections with ndashdiff

exachk ndashmerge ltcollection_1gtltcollection_2gt

exachk ndashmerge ltcollection_1gtltcollection_2gt -force

exachk ndashdiff ltcollection_1gtltcollection_2gt

exachk ndashtag lttag_namegt

58

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 59: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Profiles provide logical grouping of checks which are about similar topics

bull Run only checks in a specific profile

bull Run everything except checks in a specific profile

Profiles

exachk ndashprofile ltprofilegt

exachk ndashexcludeprofile ltprofilegt

Profile Description asm ASM Checks avdf Audit Vault Configuration checks

clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)

No cross node checks corroborate Exadata checks needs further review by user to determine

pass or fail dba DBA Checks ebs Oracle E-Business Suite checks

eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks

goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems

maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking

platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks

solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks

sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml

59

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 60: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Run or exclude individual checks

bull Granular control to execute or exclude a single check

bull Ideal for testing new checks or troubleshooting

bull Run only specific check(s)

bull Exclude a specific check

bull Find check id either from report or Health Check Catalog

exachk -check ltcheck_id_1gtltcheck_id_2gt

exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt

60

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 61: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

61

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 62: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Dashboard

62

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 63: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Dashboard Filters

Filter by Interval

Filter by configurable

business units Filter by systems

Click on color coded area to drill down

63

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 64: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures amp Warnings

Click to see the recommendation details

64

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 65: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Failures

Click to drill into failures

65

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 66: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Most Warnings

Click to drill into warnings

66

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 67: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

No difference OR No regression failed in current collection

At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection

At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection

Non clickable green flag - Preceding collection not found

Recent Collections

Health Score Warning count

Fail count Info count Pass count

Ignore count

67

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 68: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Collection Link

68

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 69: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

View Collection

Recommendation

69

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 70: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks bull Use as a Health Checking Platform

bull You write your own business specific User Defined Checks

bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool

bull OS or SQL logic

bull Generates user_defined_checksxml sample in install directory

bull Utilizes framework features such as result output email notification CM storage etc

70

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 71: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

User Defined Checks

bull Have their own profile user_defined_checks

bull Can be excluded exachk -excludeprofile user_defined_checks

bull Have their own section of the report

exachk -profile user_defined_checks

bull Can be run on their own

71

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 72: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Manager Setup

1 Ensure APEX 42 is installed amp configured

2 Follow Collection Manager installation in Collection Manager User Guide

3 Login to Collection Manager Application via a URL like the following

o Format will depend on choices during installation

httphostnameportapexfp=ApplicationID

httphostnameportplsapexfp=ApplicationID

72

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 73: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Collection Storage Table

bull Collection zip files are stored in the RCA13_DOCS table

o Already created during collection manager installation

bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts

bull Get current values with bull Unset values with

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD

exachk -getdbupload

73

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 74: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Store DB Upload Variables in Wallet bull Set all with

bull Set specific variables by specifying comma separated list

bull Unset all with

bull Check if variables are set correctly

exachk -setdbupload all

exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD

exachk -unsetdbupload all

exachk -checkdbupload

74

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 75: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

75

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 76: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Oracle Enterprise Manager Cloud Control 131

76

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 77: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Enterprise Manager Integration

bullCheck results integrated into EM compliance framework via plugin

bullView results in native EM compliance dashboards

bullRelated checks grouped into compliance standards

bullView targets checked violations amp average score

bullDrill down into compliance standard to see individual check results

bullView break down by target

77

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 78: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support

bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware

and software are at the supported level as described in Supported Hardware and Software Versions

o All Engineered System plug-ins should be deployed

o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system

o Expect package should be installed on the hosts

Enterprise Manager Plugin Perquisites

Hardware Types Supported By Plugin

Exadata (physical configuration only) Yes

Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No

78

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 79: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Setting Up Enterprise Manager EXAchk Plugin

bull The plugin is already installed by default with Enterprise Manager 131

1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature

2 Provision the plugin to setup the EXAchk daemon

79

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 80: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk

bull After selected this will launch the provisioning wizard choose the system type

Provision EXAchk

80

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 81: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Provide new or select existing credentials

bull Specify install location

bull Select when daemon should be run

81

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 82: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Provision EXAchk

bull Verify CRS Home

bull Finally choose Cell amp InfiniBand configuration

82

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 83: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Drill into applicable standard and view individaul checks amp target status

View Results by Compliance Standard

Filter by Exachkrdquo

Click individual checks for recommendation details

83

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 84: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With 3rd Party Tools

84

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 85: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

JSON Output to Integrate with Kibana Elastic Search etc

85

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 86: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull EXAchk results are also output in JSON format in the upload directory of the collection

bull Writing JSON Results With syslog

1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg

2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo

3 You can verify syslog configuration by running the following commands

4 Then verify in your configured message location (eg varadmmessages) that each test message was written

JSON Result Output

exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo

86

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 87: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Integrating With Your Own Application

87

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 88: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Custom App Integration

bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results

1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables

2 Tell EXAchk about these tables

exachk -setdbupload all

bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App

88

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 89: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Agenda

1

2

Why EXAchk

Development Methodology

Recommended Usage Overview

Report Structure

Detailed Usage Options

Collection Manager

Integration With Other Tools

Troubleshooting

3

4

5

6

7

8

89

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 90: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull View Component Elapsed Times

o Identify if one particular component is slow

bull View top 10 Time Consuming Checks

o If some in particular are slow show check_id and run again with

Troubleshooting Performance

exachk ndashexcludecheck ltcheck_idgt

bull Try increasing the number of parallel slave processes (Note will increase resource usage)

exachk ndashdbparallelmax

90

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 91: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull If checks are being killed this will be due to timeout

bull Timeouts can be caused by slow checks or interactive prompts

bull Try increasing default times with applicable environment variables

bull EBS checks query application data which can vary hugely in size depending on the size of your data

o If you suspect EBS checks try excluding EBS profile

oUse RAT_TIMEOUT to increase to a value where they are no longer killed

Troubleshooting Timeouts

exachk ndashexcludeprofile ebs

91

Environment Variable Timeout Controlled Default Value

(seconds)

RAT_TIMEOUT non-root individual

commands 90

RAT_ROOT_TIMEOUT root userid command

sets 300

RAT_PASSWORDCHECK_TIMEOUT ssh login DNS

handshake 1

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 92: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

bull Debugging via Daemon bull Debugging adhoc EXAchk runs

Collecting Debug

exachk ndashd start_debug exachk ndashdebug

bull Debug output

92

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 93: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved |

Safe Harbor Statement

The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle

93

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 94: EXAchk for Exadata Presentation

Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94

Page 95: EXAchk for Exadata Presentation