Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom...

Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom Utilities

Eric Boyce - DBABryan Mack – BI Development Team Lead

April 14, 2015Session 11951

Session Rules of Etiquette

• Please silence your cell phone/pager/purse dog

• If you must leave the session early, please do so as discreetly as possible

• Please avoid side conversation during the session

• Please hold questions for the end of the presentation

Thank you for your cooperation!

Session rules of etiquette

Bryan L. Mackbryan.mack@cccs.edu

Higher-Ed Employment History

• Colorado School of Mines, Asst. Director of Advancement Services 2002-2012

• Colorado Community College System, Team Lead/Data Warehouse Developer, 2012-2015

Session rules of etiquette

Eric Boyceeric.boyce@cccs.edu

Higher-Ed Employment History

• University of Northern Colorado, BPRA Database Administrator 2009-2012

• Colorado Community College System, BI Sr. Oracle Database Administrator, 2012-2015

Purpose

• To create an understanding of how simple it can be to add desired functionality to the BPRA Administration Interface. Further, we’ll step through two scenarios whereby the Colorado Community College System benefited from augmenting the BPRA Administration Interface’s functionality.

1 Problem I – Inconsistent ODS Refresh Times

2 Potential Fixes – Baselines, Profiles, Statistics

3 Implementing a Solution – Stats Gathering

4 Problem II – BPRA Batch Window Utilization

5 Implementing a Solution – Chain Builder

Agenda

Problem I – Inconsistent ODS Refresh Times

Inconsistent ODS Refresh Times

• Greeted with this in the Morning:

Why the Poor Performance

Root Cause: Stale Statistics

● Over 4,300 daily batch jobs in CCCS’s Banner environment (AppWorx)● Due to process ordering and contingencies batch execution times are dynamic● Changing Maintenance Window for statistical collection ineffective

Bad Stats Example

NUM_ROWS – Updated by stats collection

Potential Fixes – Baselines, Profiles, Statistics

Baselines, Profiles, Statistics

● There is nothing wrong with using baselines, profiles, or dynamic sampling to achieve performance improvements

● We wanted a tool that was simple, yet effective, in improving/maintaining ODS performance● We needed something that could handle Ellucian upgrades without

adversely affecting performancei.e. Not reliant upon sql_id values

● We know the problem is poor statistics. No need to put cart in front of the horse by creating baselines or profiles

When Should Statistics Be Collected?

According to Oracle

Source: http://docs.oracle.com/cd/E25054_01/server.1111/e16638/stats.htm#i41884

Requirements

● Collect statistics on change tables, composite tables, and indexes just prior to the execution of a mapping

● Must be simple interface to understand and utilize… otherwise nobody will use it

● Must have ability to ascertain mappings’ objects dependencies

● Should have the ability to handle various DBMS_STATS parameters

Implementing a Solution – Stats Gathering

Identification

Identify exact locations where mappings are executed:

Digging Deeper

● Matching String Literals from Control Report to MGKMAP Body

Verification

Query against all_source

Before we go further…

Ellucian’s stance on the modification of baseline code:

-Any customizations to baseline code are NOT supported-

Customizing Baseline Code

IMPORTANT: Assure that the baseline scripts are saved off and available. If an error is encountered you can always revert to the functional baseline version.

Testing New MGKMAP

● It worked! We now know where the stats collection should take place● We also have the mapping object name

Creating a Package to Gather Statistics

PACKAGE: IA_ADMIN.CCCS_GATHER_ODS_STATS

PROCEDURES: p_gather_mapping_stats (mapping_name, estimate %) p_gather_composite_tab_stats(mapping_name, estimate %) p_gather_srce_and_change_stats(composite_view, estimate %) FUNCTIONS: f_get_stats_check (mapping_name)

p_gather_mapping_stats 1ST – Gather Composite Table Stats

- Determines Composite Table using DBA_DEPENDENCIES

p_gather_mapping_stats 2nd – Evaluate Mapping Type

- If it is a DELETE mapping, gather statistics on change table- Determines Change Table using DBA_DEPENDENCIES

p_gather_mapping_stats 3rd – If mapping is not a DELETE, then it must be an UPDATE or LOAD

mapping. - Determines Composite View and dependent tables

p_gather_mapping_stats 3rd (cont.)– If composite view found, call p_gather_srce_and_change_stats - Loops through source tables and collects stats

f_get_stats_check● Called from MGKMAP prior to any collection of stats

● Function checks to see if statistics should be collected for a particular mapping

● Allows for control as to which mappings collect statistics

● Uses a custom MTVPARM record

Remember when…

● We want to collect stats on this mapping prior to execution to prevent this from happening in the future

Create MTVPARM Parameter

● Beneficial to store the parameter values in MTVPARM as they’re easily accessible via the Admin UI

Once again…

● Once again, we have bad statistics.

Control Report – With Stats Collection

Control Report – No Stats Collection

● If there is no MTVPARM record for a mapping, the f_get_stats_check will evaluate to FALSE and statistical collection is simply ignored.

Control Report – Stats Collection on UPDATE

Problem II - BPRA Batch Window Utilization

Batch Window

● CCCS’s BPRA environment has a batch window from 00:00 – 07:30.

● ODS refreshes are executed concurrently as REFRESH_ALL takes too long.

● Implementing EDW and SRP caused overruns(52 Hours initially, cut down to 3 hours after in-house tuning)

● Varied ODS refresh times made it difficult to predict when to schedule the EDW Operational Refresh Job

Need for Contingency

● SRP reliant upon ODS Student, ODS Finaid, and ODS General refresh completion

● Wanted a way to maximize our batch window by not allowing time between jobs.

● Also wanted the ability to run concurrent jobs

Oracle Chains

“A chain is a named series of tasks that are linked together for a combined objective. Chains are the means by which you can implement dependency based scheduling, in which jobs are started depending on the outcomes of one or more previous jobs.”

Source: http://docs.oracle.com/cd/B28359_01/server.111/b28310/scheduse009.htm#ADMIN10021

Oracle Chains

Chains can be thought of as a series of steps with defined rules as to when a step should be performed. A step is associated with a task or action.

Step Action When to start-------------------------------------------------------------------------------------------------------

Step 1 - Choose Bread TRUE (Immediately)Step 2 - Add Condiments

Step 1 CompletedStep 3 - Add Meat/Veggies Step 2 CompletedStep 4 - Assemble

Step 3 Completed Step 5 - Eat Sandwich Step 4 Completed

Step 6 - Go to Sleep (End) Step 5 Completed

In our Case

Step Action - Program When to start - Rule-------------------------------------------------------------------------------------------------------

Step 1 - Refresh Student TRUE (Immediately)

Step 2 - Refresh FinaidTRUE (Immediately)

Step 3 - Refresh General TRUE (Immediately)

Step 4 - Refresh Custom MViews Step 1 CompletedStep 5 - Refresh EDW Operational Steps 1, 2, 3,

4 Succeeded

Defining Actions via Programs

● Oracle Scheduler Program Objects define an action for a Chain step● Can be used to abstract a PL/SQL Block, Stored Procedure, or an

executable

Building Programs

● So couldn’t we just put these Refreshes into Programs using PL/SQL blocks?

DECLARE JOB_NUM NUMBER;

BEGIN SELECT sys.jobseq.nextval INTO JOB_NUM from dual; mgkmap.P_RunETLMapSlots(9,JOB_NUM,'REFRESH_FINAID',NULL, '');

Creation of a Program

● Could be done in Enterprise Manager but can be tedious when doing a large quantity.

Implementing a Solution – Chain Builder

Sample Admin UI Job Script

● Ellucian delivers a sample Admin UI job script in the ODS installation code tree:

ods\odsdevl\ods\ia_admin\dbscripts\utility_scripts\sample_job.sql

● Serves as a good reference for the necessary pieces when creating a custom job.

INSTALLED PROCESS Parameter

● Creation of an INSTALLED PROCESS Parameter creates a link in the Admin UI’s ‘Schedule a Process’ Menu

JOB Parameter

● Creation of a JOB Parameter links the Installed Process to a Packaged Procedure

Basic Packaged Procedure Header

● Main procedure of sample job noted earlier

● Main procedure of CCCS_CHAIN_ADMIN

Questions…Questions…Questions…

● Before we go any further we have to ask ourselves:

1. How many steps should the chain builder handle?2. How many rules will we need?3. How are the rules created?4. How can we monitor progress as chain is executed?5. How do we avoid having objects of the same name?

First Question

1. How many steps should the chain builder handle?A: CCCS chose to start with 7. This was the fewest number of

steps we would need to create a chain that could handle our EDW/SRP requirements.

Note: It was decided that although CCCS would start with 7 steps, the utility needed to be flexible enough to easily handle more steps in the future.

Second Question

2. How many rules will we need? A: We need the number of steps decided upon +1

- This is because there is a final rule that states when all steps are finished, the chain is has completed–

- Additionally, the final rule need not be a parameter for a user to control as this could cause the chain to stall.

Creating Parameters

Rules and Steps

7 Rules and 7 Steps

Additional Parameter

● We also wanted to allow the user to specify their own Chain Name

So How’s it Look

Parameter Handling

● We use tables to store the values we’d like to display for parameter values. The ‘SELECT’ parameter type simply displays the results of a query.

Parameter Handling (cont.)

Example.

Translation Table

● We use the ia_admin.cccs_chain_prog_xwalk table to map Display names to actual object names. Further, the SEQUENCE_NUMBER

gives us a means of easily controlling, which processes are displayed in Admin UI.

Rules Table

Rules possibilities are also stored in a table.

Admin UI Submission

How Parameters are Passed

● The job generated contains all of the parameters inputted, in a pipe delimited format.

● Knowing all the parameters, we can now build a chain

Parsing Parameters

● All parameters come into the parms input parameter

● Use mgkutil.F_GetParameterValue to determine what the inputted values are

Building Chain

● After validation that the job has basic parameters in place, we verify that no other chains of the same name exist. If they do, they are removed and an entry noting this is made in the control report. Following this step, an empty chain is created

Building Chain Steps

● Once the chain is built, we add the steps for it● Display names for programs are parsed to their actual object names● Using a counter variable we loop through all the programs creating chain steps along the way.

Building Chain Rules

● Chain Rules are built in a similar manner as the Chain Steps with a loop and a counter variable.● There is one final rule that is built. It is the summation of all steps and informs Oracle that the chain is complete.

Creating Scheduler Job

● We’ve now created a chain, added steps to it, and added rules to it.

● The only thing left to do now is to submit it.

Checking Mechanism

● As we wanted to know the status of the scheduler job, we created a self checking mechanism● Every five minutes the chain run is evaluated and it’s status is reported in the control report

Final Control Report

Adding More Steps in the Future

● To add additional steps, all one needs to do is create additional PROGRAM and RULE parameters following the naming convention laid forth● Additionally, rules must be added to cccs_rule_possibilities or

whatever table is chosen to store the RULE literals

Summary

With an understanding of how the Admin UI works, a developer can easily create utilities/functionality to augment what is already a great product delivered by Ellucian

Questions & Answers

Thank you!

Eric Boyce - DBABryan Mack – BI Development Team Lead

Please complete the online session evaluation form.Session 11951

Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom...

Software

Schema Refresh

Refresh Course

ReFresh Deck

ITIS 3110 Network Auditing. overview networking refresh protocols refresh

Refresh Ng

Refresh OKC

BPRA Laguna Chapter 2

Brand Refresh: 3 Signs You're Ready for a Refresh

Refresh alpha

Refresh Difference

Increase Sales on Twitter (Proactively!)

Proactively Released

Extreme Refresh

Summer2011 Refresh

Scrum refresh

UNDERGROUND REFRESH

DRAM Refresh

Refresh Water_N8295077

Performance Management Overview Mike Salisbury BPRA Product Manager SunGard Higher Education Mike.salisbury@sungardhe.com Pennsylvania Banner Users Group

ReFresh Milwaukee