Summit 2011 ods edw technical

Business Intelligence and Data Warehouse

Goals, Trends, and Complexities of Supporting an Integrated SGHE ODS/EDW

Data and Date Driven Knowledge System

BAS S118

Greg TurmelSr. Database AdministratorTennessee Board of Regents

Date/Time: Monday, October 17 10 am ‐ 12:30 pm1


ODS foundations:

Project Vision Initial server configuration ODS Metadata STREAMS concepts Feeding and Care of the data stream:

A total commitment to quality (from everybody) Technical Challenges

Open discussion:2


Project baselines: A vision & sponsorship

Executive Management

Vision

Operational Management

Implementation Plans

Infrastructure and Technical Support

Project Resources – Integration Systems Management

Sponsorship

Resource Pool Defined

Commitment to Quality

First Stage

Second Stage

Third Stage

3


Additional questions to ask:

What is the project scope? What business requirement is driving the effort? What is the source system for record retrieval? What resources exist for the datastore? What resources exist to support a Data

Warehouse? What resource does the organization have in

supporting data modeling / design? What experience does the resources have?

(Source/Target – Logical/Physical designs) What Extraction methods (tools/technologies)

will be used?Can’t answer these basic questions then the project will have questionable results…… 4


Server Preparation: Key things to remember

Oracle Solaris 10 update 6 or above Oracle 11gR2 (11.2.0.1) Binaries installed/patched Oracle account – Modified .profile

ulimit – n 65536ulimit – s 16384

Note: [OWB JAVA errors] Oracle 11g memory management uses ‘projects’ Oracle 11g no longer uses the companion cd Oracle 11gR2 drops 32 bit libraries – binary

support Oracle OWB is now a default / part of the database

5


Configuration issuesServer Preparation: Oracle user account creation: cat /etc/password

oracle:x:101:101::/u01/home/oracle:/bin/ksh

Oracle Project creation: cat /etc/project user.oracle:100:Oracle:::process.max‐sem‐nsems=(priv,256,deny);project.max‐sem‐ids=(priv,100,deny);project.max‐shm‐ids=(priv,100,deny);project.max‐shm‐memory=(priv,7516192768,deny)

Assign memory allocation to the project: limits6


Note:

Bannervs.

Oracle

7


ODS metadata During deployment a static webpage is delivered for user/developer and architectural purposes

8

The metadata can be updated by your IT staff [e.g.] Feeding the database with Campus data


ODS metadata

9 http://tbrtap01.tbr.edu:9250/metadata/ODS_index.html


STREAMS concepts:1st – You have to get it integrated and running

DBMS_CAPTURE_ADM DBMS_PROPAGATION_ADM DBMS_APPLY_ADM

These Oracle packages are utilized by the ODS installation to create the Oracle® Streams queues.

These packages are called from the ODS package ODSSTG.MGKSTRC during the installation of ODS 8.2.

SGHE:FAQ 1‐81W3F1 ‐‐ ODS 8.2 and Oracle® StreamsFAQ 1‐81W3A9 ‐‐ Oracle Doc Id 290605.1 Oracle Streams STRMMON Monitoring UtilityFAQ 1‐81W38Z ‐‐ Oracle Doc Id 273674.1 Streams Configuration Report and Health Check Script

10


STREAMS concepts:

11

Streams Capture: Reads the database redo logs. Collects DML and DDL changes that have been made on the Banner source table

Streams Propagation: The process moves changes (LCR) from the Banner source DB to the ODS target DB

Streams Apply: Takes the changes into the ODS stage tables. Matches and updates the modified record in the ODS target table

Error Queue: Low level streams data errors encountered can be reviewed and reprocessed.


select CAPTURE_NAME, STATUS from DBA_CAPTURE;

BPRA_BANNER$CAP ENABLED

select PROPAGATION_NAME, DESTINATION_DBLINK, STATUS from DBA_PROPAGATION;

BPRA_BANNER$PROPODSD.TBR.EDUENABLED

Validate streams configuration: No duplication

12

STREAMS Validation: Source systems


SELECT MTVPARM_EXTERNAL_CODE, MTVPARM_INTERNAL_CODE_2 from MTVPARM WHERE MTVPARM_INTERNAL_CODE_GROUP = 'STAGE CONFIGURATION' AND MTVPARM_INTERNAL_CODE = 'SOURCE ALIAS';

TEST.TBR.EDU BPRA_BANNER

Populating MGRSDAX from Banner GTVSDAX

On ODS: desc MGRSDAX; (describe) On Banner: desc GTVSDAX; (describe)

13

STREAMS Validation:


14

STREAMS Validation Ensure that aq_tm_processes is not explicitly set

to 0 (Oracle Support recommends – unset the parameter):

‐ FAQ 1‐2USESE Oracle Metalink Note 428441.1 ‐ "Warning: aq_tm_processes Set To 0" – Alert Log

ALTER DATABASE RENAME GLOBAL_NAME TO fully_qualified_database_name;

DATA_PUMP_DIR – setting it is a requirement.

select DIRECTORY_PATH from dba_directories where DIRECTORY_NAME = 'DATA_PUMP_DIR';


15

Bann

er Transactio

nal System

Use

r or p

roce

ss in

put

Propagation Streams ApplyQueue Queue

Streams Capture

Chan

ges

Query

Write

Com

plex

Joi

n

Ope

ratio

nal D

ata Store

Database Server # A Database Server # B


16


Feeding and Care of the data stream: Data security: Are the feeds behind fire walls?

Masking: Are you using production quality data in TEST?

Compliance: Are you capturing and transmitting SSN, Health, or transactions?

Exploits: Are you blocking SQL Injection?

Audit: Are you regularly scheduling a review?

Monitor: Are you proactively monitoring systems?

17


Technical challenges:Impact analysis – testing change management?

Dataload

Dataload

Dataload

Chan

ge M

anag

ement B

anne

r

Dataload

Dataload

Dataload

Dataload

Dataload

Dataload

18

Chan

ge M

anag

ement O

DS

Chan

ge M

anag

ement E

DW

Chan

ge M

anag

ement D

atam

art


Technical challenges:

Demand Management: A Lesson in what not to allow…

Chan

ge M

anag

ement B

anne

r

19

PPRD DEVLPROT PROTROY

TSTHCSPPRD

8* Methods MTP (Move to production process)


20

Technical challenges: Change management


21

Technical challenges: Data Quality Access

data /odbc uploads

Data

Data

Data

Data

Data

Data

Data

Oracle 11g datamart

Oracle 10g datamart

Oracle 9i datamart

EDW


Design/Analyze Major System changes: Require an understanding in how changes affect down line systems: Reporting – [e.g.] KPI’s

TN Mods: SMO Banner – June 15: Gen 8.4 July 15: Student 8.51 - delivered

Banner 9: ReleasedSept 2011

Comm FrameworkStu Class ScheduleStu Course Catalog

Oracle 11gR1Banner Database

Oracle FusionMiddle Tier WLB 11gR1

Luminis 4.3 - Solaris 10

April: End of Product life:SGHE Banner 7.x

HR 8.5: IPEDS Note – No 8.2

April: Prepare and begin delivering SGHE EDW Workshops - Enhancing BIefforts across the State

EDW 8.2 GOAL: testing

EDW 8.2 workshopsApril: Installing EDWMay: Installing EDWJune: Installing EDW

EDW 8.2 GOAL:production

22


FAQ 1‐BFL58F ‐ ODS 8.2 install stop responding when ETL scripts (section 3.7.3) are executed in a production environment.

Defect 1‐BNA5XL student_etl_install.sql SATURN.SOTVCUR ERROR: ORA‐20000: Adding supplemental log data terminated with the following error: ORA‐14450: attempt to access a transactional temp table already in use

Related FAQs:FAQ 1‐BPCU06 ODS 8.2 Report Staging Area Status DBA_APPLY_ERROR ORA‐01403 FAQ 1‐DKNNNB ODS 8.2 Streams error ORA‐01435: user does not exist.FAQ 1‐GBP75K ODS 8.2 ORA‐04031 unable to allocate bytes of shared memory FAQ 1‐AXHXP3 ‐ How to restage the ODS 8.2 / 8.2.1 staging tablesFAQ 1‐C8Y1U1 ‐ How to restage one specific ODS 8.2 stage table.FAQ 1‐CHWKO1 ODS 8.2 Streams Banner Source data not propagating to ODS targetFAQ 1‐IEH1Z4 ODS 8.2 Oracle Streams Spilled messages and BPRA_BANNER$CAP queueFAQ 1‐IGVVDY ODS 8.2 and Oracle Streams and Init.ora Parameter GLOBAL_NAMESFAQ 1‐JF68GU ODS 8.2 and How to reload the data only for a staged table in BPRAFAQ 1‐I4UP5C ‐ Is my Oracle environment ready for ODS 8.2 and Oracle Streams?FAQ 1‐BU6KRN ‐ ODS8.2 how to reconcile stage tables

Technical challenges: Defects and Config

23


Note 273674.1 Streams Configuration Report and Health CheckNote 437838.1 Streams Specific PatchesNote 238455.1 Streams Supported and Unsupported DatatypesNote 782541.1 Streams Replication Supplemental LoggingNote 290605.1 Oracle Streams STRMMON Monitoring UtilityNote 365648.1 Explain TXN_LCR_SPILL_THRESHOLD in OracleNote 265201.1 Troubleshooting Streams Error ORA‐1403 No DataNote 779801.1 Streams Conflict ResolutionNote 461278.1 Example of a Streams Heartbeat TableNote 313478.1 Performing Manual DDL in a Streams EnvironmentNote 335516.1 Streams Performance Recommendations Note 730036.1 Overview /Troubleshooting Streams Performance

A complete list of streams articles on Metalink: Knowledge tab ‐> Database ‐> Information Integration ‐> Streams

Technical challenges: Oracle support

24


Open Discussion

25


10 Minute Break

26


EDW foundations: In this section we discuss

Establishing project timelines

What STREAMS baselines are needed

Technical challenges

EDW Metadata

Argos delivered cubes for the reporting community

Open discussion:27


*Refresher Tra ining

Task or Effort SEPT OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEPT OCT NOV DEC JAN FEB MAR APR

ODS 8.2 Tra ining *

ODS 8.2 Test Implement ODST

ODS 8.2 Prod Implement ODSP

EDW 8.2 Tra ining

EDW 8.2 Test Implement EDWT

EDW 8.2 Prod Implement EDWP

ODS 8.3 Patching Tentative Schedule

EDW 8.3 Patching Tentative Schedule

ODS/EDW 8.2 implementation Timeline as of September 2011

2011 2012 2013

Building Time lines supporting your projects

28


EDW work is:

Building Patterns out of Random Chaos/Mashups? ‐ Not!

It is highly refined with specific deliverables:

Goal orientated – Subject studied over time ‐ (Trending, Strategic, Decisions)

Non‐Volatile – data is historical based on criteria ‐ (Month, Year) ‐ not dynamic (Do not edit data)

29


Technical challenges: Organization: structured or a mashup?

Business Requirements

Analysis

Design

Implementation

Testing & Evaluation

Deployment

Usage

Retirement

Business Requirements

Mashup Ideas

Discovery

Selection

Composition

Deployment

ApplicationUsage

Retirement

Evolution

Resource: IEEE computer.org/itpro30


EDW metadata:

http://tbrtap01.tbr.edu:9250/metadata/EDW_index.html 31


32

EDW work supports predictive analysis: [e.g.] Profiling at risk students (goal focused)

‐ behavior in their high school, freshman, sophomore, junior years…

‐ Financial assistance, parental capacity to help, scholarships available…

‐ Leadership roles, involvement in the community

[e.g.] Cost associated with lost opportunities with graduating a student across the system


33

Analytics: Predictive Analysis methods using Longitudinal Studies on Students

What if: We identified the Cost associated with losing students? Revenue over time lost associated with these students? Educated to a certain level and they disappear? What was the impact to the University or College system? The economic area impacted? The State education results?

Student Population Lost Cost per class

Avg # of Class hours taken Avg cost of tuition

Sum total of all students lost Campus level

5367 $166.00 24 3984 $21,382,128.00 CC Sophomore

11629 $166.00 24 3984 $46,329,936.00 CC Freshman

853 $366.00 24 8784 $7,492,752.00 University Junior

5118 $366.00 24 8784 $44,956,512.00 University Sophomore

11089 $366.00 24 8784 $97,405,776.00 University Freshman

$217,567,104.00


Storage Requirements: Partitioning data

Schemes such as: Single or Composite Why: Creates faster queries in reporting (tuning)

Single: Range, Hash, or List

Composite: Two dimensions [e.g.] Order/Shipped Range – (Range) (Hash) (List)List – (Range) (Hash) (List)

1. Manageability2. Availability3. Pruning

34



Range – maps data to a partition based on a rangeof values.

01.01.2010 until 12.31.201001.01.2011 until 12.31.2011

Hash – maps data to a partition based on an algorithmkey that evenly distributes across devices.

List – maps data specifically to lists of discrete valuessuch as West Tennessee Region, MiddleTennessee Region, or East Tennessee Region.

35



Schemes such as: Single or Composite Single – Level (Range, Hash, or List)

http://download.oracle.com/docs/cd/E11882_01/server.112/e16541.pdf36



Schemes such as: Single or Composite Composite – Level (Range, Hash, or List)

http://download.oracle.com/docs/cd/E11882_01/server.112/e16541.pdf

37


38


Schemes such as: Composite Composite – Level (Range, Hash, or List) Fiscal Year – July 1st ‘11 until June 30th ‘12 Calendar Year – Jan 1st ‘11 until Dec 31st ‘11 Academic Year – August 2011 until May ‘12

Joined with regional designation:

West Tennessee Middle Tennessee East Tennessee


39

ETL: Oracle 11g Warehouse Builder(extract, transform, & load)

Install and Admin Guide ‐ b31280

Lesson 1: Creating OBIEE Meta for OLAP 11g Cubes

http://st‐curriculum.oracle.com/obe/db/11g/r1/olap/biee/createbieemetadata.htm

Lesson 2: Building OLAP 11g Cubes

http://st‐curriculum.oracle.com/obe/db/11g/r1/olap/cube/buildicubes.htm

Lesson 3: Querying OLAP 11g Cubeshttp://st‐curriculum.oracle.com/obe/db/11g/r1/olap/cube/querycubes.htm


Reporting:

Snapshots of the system: FREEZE dates for Analytics

Trend Analysis: Focus points Scatter plotsDealing with outliers

Define collection: Build cube based on need, not everything

40


Reporting:

Impact of load: Scheduling and heavy resource use

Adding Legacy Data: Criteria to set – Integration of archival data

Updates:No update processes / no data tweaking activities

41


Argos Co‐op datablock code available:

Admissions Dashboard:

Admissions Dashboard with multiple forms, enrollment OLAP, KPI, Demographic and test scores.

Course Registration EDW:

The Course Registration datablock displays the generated credits and number of seats (and all available measures) by academic year and academic period.

42


Course Enrollment Summary Report:

Course enrollment by academic period, college, and department.

Enrollment Ethnicity:

Shows ethnicity of enrolled students based on term.

Enrollment Trend Analysis:

Displays enrolled student name and Banner ID based on Academic year and term.

43


Financial Analytics ‐ Departmental Expense:

Uses OLAP cube for financial expenses by department with several dimensions

Financial Aid Pre‐Student:

Uses the Financial Aid Pre‐Student datablock to understand the trends in pre‐student acceptance and enrollment based on how financial aid amounts are allocated.

44


Financial Aid Student:

The Financial Aid Student datablock can be used to understand the trends in packaging financial aid awards, and to support improved allocation of financial aid amounts

General Ledger EDW:

Use the General Ledger datablock to understand the trends in the general ledger activity, and to better manage overall financial health of the institution.

45


Grant and Project:

Use the Grant and Project datablock to understand the trends in grants and other sponsored research projects, and to better understand and manage research funding and spending.

Recruiting and Admission:

Recruiting and Admission datablock trends in recruiting and admissions. To better manage the enrollment funnel, and to understand trends in financial aid awarding to new students to better manage financial aid funds.

46


Workforce Turnover Summary Report:

Work force turnover by start/end date, gender, department, employer, class and ethnicity.

Workforce Headcount Analytics:

Uses OLAP cube to present Workforce analytics. Comes with several dimensions.

47


Graduation Completion:

The Graduation Completion datablock can be used to understand graduation trends, and to monitor and improve graduation rates.

Operating Ledger:

The Operating Ledger datablock can be used to understand the trends in operating expenses and revenue to help you to better plan and forecast.

48


Program Enrollment Trends:

Uses OLAP cube to present program enrollment trends. Several dimensions are included.

49


50

Build a system the customer can use ‐ Think: Customer oriented results ‐ quick, meaningful, clear

Answer problems that keep coming up during conversations... [e.g.] Student Retention & Graduation Rates

Big picture is Important, but... create value now.

Don't push people into a project, create a project customers find value in

Twelve project ‘success’ factors to remember:


51

Analytical efforts provide insight into solving the problems – knowing the difference in what is pushing/pulling a project (Discovery vs. Economic)

Did you hear the customer? Did you arrive at a consensus? Did you use the KIS (keep it simple) method? Did you engage the customer in a review? Did you tell friends (expand its value)

Feedback... user experience

If you lost your audience... why, then fix (change) the direction



52

Truth and Transparency in the data, the collection methods, the logic in any transformation = competence and value to the customer

No one cares about a personal ego, the technology, or commitment ‐What is the customers viewpoint = What's the product and What's in it for them?

Measure and cost out the project including human resources (time & energy)

Never give up. One failure does not equal the abandonment of a project



53

Q&A ?


54

Master Note for Streams Recommended Configuration [ID 418755.1]

Streams Secure Queues & Using DBMS_STREAMS_ADM.SET_UP_QUEUE to Setup the Secure Queue [ID 230902.1]

Streams Support for Compression [ID 763997.1]

How To Exclude A Table From Schema Capture And Replication When Using Schema Level Streams Replication [ID 239623.1]

How To Configure Streams Real‐Time Downstream Environment [ID 753158.1]

How To Setup Schema Level Streams Replication with a Downstream Capture Process with Implicit Log Assignment [ID 733691.1]

How To Access Streams Or Advanced Queuing Information From The 10.2 Dbconsole [ID 336061.1]

Banner Operational Data Store 8.2 Handbook / Architecture


55

11gR2 Streams Concepts and Administration ‐ e17069.pdf

11gR2 Streams Replication Administrators guide ‐ e10705.pdf

11gR2 Streams Advanced Queuing users guide ‐ e11013.pdf

11gR2 Oracle Database Warehouse Builder ‐ Concepts ‐ e10581.pdf

11gR2 Oracle Database Warehouse Builder Release Notes ‐ e10585.pdf

11gR2 Oracle Warehouse Builder Data Modeling ‐ ETL ‐ Data Quality guide ‐ e10935.pdf

BPRA Upgrades ‐ Oracle 11g/OWB 11g, PM App Dependencies, Jim Carter, Brian Large: http://www.edu1world.org/CommonsBI/wiki/document/4754

Post‐Installation Steps for Oracle Warehouse Builder 11g Release 2 (11.2.0.2) http://download.oracle.com/docs/cd/E11882_01/relnotes.112/e10585/toc.htm


56

Contact Information:

Greg TurmelSr. Database AdministratorOffice of Information TechnologyTennessee Board of Regents1415 Murfreesboro Rd. #358Nashville, TN. 37217

615.366.4467 (Office)615.365.1598 (Fax)http://itinfo.tbr.edu (IT website)http://twitter.com/datahaulrhttp://www.linkedin.com/in/gturmelhttp://www.slideshare.net/gturmelhttp://itinfo.tbr.edu/itinfo/tiki‐download_file.php?fileId=515