Run Your Oracle BI QA Cycles More Effectively

Contact Us510.818.9480 | www.kpipartners.com

© KPI Partners Inc.

Start Here

Run Your OBI QA Cycles More Effectively

Jeff McQuigg

Sr. Architect

KPI Partners, Inc.

BIWA Summit

Jan 10, 2013

� Senior Architect at KPI Partners

� 10 years OBIEE consulting experience, 20+ years overall

� Personally been involved with over 45+ OBI projects in every capacity (BI

Architect, Data Modeling, RPD Metadata, Business Analyst, Report

Developer, ETL Architect/Developer, Project Manager, Pre-Sales)

� Oracle Ace thought leader for BI & OBI:

• Blogging on OBI best practices since 2007 at

GreatOBI.WordPress.com

• Frequent Oracle Open World speaker

� Personal: My 3,000+ beer bottles of beer are on display at Brewpalace.com

2

www.kpipartners.com

The Leader In Oracle BI & EPM 3

Strategic Consulting | Systems Implementation | Training

Depot Repair Analytics

Fixed Asset Analytics

Manufacturing Analytics

Salesforce.com Analytics

Student Info Analytics

Subledger (SLA) Analytics

and moreD

Transform Data Into InsightTransform Data Into Insight

� Staff built from Oracle/Siebel/Hyperion engineering teams

� On-site, off-shore and blended shore delivery models

� Exclusive pre-built solutions for Oracle BI & E-Business Suite

Oracle BI

Hyperion

Endeca

Exalytics

Oracle BI

Hyperion

Endeca

Exalytics

1. Why QA plans are always different

2. The importance of Builds

3. What resources will you need

4. The benefits of automation

5. How to test OBI Meta Data

Goal: Build better QA plans!

4

� Break the problem down to its layers

� Each layer can be tested independently for the most part• Some assumptions needed

� End-to-End and End-to-Mid testing ensures proper handoff between layers

Extracts

Loads

OBI Model (Ad-hoc)

Security Model

Reports

UI Alerts

5

Test the OBI stack for the following:1. Data Validation – Is the data accurate?

2. User Functionality – Does the UI work properly?

3. Security – Are the appropriate visibility rules applied?

4. Performance – Do the loads and reports run fast enough?

5. Infrastructure – Is the infrastructure reliable and robust?

6

1. Resources always differ• Consultants, IT BI &/or DW team, source system SMEs, internal QA teams, power users

2. Project execution always differs• Agile or iterative vs. traditional waterfall• Document robustness• QA team members’ involvement during project• Different corporate toll gates/methodology

3. Legacy reports / Re-platforming• May or may not have something to compare to

4. QA source application environments differ• Dedicated vs. shared, controlled vs. no control

5. Technical stack differences (less so on BI Apps)• E.g.: Real-time layer, ODS, DW, DM, UI and security integrations, large user volumes

No two QA Plans are ever the same!

“QA Plans are like a box of chocolates”

- Forrest McQuigg

7

� A BI system is different than compiling code modules for a new build

� It’s about the Data Transformation:Applying code to data to make different data

� ETL loads drive the QA plan as they take time (12-48 hours typically)

� ETL has a high degree of integration• Facts depend on Dimensions• Extract � Load � Post Load Process � Aggregate

� Several iterations of ETL Builds are needed to get it right• Full load for base logic• Incremental loads for subsequent loads

� In-line ETL testing during Development can lower QA risk, but Builds are still needed

The ETL Machine

8

Putting the Build cycles together yields a staggered plan

Focus on the FullLoad first

then on Incremental Loads

DMore than 2 Full

loads may be neededQA Begins

Staggering can help if two environments

9

Consider two levels of QA test cases:

1. Code-to-Spec• Identify transformational logic and develop test cases based on

the design spec

2. Spec-to-Business Objective• Ensure the spec was written correctly. Examples:

� All widgets should be assigned to a customer

� If an owner doesn’t exist in System A, then it should use the record from system B

• Perhaps 2-5 goals per spec

• More applicable for custom coded solutions and less so for OOB BI Apps extensions

10

QA Script Development should occur alongside ETL development using the same spec

• Keep QA and Dev resources separated

Design

Spec

Object Code Dev

Object QA Script Dev

QA

Execution

11

� Pipeline as much as your team and environment can handle• Offshore capability helps tremendously• Weekend builds are important• Baby sitting loads takes time & effort

� Multiple DW environments are a must• QA Pass #1 in Environment #1 while Load #2 ongoing in Environment #2• Flexibility is key; use Pre-Production server if available• Greater complexity when SIT and UAT are used in parallel• Post Release 1 Deployments are even more complicated

� Plan on at least 2 QA iterations for Full Load and at least 3 or more for Incremental Load

� Keep in mind any special loads like weekly or monthly jobs

� OBI RPD and Reports can be layered easily on any of these environments

Fix

+

Build

Test

+

Solve

12

� Now is not the time to be optimistic!

� Add sufficient buffers in the schedule• Problems are 100% guaranteed

� Dependencies: A bug in a dimension may require a full reload to retest

� QA Cycles typically can run for several months on a complex system

� Plan on your source system support approach for Incremental Loads early

• Multiple Prod snapshots to manage – or –• Manual creation of test cases in a QA system

13

There are a variety of tests to run for ETL• Table row counts

• Allocations & Summations (Total $ by month)

• Attribute ranges (Min and Max values)

• Specific Transformation logic (If-Then-Else)

• Slowly Changing Dimensions, Snapshot Facts

• Metadata columns (minimal on BI Apps)

• Aggregates sync with base tables

• 10 Random Records tracing

• Engineered Records (great for incrementals and special test cases)

14

� Use SQL script files to automateSelect ApplLogic(Status_CD), Count(*)

from Source.Table

Group by Status_CD order by Status_CD

Minus

Select Status, Count(*)

from Target.Table

Group by Status order by Status;

� For Extracts, consider external file comparison tool

� Source system technical SMEs write source scripts

• Challenging to replicate business transformation logic – get your top experts

� Compare source SQL with Metric totals in OBI Answers (Anchor Metrics)

� Perform both single-hop (Extracts vs. Loads) and multi-hop tests (Source vs. DW Target)

� Leverage database constraints (NOT NULL, FK) during DEV and QA to assist

Status_CD Count(*)

Open 4,129

Closed 65,536

Rejected 80,085

UnSpecified 1,024

Status Count(*)

Open 4,129

Closed 65,536

Rejected 80,085

UnSpecified 1,024

MINUS

15

� Ad-hoc testing of the OBI Model is skipped too frequently

� Even if reports are accurate, what about ad-hoc queries?

� Reports are built on top of the ad-hoc subject area

� OBI thinks and generates SQL – does it do so correctly?

� There are tradeoffs between QA effort vs. confidence

• 100% confidence is not possible Confidence

# Tests

Database

OBI Model

Reports

16

� Ensure proper SQL generation and consistent results

� Can be done on a buggy DW (within reason)• ETL QA Team ensures raw numbers in tables are accurate

• OBI Tests are relative to those numbers even if incorrect

� Automation of OBI Testing: Build test reports in OBI alongside reports in DEV

• Place in separate IT only dashboard

• Can be run in any environment at any time

• Excellent Automation technique

• Great for fast diagnosis of problems – catching unintended consequences

17

1. Skeleton accuracy: (Tables & Joins)• Does OBI generate the proper SQL? (BI Architect)• Do the metric values remain constant for each dimension?

(Tests join paths and aggregates)

2. Derived metric accuracy:• Check OBI Derived metrics based on an Anchor metrics

3. Dimensional attribute accuracy:• Is descriptive data coming over from the source properly?

4. Data security model accuracy:• Are filters applied properly to existing queries?• Is the SQL generated still correct & results consistent?

5. Drill Paths:• Confirm that dimensional drill paths are correct

18

“Tree Top” Tests• Break an Anchor metric out by the tops of each dimension• Make sure correct SQL & correct tables are used (Architect)• Demonstrates Unit Test

• Also try multiple dimensions at the same time if possible

19

� Raw OBI mappings to base Fact table fields are “Base” or “Anchor” metrics

• E.g.: Count(Headcount_Ind) or Sum(Total_Amt)• There can be filtering logic in the RPD• ETL QA verifies that these are correct

� Especially for BI Apps projects

� Derived metrics are those calculated in OBI:• Filtered metrics: Headcount vs. Employee Headcount • Time Series: Prior Year Employee Headcount• Rates & Ratios: # Cases per Employee• Complex Metrics: Rolling 12 Avg. Headcount

� Incorrect Anchor metrics due to ETL do not matter• Prior Year $ should match even if the TOTAL_AMT_USD field is

wrong in the Database• Data fields are variables, just like algebra: Prior Year(x)

M3 M4

M5 M6

M7

M1 M2

Database Table

20

Hint: Capture definitions leveraging other defined fields

� Reuse BI definitions as opposed to always mapping to raw tables

1. Order.Status: IfNull(ORDER_TABLE.STATUS_CD, ‘Unspecified’)

2. # Orders: Count(ORDER_TABLE.ORDER_NUM)

3. # Open Orders: # Orders where Order.Status = ‘Open’

Reuse business terminology as much as possible

Three benefits:1) Makes creating test cases much easier2) Communicates the definitions better to business users3) Helps developers reuse logic when building in RPD

21

� Make reports that confirm the Derived metrics using their Anchor metrics

� Use color coding to assist� Use report calculations to demonstrate� Creative solutions are a must!

Report Calc

22

� Try to avoid downloads to Excel if possible• Hinders automation

� Use two reports if needed – Be Creative!� Provide some instructions;

• These reports should be used for a very long time

RSUM() RCOUNT()

RSUM()

RCOUNT()

23

� Again be creative!� This test verifies Prior Year Ship $ is accurate� Solution: Run the report for 2012 and 2011

• Compare 2011.CY Ship $ to 2012.PY Ship $

2011 2012

24

Extensive OBI ad-hoc testing may take too long

Leverage your reports as surrogates for much of the OBI ad-hoc tests

• They will include multiple dimensions (Skeleton test)

• Various versions of the metrics within the many report structures

• “Hit it from multiple angles”

� More reports per topic � greater confidence

25

� Can be done by an internal QA group not familiar with all the details

� Must have a decent report spec to use• Difficult if an iterative report design approach is taken – minimal specs to use

� Displaying the proper data set - filters (Report shows Open Orders only)

� All columns relatively match (% Deviation = 100 * (Plan – Actual)/ Actual)• Can be done without OBI & ETL testing (Algebra again!)

� Drill downs and navigations work• For navigations, key test is making sure #s remain the same!

� Interactions with prompts

� UI Items: Labeling, formatting, colors, conditional formats, UI Standards, Help links, etc.

26

� 3 Aspects:• Visibility – see the correct dashboards & subject areas & folders• Capability – Answers access, create iBots, etc.• Data Access – Correct dataset - data filtering is happening

� Some security testing should be done during OBI tests• Are the basics working?• Will it work for ad-hoc?

� Creation of test accounts flows easily from security model design

� Final layer is to run reports as test users and verify data set accuracy

• For this user, for this report, are the numbers what they are supposed to be?

Database

OBI Model

Reports

Security

27

ETL QA Roles

Strong SQL, source system knowledge, source

system access for entering records

Systems Analysts, Developers,

Source SMEs

OBI Model

OBI Answers, knowledge of data model, metric

definitions guide

Systems Analysts, QA Team, OBI

Developers, OBI Architect

Reports

Ability to independently confirm data from the

source, general analyst, business user

Systems Analysts, Business

Analysts, End Users, QA Team

Security

General Analyst or power user skillset, Answers Systems Analysts, Business

Analysts, End Users, QA Team

Infrastructure

Deep technical skills, typically those who set up

the infrastructure

Infrastructure Admins

28

1. Perform QA as early in the process as possible

2. Design a QA plan with your team’s skill sets in mind

3. Plan QA Cycles around the Build

4. Automate as much as possible

5. Don’t ignore OBI Ad-hoc tests

29

Q&A

30

Contact Us

The Leader In Oracle BI & EPM

facebook.com/kpipartners

linkedin.com/company/kpipartners

twitter.com/kpipartners

youtube.com/kpipartners

31

Email: info@kpipartners.com

Web: kpipartners.com/contact

KPI World Headquarters

39899 Balentine Drive

Suite #375

Newark, CA 94560

Phone: (510) 818-9480

Contact Us

The Leader In Oracle BI & EPM

New York, NY

Chicago, IL

Boston, MA

Minneapolis, MN

San Diego, CA

Greensboro, NC

North America Offices

Bangalore, India Hyderabad, India

Global Offices

32

www.kpipartners.com

33