28
BID210: Using Sybase IQ as a Report Server for Oracle John Barton Principal Systems Consultant [email protected] August 15-19, 2004

Business Intelligence

  • Upload
    tess98

  • View
    734

  • Download
    0

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: Business Intelligence

BID210: Using Sybase IQ as a Report Server for Oracle

John BartonPrincipal Systems Consultant

[email protected] 15-19, 2004

Page 2: Business Intelligence

2

The Enterprise. Unwired.

Page 3: Business Intelligence

3

The Enterprise. Unwired.

UnwirePeople

UnwireInformation

ManageInformation

Sybase Workspace

Industry and Cross Platform Solutions

• Adaptive Server Enterprise• Adaptive Server Anywhere• Sybase IQ• Dynamic Archive• Dynamic ODS• Replication Server• OpenSwitch• Mirror Activator• PowerDesigner• Connectivity Options• EAServer• Industry Warehouse Studio

• Unwired Accelerator• Unwired Orchestrator• Unwired Toolkit• Enterprise Portal• Real Time Data

Services

• SQL Anywhere Studio• M-Business Anywhere• Pylon Family (Mobile Email)• Mobile Sales• XcelleNet Frontline

Solutions• PocketBuilder• PowerBuilder Family• AvantGo

Page 4: Business Intelligence

4

Moving Oracle Reporting into Sybase IQ

1. Migrate DDL2. Initial Data Load3. Incremental Data Loads4. Reporting SQL

Page 5: Business Intelligence

5

Migrating DDL from Oracle to Sybase IQ

The first step is to convert DDL from Oracle to Sybase IQ

Two recommended methods for migrating DDL

1. Oracle DDL ScriptsData Type Mapping Oracle to Sybase IQ – next slide

Search and Replace in a text editor

2. PowerDesigner Model Migration Use PowerDesigner to reverse engineer the DDL Change the Target Database from Oracle to Sybase IQ Verify Translation

Page 6: Business Intelligence

6

Data Type Mapping Oracle to Sybase IQ

Oracle Datatype Oracle Domain Sybase IQ Data Type Sybase IQ Domainnumber(1,0) 1 <= p tinyint range between 0 and 255number(2,0) 1 <= p <= 2 tinyint range between 0 and 255number(3,0) 1 <= p <= 3 smallint range between -32768 and 32767number(4,0) 1 <= p <= 4 smallint range between -32768 and 32767number(5,0) 1 <= p <= 5 integer 6 <= p <= 9 range +-2147483647number(p,0) 6 <= p <= 38 integer 6 <= p <= 9 range +-2147483647number(p,0) 1 <= p <= 38 bigint 10 <= p <= 18 range +-9223372036854775806number(p,0) 1 <= p <= 38 numeric (p,0) 1 <= p <= 126number(p,s) 1 <= p <= 38 and -84 <= s <= 127 numeric (p,s) 1 <= p <= 126 and 0 <= s <= pdate 4712-01-01BC to 9999-12-31 datetime 0001-01-01 00:00:00.000000 to 9999-12-31 23:59:59.999999date (default time) 4712-01-01BC to 9999-12-31 date 0001-01-01 to 9999-12-31nchar(n) Maximum is 2000 bytes varchar(n) n <= 32767float precision 38 float/real 1.175494351e-38 to 3.402823466e+38. 6 significant digitsfloat(n) precision n can range from 1 to 126 double 2.2250738585072014e-308 and 1.797693134862315708e+308 15 sig digitschar(n) Maximum is 2000 bytes char(n) n <= 255char(n) n > 255 bytes varchar(n) n <= 32767varchar(n) Maximum is 4000 bytes varchar(n) n <= 32767varchar2(n) Maximum is 4000 bytes varchar(n) n <= 32767nvarchar2(n) Maximum is 4000 bytes varchar(n) n <= 32767raw(n) Maximum is 2000 bytes binary(n) n <= 255raw(n) n > 255 bytes varbinary(n) n <= 32767clob character up to 4 GB lob or clob in 12.6 <= 512 TBnclob character up to 4 GB lob or clob in 12.6 <= 512 TBblob character up to 4 GB lob <= 512 TBbfile character up to 4 GB lob stored in IQ compressed!long character up to 2 GB lob <= 512 TBlong raw character up to 2 GB lob <= 512 TB

Use IQ 12.5 ESD 11 or higher with LOB

Page 7: Business Intelligence

7

Using a Text Editor to Migrate DDL

sed – UNIX utility for search and replace

Example: Replace number(1) with tinyint

sed 's/number(1)/tinyint/' ddl_1.sql > ddl_2.sql

Textpad – http://www.textpad.com/ great text editor for full control over search and replace

Page 8: Business Intelligence

8

Using PowerDesigner to Migrate DDL

1. Reverse Engineer Oracle DatabaseDDL ScriptODBC DSN

2. Validate Model

3. Change the Target DatabaseCurrent: OracleNew: Sybase IQ

Page 9: Business Intelligence

9

PowerDesigner DBMS Resource Editor

Oracle 8, 9, 10 Sybase IQ 12.5

Source Database(PhysDataType)

PowerDesignerInternal Data Type

Target Database(AMCDDataType)

Customize DBMS Using Resource EditorThis provides granular control on how data types are converted between source and target databases.

DBMS Properties can be customized (edited)Click on Tools -> Resources -> DBMS

Create a new copy of the DBMS definition file.

Page 10: Business Intelligence

10

Source Data Type Mapping to Internal Types

Page 11: Business Intelligence

11

Target Data Type Mapping from Internal Types

Page 12: Business Intelligence

12

Other DDL Mapping Considerations

Table of data types and mapping Trailing space issue

In Oracle Trailing spaces matter Data Type mapping issues Understanding Data Domains

Select max(col_x), min(col_x) from table_zSelect count(distinct col_x) from table_z

Page 13: Business Intelligence

13

Extracting Data from Oracle

Using Sybase Direct Connect for Oracle Using SQLPlus for Data Extraction

Page 14: Business Intelligence

14

Loading Directly from Oracle

Sybase Enterprise Connect Product FamilyDirectConnect for OracleDirectConnect for Other Supported Databases

DirectConnect for

Oracle

Sybase IQ

Oracle Database

Non-Sybase

Database

OracleInformixMS SQL ServerDB2 UDBDB2 400ODBCMainframe SourcesDirectConnect

Page 15: Business Intelligence

15

Loading Sybase IQ From Oracle

DirectConnect for

Oracle

Sybase IQ

Oracle Database

1 2 34

Client Application

Sequence Steps1. A Client application connects to the Sybase IQ Server and

issue the insert from location command2. The IQ Server opens an Open Client connection to the

DirectConnect for Oracle Access Service3. The DirectConnect Access Service opens a native OCI

connection to the Oracle Server4. The IQ Server sends the Oracle SQL Select statement from

the insert from location command to the DCO Access Service

Page 16: Business Intelligence

16

Loading From Oracle – Continued

DirectConnect for

Oracle

Sybase IQ

Oracle Database

1 2 34 5

67Client Application

8

Sequence Steps - continued

5. The DCO Access Service passes the SQL statement to the Oracle Server

6. The Oracle Server executes the SQL statement and returns the result set to the DCO Access Service

7. The DCO Access Service translates the Oracle result set into an Open Client result set and forwards it to the IQ Server

8. The IQ Server loads the result set

Page 17: Business Intelligence

17

Sybase IQ Syntax to Load from a Remote Table

Connect to Sybase IQ and issue the following command

INSERT INTO OWNER.TABLE-NAME (COLUMN LIST)LOCATION '<DCO SERVER>. <DB NAME>' {ORACLE SQL STATEMENT};

The ORACLE SQL STATEMENT inside the curly brackets can contain any valid SQL in native ORACLE syntax which returns a relational result set.

This SQL gets passed directly through to the ORACLE Server without modification.

Page 18: Business Intelligence

18

DirectConnect – Load Performance

Customer Testimonial

“We run batch processes at night that use the LOCATION clause to insert data from Oracle into Sybase IQ”

“We are generally inserting about 1 million rows into tables that contain 22 to 34 columns. The insert for each table takes about 5 minutes. This is on a 2 CPU server with 4 GB memory. We also load very similar data from a file using the IQ LOAD TABLE command which takes about 60 seconds”

“If the volume of data is low (a couple million rows or less), then using the LOCATION clause to get Oracle data into Sybase IQ is definitely the way to go because of the ease of use and convenience”

Remember to consider total time = extract, transfer and load

Page 19: Business Intelligence

19

Extracting Data from Oracle Using SQLPlus

A SQL Plus script can be used to extract data from Oracle Database The results are written to a file or named pipe in delimited format IQ can load from a file or named pipe which is feed by SQLPlus extract Limited only by Oracle SQL Capabilities

Gather Timings by running SQLPlus from local and remote hostsMeasure Extract Speed of local transfer vs remote transferNetwork overhead involved in the network transfer

Network transfer overhead = Remote transfer time - Local transfer time

Sybase IQ

Oracle Database

SQLPlus

File System

SQLPlus

File System

Page 20: Business Intelligence

20

SQLPlus Extraction Script

A SQL Plus script

--Set the line size to accommodate the longest possible line.SET LINESIZE 136--Turn off all page headings, column headings, etc.SET PAGESIZE 0SET FEEDBACK OFFSET TERMOUT OFFSPOOL /stage/emp_data.csvSELECT TO_CHAR(employee_id) || ‘,’ || employee_name || ‘,’ || TO_CHAR(employee_hire_date,’YYYYMMDD’) || ‘,’ || TO_CHAR(employee_termination_date, ‘YYYYMMDD’) || ‘,’ || TO_CHAR(employee_billing_rate) || ‘,~’ FROM employee;SPOOL OFF

Page 21: Business Intelligence

21

Load Processing Techniques

Initial Load Ongoing Load Incremental Load Techniques Converting SQL Loader Scripts

Page 22: Business Intelligence

22

ETL Examples - Managing Data Loads

Tables and Dimensions can be categorized as follows:Dimension Type   Change

ProcessingHistory Tracking

  Static None None

  Slowly Growing New records added

None

Type 1* Slowly Changing, Update with Changes

Content updated

None

Type 2* Slowly Changing, Versioning of Changes

Content updated, new records added

Full

Type 3* Slowly Changing, Keep Current and Previous values

Content updated

Partial

* Dimensions as defined by Ralph Kimball

Page 23: Business Intelligence

23

Slowly Growing Dimensions or Tables

CREATE TABLE CUST( CUST_ID INTEGER, FIRST_NAME char(30), LAST_NAME char(30), ADDRESS char(60), CITY char(30), STATE char(2), ZIP char(5));

Dimension Type

  Change Processing

History Tracking

  Static None None

  Slowly Growing New records added

None

LOAD TABLE CUST( CUST_ID '|', FIRST_NAME '|', LAST_NAME '|', ADDRESS '|', CITY '|', STATE '|', ZIP '|‘) from 'C:\\Temp\\techwave\\cust.txt'quotes off escapes off preview onrow delimited by '\n';

INSERT INTO CUST (CUST_ID, FIRST_NAME, LAST_NAME, ADDRESS, CITY, STATE, ZIP) LOCATION ‘SERVER.DBNAME’ {SELECT CUST_ID,

FIRST_NAME, LAST_NAME, ADDRESS, CITY, STATE, ZIP FROM CUST};

Load Table or

INSERT LOCATIONUsed interchangeably

Page 24: Business Intelligence

24

Loading Type 1 Dimension Tables

INSERT INTO STAGE_SUPPLIER ( S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT) LOCATION ‘SERVER.DBNAME’

{ SELECT S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT FROM SUPPLIER};

Dimension Type   Change Processing

History Tracking

Type 1 Slowly Changing, Update with Changes

Content updated

None

UPDATE SUPPLIER SET S_ACCTBAL = STAGE_SUPPLIER.S_ACCTBAL FROM STAGE_SUPPLIER, SUPPLIERWHERE STAGE_SUPPLIER.S_NAME = SUPPLIER.S_NAME AND STAGE_SUPPLIER.S_ADDRESS = SUPPLIER.S_ADDRESS AND STAGE_SUPPLIER.S_PHONE = SUPPLIER.S_PHONE AND STAGE_SUPPLIER.S_ACCTBAL != SUPPLIER.S_ACCTBAL

INSERT INTO SUPPLIER SELECT STAGE_SUPPLIER.* FROM STAGE_SUPPLIER WHERE STAGE_SUPPLIER.S_NAME NOT IN (SELECT SUPPLIER.S_NAME FROM SUPPLIER WHERE STAGE_SUPPLIER. S_NAME = SUPPLIER. S_NAME)

UpdateExisting

InsertNew

LoadStagingTable

Page 25: Business Intelligence

25

Loading Type 2 Dimension Tables

Dimension Type   Change Processing

History Tracking

Type 2 Slowly Changing, Versioning of Changes

Content updated, new records added

Full

UPDATE SUPPLIER SET S_CURR_FLAG = ‘N’ FROM STAGE_SUPPLIER, SUPPLIERWHERE STAGE_SUPPLIER.S_NAME = SUPPLIER.S_NAME INSERT INTO SUPPLIER SELECT STAGE_SUPPLIER.*, ‘Y’ FROM STAGE_SUPPLIER

UpdateExisting

Insert

LoadStagingTable

INSERT INTO STAGE_SUPPLIER (S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT) LOCATION ‘SERVER.DBNAME’

{ SELECT S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT FROM SUPPLIER};

Page 26: Business Intelligence

26

Generating Sequence Numbers for Keys

Adding Sequence Numbers / Generating Warehouse Keys

CREATE TABLE STAGE_SUPPLIER ( S_NAME char(25), S_ADDRESS varchar(40), S_NATIONKEY integer, S_PHONE char(15), S_ACCTBAL double precision, S_COMMENT varchar(101) );

CREATE VARIABLE suppkey INTEGER;SELECT isnull(max(S_SUPPKEY),0) INTO suppkey FROM SUPPLIER;

INSERT INTO SUPPLIERSELECT number(*)+suppkey, S_NAME, S_ADDRESS, S_NATIONKEY, S_PHONE, S_ACCTBAL, S_COMMENT FROM STAGE_SUPPLIER;

FIND MAX KEY

VALUE

LOADDATA

number(*) Function:Sequential numbers 1 to N

Page 27: Business Intelligence

27

Why Use Sybase IQ over Oracle

ANSI SQL Reduced Administration Reduced Ongoing Tuning Data Storage Simplified Multi User Performance Hardware Costs IQ Indexing outperforms Oracle Indexing

Page 28: Business Intelligence

28

Changing Tools from Oracle to Sybase IQ

Query Tools SQL

Standard SQL 92 Entry Level Compliant

ODBC Business Intelligence Tools Business Objects Crystal Reports Cognos MicroStrategy Brio MS Access/Excel