Sybase IQ Data Warehouse

BID201: Sybase IQ Data Warehouse

Can Alhas Application Development Mng.

Yapi Kredi [email protected]

August 15-19, 2004

mailto:[email protected]

Overview

Yapi Kredi Bank

Why Old DW system Requirements Looking for solutions

Benchmark Choose vendors Benchmarking Site visit

•Implementation•Project planning•How We Implement•ETL Tool

•Our New DW System•Technical advantages•Price advantages

•ROI•Today•Questions

Yapı Kredi Bank

Yapi Kredi has over 420 domestic branches and various other subsidiaries, affiliated companies that are active in leasing, factoring, investment banking, insurance, brokerage and new economy companies. Complementing its wide domestic network, the Bank also maintains an important international presence with a subsidiary bank in Dusseldorf and Amsterdam, a bank in the Russian Federation, an off-shore banking unit in Bahrain and four representative offices in Moscow, Munich, Cologne and Stuttgart. It's retail services rank top in the Turkish marketplace with a marketshare greater than 20 percent. Yapi Kredi also ranks number one with respect to the number of issued cards and respective card business volume. It ranks number two among private banks with respect to number of available ATM's.

Old DW System

IBM S80 Model Server12 CPU12 GB RAM

Oracle 8i RDBMS 840 GB Database size

30 Users 2 DBA

2 DataMart96 Tables

Total refresh time : 11 days

IBM P690 Model Server10 CPU16 GB RAM

Oracle 9i RDBMS 2 TB Database size

100 Users 2 DBA

+ DataMart + Tables

Total refresh time : 1 day

Old System Old Target System

Summer- 2003

Requirements

Daily refresh Changing ETL process & design Additional functions New DataMarts

H/W investmentsNeeds growing in size ( New Storage units )Needs growing in CPU ( HW change )

S/W InvestmentAdditional RDBMS licencing

Consultancy fee

Continue with existing System ?

Looking forAlternative solutions ?

Requirements

Investment need

Looking for alternative solutions ?

• Must be in same or low total price• Must be in production in october 2003 ( appr. 3 months )• Must fit all requirements• Must increase query response times• Must integrate with existing tools & systems• Must Keep the Investments For Previous DW

DB Selection For DW

We Choosed 4 main DB vendors to investigate; one of them was Sybase IQ

Our Main Prequisite was the OS it should have work on Unix (AIX) to keep the previous investments.

Chose Vendors For Benchmarking(Candidates)

Define newTehcnical Criterias

Benchmarking

Site Visit

Product Selection

Product Selection Phase

DB Selection For DW

Our Technical Criterions :

•Managability•Concurrency•Partitioning •Compression•Self-tuning memory management•Administration Tools •Programming Language•Performance•Connectivity•Resource Usage



Benchmarking

Product Selection andImplementation

Site Visit


DB Selection For DW

Benchmarking :

We have measured followings :• Query Performance• Storage usage• ETL times• Connectivity

When we try to measure these criterias we realize that



Benchmarking


Site Visit


DB Selection For DW

ETL times : Incremental and Full Refresh

Incremental Refresh

0

500

1000

1500

2000

2500

3000

3500

4000

4500

DB-A DB-B IQ

DB

Tim

e(S

eco

nd

)

Full Refresh

0

20

40

60

80

100

120

140

DB-A DB-B IQ

DB

Min

ute

s



Benchmarking


Site Visit


DB Selection For DW

Storage usage :

DB Size

0

5000

10000

15000

20000

25000

30000

DB-A DB-B IQ

DB

MB



Benchmarking


Site Visit


DB Selection For DW

Query Performance :



Benchmarking


Site Visit


DB Selection For DW

Connectivity :

We are successfully using following tools in our DW and BI enviroments by connecting to Sybase IQ

Reporting: Business ObjectsData Mining: SASCampaign Managements : In-HouseOLAP: MS SQL Server



Benchmarking

Product Selection

Site Visit


DB Selection For DW

Connectivity :

We are successfully using following tools in our DW and BI enviroments by connecting to Sybase IQ

Reporting: Business ObjectsData Mining: SASCampaign Managements : In-HouseOLAP: MS SQL Server



Benchmarking

Product Selection

Site Visit


DB Selection For DW



Benchmarking

Product Selection

Site Visit

Monthly FiguresNumber of queries : 115.000Loaded volumes : 1.5 TB (versus 6/7.5TB according to Gartner)

Load speed : 15 - 30 GB /hr =f(#indexes)Response time :

<1 sec57%

1-3 s16%

1-3 min3%

>3min1%

10-60s12%

3-10s11%

Second largest financial services provider in Benelux


DB Selection For DW



Benchmarking

Product Selection

Site Visit

Sybase IQ is not a ordenary DB that we used to use in our OLTP Systems.

It is designed especially for the datawarehouse and datamart applications. Not for OLTP systems.

And it satisfied all our DW requirments

Especially for Query performance and Storage saving it was much more better than we expected.

We conclude that Product Selection Phase

Implementation

PROJECT PLAN

How We Implement – ETL Tool For IQ

Flowsak is an ETL tool Developed using Java Consists of about 70000 LOC Used in all stages of the DW process, from extracting data

from MVS-DB2 to loading into Unix-IQ In YKB DW Migration Project Total of 2159 files and 178.000

LOC was generated by using Flowsak

How We Implement - Flowsak

Flowsak is used in the following tasks in DW: Create DB2 extract scripts and Mainframe JCLsCreate tables and table indices in IQ DBDetect column cardinalities in Sybase IQ DBCreate views to make transformationsCreate Sybase IQ Procedures

Flowsak is used by all developers in the project

It is not necessary for developers to have deep knowledge in Unix or Mainframe scripting. All scripts are generated by Flowsak



Metadata database is SQL Server 2000 or Sybase ASA First, DW project was developed using Oracle, then it was

transformed into Sybase IQ within 2 months One of the biggest reasons for easy transformation is

Flowsak, the other one is Sybase IQ


Daily Refresh

Daily refresh methodology is used in DW Batch window is between 00:45 – 08:00 About %90 of all data in DW is refreshed daily The rest is refreshed monthly

Sybase IQ Advantages Used

Created LF, HG or DT indices on every column in all tables in DB

Used views to make transformations Used cardinalities for columns in create table and create

index scripts, resulting in less space usage and performance gain

Very high load and index create performance Used JDBC driver to connect from Java

Our New Dw System


1 TB Database size

100 Users 2 DBA

+ DataMart + Tables

Refresh period : Daily

Prev. Target System

Autumn - 2003


300 MB Database size

100 Users 1 DBA

+ DataMart + Tables

Refresh period : Daily

IQ DW Solution

Our New Dw System

After Conversion

Saving in CPU 60%

Saving on disk storage 65%

Saving on RDBMS software licences 60%

Query Response time 10 times

DBA 50%

Nucleus Research – ROI Case Study

ROI: 154%Payback: 10 months

Reponse Time Number of Queries

%

> 5 min 424 (0.59 %)

> 60-120 sec 502 (0.70 %)

> 30- 60 sec 696 (0.98 %)

> 10 -30 sec 1.197 (1.69 %)

> 1- 10 sec 5.006 (7.07 %)

< 1 sec 62.499 ( 88.37 %)

Our New Dw System

Query Results

DW Environment

System Model: IBM,7040-681

Number Of Processors: 4 ( 1704 MHz)

Memory Size: 24576 MB

AIX 5.2

Adaptive Server IQ 12.5

TOTAL DB SIZE : 1.300TB

Current System

DW Environment

# of TABLE : 1397

# of VIEW : 648

# of STORED PROCEDURE : 887

# of COLUMNS : 24354

# of INDEX : 10544

# of HG : 3986 %39

# of HNG : 155 %2

# of LF : 6183 %59

# of UNIX & SQL SCRIPT : 9566

# of USERS : 123

Current System

TABLE WITH MAX ROW COUNT

2.197.961.237

[2 BILLION]

SASUNIX (S80)CUBES

(MS Analysis Server)

OPEN SYSTEMS(Teletel on Oracle etc.)

LO

TU

S D

OM

INO

DB

LO

TU

S D

OM

INO

DB

LEILEI

FLAT FLAT FILESFILES(UNIX)(UNIX)

CDSCDS ODSODS DWDWSTGSTG

DWDW DMDMSTGSTG

DMDM

EDS

PROCESS (ON SYBASE IQ)

P690 R

egatta (UN

IX)

CAMPAIGN

PERSONALDATAFILES

CAMPAIGN DATA &

FEEDBACK

PROCESS (ON SYBASE IQ)

SOURCE SOURCE TABLESTABLES

(DB2)(DB2)

SOURCE FILES(VSAM)

MVSMVS

FTP

FTP

FTP

FTP

CD

S(D

B2)

Data P

ropagator

SE

QU

EN

TIA

LS

SE

QU

EN

TIA

LS

UNLOAD

SORT

UNLOAD

LOAD

Can Alhas

Application Development Mng.

Yapi Kredi Technology

[email protected]

Thank you

Engin Tavşanlı

Database Administration Mng.

Yapi Kredi Technology

[email protected]

http://www.ykb.com/index.html

Documents

Sybase IQ Data Warehouse