Platform: z/OS Visualizing DB2 Performance for Your

1

Visualizing DB2 Performance for Your Applications

Rob Schumm / HSBC-US Technical Services

Session: A4Tuesday, May 24, 2005 / 8:30 AM - 9:40 AM

Platform: z/OS

2

2

System Workload

0

100000

200000

300000

400000

500000

600000

02/15

/2004

02/29

/2004

03/14

/2004

03/28

/2004

04/11

/2004

04/25

/2004

05/09

/2004

05/23

/2004

06/06

/2004

06/20

/2004

07/04

/2004

07/18

/2004

08/01

/2004

08/15

/2004

08/29

/2004

09/12

/2004

09/26

/2004

10/10

/2004

10/24

/2004

11/07

/2004

11/21

/2004

12/05

/2004

12/19

/2004

01/02

/2005

01/16

/2005

01/30

/2005

02/13

/2005

Week Beginning On

DB

2 C

lass

1 C

PU S

econ

ds

Business Unit 1OtherBusiness Unit 2Business Unit 3Corporate SystemsSystems Software

This presentation is about producing and using “workload charts”, like this . . .One of the many DB2 performance attributes you can chart.

CPU consumption aggregated over a 1 week period, in a “stacked area” format. A stacked area chart represents a workload as a sum of discrete components.

Workload breakdown on this chart is by business unit.

53 week rolling window

3

3

“Drill down” through layered charts, to detailed performance data . . .

System Workload

0

100000

200000

300000

400000

500000

600000

02/15

/2004

02/29

/2004

03/14

/2004

03/28

/2004

04/11

/2004

04/25

/2004

05/09

/2004

05/23

/2004

06/06

/2004

06/20

/2004

07/04

/2004

07/18

/2004

08/01

/2004

08/15

/2004

08/29

/2004

09/12

/2004

09/26

/2004

10/10

/2004

10/24

/2004

11/07

/2004

11/21

/2004

12/05

/2004

12/19

/2004

01/02

/2005

01/16

/2005

01/30

/2005

02/13

/2005

Week Beginning On

DB

2 C

lass

1 C

PU S

econ

ds

Business Unit 1OtherBusiness Unit 2Business Unit 3Corporate SystemsSystems Software Business Unit 1

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

02/15

/2004

02/29

/2004

03/14

/2004

03/28

/2004

04/11

/2004

04/25

/2004

05/09

/2004

05/23

/2004

06/06

/2004

06/20

/2004

07/04

/2004

07/18

/2004

08/01

/2004

08/15

/2004

08/29

/2004

09/12

/2004

09/26

/2004

10/10

/2004

10/24

/2004

11/07

/2004

11/21

/2004

12/05

/2004

12/19

/2004

01/02

/2005

01/16

/2005

01/30

/2005

02/13

/2005

Week Beginning On

OtherSupportEnd User SQLData WarehouseODSShadow Tables

BU1-ODS

0

10000

20000

30000

40000

50000

60000

02/1

5/20

04

02/2

9/20

04

03/1

4/20

04

03/2

8/20

04

04/1

1/20

04

04/2

5/20

04

05/0

9/20

04

05/2

3/20

04

06/0

6/20

04

06/2

0/20

04

07/0

4/20

04

07/1

8/20

04

08/0

1/20

04

08/1

5/20

04

08/2

9/20

04

09/1

2/20

04

09/2

6/20

04

10/1

0/20

04

10/2

4/20

04

11/0

7/20

04

11/2

1/20

04

12/0

5/20

04

12/1

9/20

04

01/0

2/20

05

01/1

6/20

05

01/3

0/20

05

02/1

3/20

05

Week Beginning On

Batch RetrievalPurge ProgramsDB MaintenanceUpdate ForeignUpdate US XREF TableUpdate US OtherUpdate US Sys1 FiltUpdate US System 1Update US System 2Update US System 3

DB2 accounting records, loaded into a table, through a

“workload classification” view

The workload is classified in business terms, not in technical terms. The “drill down” goes through three layers of charts, and also into the detailed performance data that produced the charts.

4

4

BU1-ODS

0

10000

20000

30000

40000

50000

6000002

/15/

2004

02/2

9/20

04

03/1

4/20

04

03/2

8/20

04

04/1

1/20

04

04/2

5/20

04

05/0

9/20

04

05/2

3/20

04

06/0

6/20

04

06/2

0/20

04

07/0

4/20

04

07/1

8/20

04

08/0

1/20

04

08/1

5/20

04

08/2

9/20

04

09/1

2/20

04

09/2

6/20

04

10/1

0/20

04

10/2

4/20

04

11/0

7/20

04

11/2

1/20

04

12/0

5/20

04

12/1

9/20

04

01/0

2/20

05

01/1

6/20

05

01/3

0/20

05

02/1

3/20

05

Week Beginning On


There is detailed performance data “behind the charts” . . .

For details on this workload, use this query:

SELECT A.TIMEPERIOD_WK, A.TIMEPERIOD_DT ,A.CONNECT_TYPE, A.CONNECT_ID, A.CORRNAME, A.PLAN_NAME ,A.CLASS1_ELAPSED, A.CLASS2_ELAPSED ,A.CLASS1_CPU_TOTAL, A.CLASS2_CPU_TOTAL

FROM ACTG_GNRL_Vx A <- a view of detailed performance dataACTG_CTRL C <- report titles & workload categories

WHERE A.WORKLOAD_CLASS = C.WORKLOAD_CLASS <- join predicateAND

C.REPORT_TITLE = 'BU1-ODS ' AND C.REPORT_CATEGORY = 'Update US Sys1 Filt ' AND

A.TIMESTAMP_DATE BETWEEN date1 AND date2ORDER BY CORRNAME, TIMEPERIOD_DT ;

Notice that the ACTG_CTRL table allows us to retrieve detailed performance data with “business language” predicates that refer directly to the chart.

This is where we tuned the unload jobs.

This is the categoryto look at!

This is a real life example from HSBC / Household. In early 2004, we had excessive costs for the ODS system. The charts cued us to the “Update US Sys 1 Filt”workload, which removes unwanted accounts from the daily ODS update. We found fourteen daily dynamic SQL unload jobs using a “finder file” to filter out the accounts. We EXPLAINed the SQL, tuned it, and implemented a fix. The savings were immediately obvious on the charts. This workload is still consuming about 10,000 CPU seconds per week, and climbing, even after the December purge. This workload is still a major concern to us.

Decemberpurge!

5

5

Visualizing DB2 Performance• Tell the “truth” about resource consumption to internal

business customers, IT managers, and the technical staff, with language that everyone understands . . .• Where are your processing resources going?

• > Subsystem level > Business unit level > Application level• What segments of your workload are the most volatile? • What are the trends?• How did a change to a system or an application affect resource

consumption?• How does one segment of the workload compare to another?• Big Picture: What matters?

6

6

DB2 Performance Metrics• Dictionary.com: “Metrics - The application of statistics and

mathematical analysis to a field of study”.• We study our DB2 systems to fix them, to tune them, and to

exploit improvements in the database management software.• Here is what we expect from our metrics.

• Accurate, comprehensive and useful information.• Information presented in business terms.

• Establish a common understanding of performance for IT managers,business users and technical staff.

• Reasonable costs to maintain and operate.• Adding a new workload to the system takes less than 1/2 day.• Weekly chart update takes about 5 minutes.

7

7

DB2 Performance Metrics• How do we use our performance metrics?

• To inform management about system and application performance.• To review the workload at various levels and determine trends.• To quantify resource usage and direct our tuning efforts.• To monitor the effects of tuning a program or process, changes in

data structures or modifications at the system level.• To point out the need for purging unneeded data from systems.

8

8

Development DB2 Subsystem (z/OS or LUW)

ETLRoutines

ACTG_GNRL_Vx View

Rollup

ACTG_SUMM_DLY Table

Performance DatabaseDB2 Statistics and Accounting Data

ACTG_GNRL Table

System Architecture

SMF(usually on tape)

ApplicationWorkload

Production System (z/OS)

TraceData

(101, etc.)

Production DB2 Subsystem

-START TRACE ...

AnalyticQueries

ACTG_CTRL Table

Workbook Graphics Application

Charts

Chart Grids

Time Period Query / Chart Query

SQL

Publish

FRONT

END

BACK

END

9

9

DB2 “Accounting 101” - System Infrastructure• DB2’s produces a wealth of performance data.

• DB2 “traces” control “what” and “where”.• Most trace data goes to the system’s SMF facility, which spools it out to

tape, interspersed with trace data from other system software.• DB2 “Statistics” records (which are not the topic of this presentation)

capture the status of the DB2 subsystem every 30 minutes.• DB2 “Accounting” records (AKA “101 records”) capture detailed

information on the application workload, for each thread. This is the data you need to produce workload charts.

• The 101 record has a complex structure, it is not a simple “flat file”• IBM (and other vendors) provide routines to pull all of the DB2 trace

data out of SMF, produce generic reports, and populate a performance database.

10

10

DB2 “Accounting 101” - System Infrastructure

• Generally, DB2 starts an accounting record when a new thread allocates, and cuts it to SMF when it terminates.• Volume depends on the application workload.

• OLAP / DSS workloads generally produce fewer accounting records than OLTP workloads, because there are fewer threads.

• Volume depends on how your DB2 is tuned and configured.• For OLTP workloads, CICS / DB2 “thread reuse” rolls up multiple

transactions into a single thread.• Accounting records can be suppressed for specific workloads.

• OLAP systems are easier to analyze.• This presentation is based on work with an OLAP system.• Understand how your systems are configured.

11

11

DB2 “Accounting 101” - System Infrastructure• Each accounting record has hundreds of data elements.

• Identifying elements correlate threads to application processes; you need this correlation to understand, manage and tune your workload.

• Date, time, plan, authid, connection id, correlation id, etc.• Measurement elements record the type and amount of processing

done, resources consumed, wait times, and high water marks.• Here are some issues to keep in mind.

• Maintaining accounting history for a year or more takes planning.• The identifying elements are technical attributes that identify the

workload on a system in terms meaningful to the technical staff.• The time and date elements need refinement to facilitate roll up into the

time periods that we want to report on.

• See Appendix 1 for more details.

12

12

Producing Workload Charts• What skills are required?• Back End - Mainframe Infrastructure (mostly)

• Build a performance database• This is a generic database

• Customize for your workload• Create a view to classify the workload and define time periods.

• Aggregate / summarize data from the accounting table• Use the performance database

• Front End - Chart Queries and Desktop Graphics• Develop chart queries to produce chart-ready data• Build a control table (optional; or you can add it later)• Build a workbook application to produce the charts

• Operating and maintaining the system

13

13

What Skills are Required?• SQL

• Date / time arithmetic• CASE logic• Pivot queries• GROUP BY aggregation• Nested table expressions

• DB2 Systems• SMF data extract and load• Create table and indexes• DB2 Connect

• Desktop Software• Downloading text files• Workbook application (i.e.

Excel)• Running queries on the host• Charts and graphs• Macros (not used here, but I

suspect that macros would be very helpful)

• Procedural Programming• None at all

Business knowledge, to classify your workload . . . Access to a color printer . . .

14

14

Back End - Build a Performance Database• The popular performance monitoring packages (IBM DB2PE

and 3rd party products) include software to:• Create a performance database.• Extract DB2 performance data (accounting and statistics) from SMF. • Untangle the data and load it into tables in the performance database.

• Build your performance database on a development system:• You do not need production change control for your metrics system.• Keep resource-intensive analytic queries off your production system.

• You do not have to build it on the mainframe, you can use another platform (Linux / Unix / Windows)

15

15

Back End - Build a Performance Database• We built our database a few years ago, using routines provided with IBM

"DB2 Performance Expert" (DB2-PE).• For the OLTP DB2 subsystems, we load detailed accounting data but we

do not retain it very long, because of the volume. For the long term, we aggregate and summarize to reduce the volume.

• For our OLAP systems, we save over a year of detailed history.• We scheduled two daily jobs to capture the DB2 trace data.

• JOB#1 extracts DB2 trace records (accounting, statistics, dynamic SQL) from daily SMF.

• JOB#2 splits these records and loads tables, including the ACTG_GNRL table that supports our workload analysis.

• We maintain about 5 million rows in our ACTG_GNRL table.

16

16

Back End - Build a Performance DatabaseOur ACTG_GNRL table contains 313 elements. You only have to understand the ones that you use.

Here are the identifying elements that we use to classify our workload:

Column Name Col Type Length Description------------------ -------- ------ --------------------------------------------------------TIMESTAMP TIMESTMP 10 When the thread finishedCONNECT_TYPE CHAR 8 BATCH, DB2CALL, TSO, SERVER, UTILITY, CICS region nameCONNECT_ID CHAR 8 Identifies remote threadsCORRNAME CHAR 8 Job name for batch jobsPLAN_NAME CHAR 8 DB2 planPRIMAUTH CHAR 8 Authorization ID

Here are some of the measurement elements that you can use to analyze performance:

Column Name Col Type Length Description------------------ -------- ------ --------------------------------------------------------CLASS1_TIME_BEG TIMESTMP 10 Track batch SLA on-time performance?CLASS1_TIME_END TIMESTMP 10 Track batch SLA on-time performance?CLASS1_ELAPSED DECIMAL 13 Track batch elapsed time performance?CLASS1_CPU_TOTAL DECIMAL 13 Captures most CPU consumed <- used in this presentation.CLASS2_ELAPSED DECIMAL 13 In-DB2 elapsedCLASS2_CPU_TOTAL DECIMAL 13 In-DB2 CPU

Now you have detailed information to analyze your workload, but there are two issues.• How do you identify time periods (day, week, month) to GROUP BY for aggregation?• You need application specific knowledge to classify the workload (translate job names, plan names, etc.).

17

17

Back End - Customize for Your Workload• Put a “workload classification” view on top of the ACTG_GNRL table.

• Generate time elements to assign each record to a month, week, day and hour.• This facilitates monthly, weekly, daily or hourly rollup.

• Add a CHAR(30) element - WORKLOAD_CLASS.• Implement your application knowledge in CASE logic to interrogate identifying

elements in the accounting record and assign a text string that describes and categorizes the workload in business terms.

• This allows you to query your accounting data in business terms.• We break our production system out into about 60 workload classes.• This view changes with the workload, over time.• This is the most time consuming task of the whole development process.• Query the ACTG_GNRL table, watch the monitors, make phone calls, track

people down, dig up old documentation, review batch job output, etc.• This is another reason to build your performance database in development;

you do not want to do all this in your production environment.• Put a version identifier in your view name (“V1”,”V2”)

18

18

CREATE VIEW ACTG_GNRL_Vx AS SELECT <-- The suffix “Vx” represents “V0”, “V1”, etc., to support versioning

DATE(DAYS(TIMESTAMP_DATE)-DAYOFMONTH(TIMESTAMP_DATE)+1) <-- Generates the first day of the month for any dateAS TIMEPERIOD_MO

,DATE(DAYS(TIMESTAMP_DATE)-DAYOFWEEK(TIMESTAMP_DATE)+1) <-- Generates the first day of the week for any dateAS TIMEPERIOD_WK (for a Sunday to Saturday week, this is always a Sunday)

,TIMESTAMP_DATE AS TIMEPERIOD_DT <-- Generates the date,TIMESTAMP_HOUR AS TIMEPERIOD_HR <-- Generates the hour of the day,DAYOFWEEK(TIMESTAMP_DATE) AS TIMEPERIOD_DW <-- Generates the day of the week (1 for Sunday, 7 for Saturday)

,CASE <-- Case statement uses ‘waterfall logic’ to assign WORKLOAD_CLASS------------------------------------------------------ CICS WORKLOAD ----------------------------------------------------WHEN CONNECT_TYPE = 'CICS' <-- Our DSS system has very little CICS work, pull it off first

THEN 'CICS Unclassified ' ------------------------------------------------------ SYSTEM SOFTWARE ----------------------------------------------------WHEN CORRNAME LIKE 'DB2%'

THEN 'DB2 Started Tasks ' - - - - - - - - - - - - - - - - 6 Line(s)

------------------------------------------------------ APPLICATION PRODUCTION BATCH ----------------------------------------------------WHEN CORRNAME LIKE 'PABC%' <-- This is a typical entry for a batch workload component

THEN ’ABC System Production Batch ' WHEN CORRNAME LIKE 'PXYZ%' <-- Ditto

THEN 'XYZ System Production Batch ' - - - - - - - - - - - - - - - 127 Line(s)

------------------------------------------------------ EVERYTHING ELSE IS NOT CLASSIFIED ----------------------------------------------------

ELSE 'Other / Unclassified ' <-- Whatever drops through all of the ‘WHENs’ ends up hereEND AS WORKLOAD_CLASS

,T.* <-- Returns every column from the accounting table

FROM ACTG_GNRL T ;

Back End - Customize for Your Workload

Time Elements

WORKLOAD_CLASS

19

19

Back End - Customize for Your Workload

Daily RollupSELECT TIMPERIOD_DT, WORKLOAD_CLASS, SUM(CLASS1_CPU_TOTAL) FROM view GROUP BY TIMEPERIOD_DT, WORKLOAD_CLASS ;Weekly RollupSELECT TIMPERIOD_WK, WORKLOAD_CLASS, SUM(CLASS1_CPU_TOTAL) FROM view GROUP BY TIMEPERIOD_WK, WORKLOAD_CLASS ;Monthly RollupSELECT TIMPERIOD_MO, WORKLOAD_CLASS, SUM(CLASS1_CPU_TOTAL) FROM view GROUP BY TIMEPERIOD_MO, WORKLOAD_CLASS ;

The TIMEPERIOD elements support rollup by day, week, month (shown), also day-of-week and hour-of-day (not shown).

TIMEPERIOD_MO TIMEPERIOD_WK TIMEPERIOD_DT TIMEPERIOD_DW WORKLOAD_CLASS TIMESTAMP_DT CONNECT_ID CONNECT_TYPE CORRNAME CLASS1_CPU_TOTAL12/01/2005 12/26/2004 12/31/2004 6 . . . w hatever . . . 12/31/2004 ? ? ? ?01/01/2005 12/26/2004 01/01/2005 7 ABC System Production batch 01/01/2005 BATCH TSO PABCPGM2 845.278301/01/2005 12/26/2004 01/01/2005 7 ABC System Production batch 01/01/2005 BATCH TSO PABCJOB2 58.981101/01/2005 12/26/2004 01/01/2005 7 XYZ System Production Batch 01/01/2005 BATCH TSO PXYZ001 43.452701/01/2005 12/26/2004 01/01/2005 7 XYZ System Production Batch 01/01/2005 BATCH TSO PXYZ002 2.720901/01/2005 12/26/2004 01/01/2005 7 XYZ System Production Batch 01/01/2005 BATCH TSO PXYZ003 25.983401/01/2005 12/26/2004 01/01/2005 7 CICS Unclassif ied 01/01/2005 CICSRGN1 CICS TXN1 0.332101/01/2005 12/26/2004 01/01/2005 7 CICS Unclassif ied 01/01/2005 CICSRGN2 CICS TXN2 1.240901/01/2005 01/01/2005 01/02/2005 1 ABC System Production batch 01/02/2005 BATCH TSO PABCPGM2 752.783401/01/2005 01/01/2005 01/02/2005 1 ABC System Production batch 01/02/2005 BATCH TSO PABCJOB2 63.201201/01/2005 01/01/2005 01/02/2005 1 XYZ System Production Batch 01/02/2005 BATCH TSO PXYZ001 49.820801/01/2005 01/01/2005 01/02/2005 1 XYZ System Production Batch 01/02/2005 BATCH TSO PXYZ002 3.445501/01/2005 01/01/2005 01/02/2005 1 XYZ System Production Batch 01/02/2005 BATCH TSO PXYZ003 28.770101/01/2005 01/01/2005 01/03/2005 2 CICS Unclassif ied 01/03/2005 CICSRGN1 CICS TXN1 0.332101/01/2005 01/01/2005 01/03/2005 2 CICS Unclassif ied 01/03/2005 CICSRGN2 CICS TXN2 1.240901/01/2005 01/01/2005 01/03/2005 2 XYZ System Production Batch 01/03/2005 BATCH TSO PXYZ001 49.820801/01/2005 01/01/2005 01/03/2005 2 XYZ System Production Batch 01/03/2005 BATCH TSO PXYZ002 3.445501/01/2005 01/01/2005 01/03/2005 2 XYZ System Production Batch 01/03/2005 BATCH TSO PXYZ003 28.770101/01/2005 01/01/2005 01/03/2005 2 CICS Unclassif ied 01/03/2005 CICSRGN1 CICS TXN1 0.332101/01/2005 01/01/2005 01/03/2005 2 CICS Unclassif ied 01/03/2005 CICSRGN2 CICS TXN2 1.240901/01/2005 01/01/2005 01/04/2005 3 . . . w hatever . . . 01/04/2005 ? ? ? ?

From the TableFrom the View

20

20

Back End - Aggregate / Summarize• Now we have access to raw (thread level) accounting data in the

ACTG_GNRL table, through the ACTG_GNRL_Vx view that classifies the workload and provides rollup time elements.• For one year of history, you probably have millions of rows to roll up; this

will be expensive.• Summarize the data into daily, weekly or monthly tables.

• We build a daily summary in ACTG_SUMM_DLY.• The volume of data in our daily summary table is reasonable: 365 days of

history x 60 workload classes rolled up each day is less that 30K rows.• Execute analytic queries on the summary tables.• For detailed information, you can go back to the detailed data in

ACTG_GNRL.

21

21

Back End - Aggregate / SummarizeCREATE TABLE ACTG_SUMM_DAILY (

---- AGGREGATION ELEMENTS FROM THE VIEW --

TIMEPERIOD_DT DATE NOT NULL ,WORKLOAD_CLASS CHAR(30) FOR SBCS DATA NOT NULL ,TIMEPERIOD_MO DATE ,TIMEPERIOD_WK DATE ,TIMEPERIOD_DW SMALLINT ,COUNT INTEGER

---- AGGREGATED ELEMENTS FROM ACCOUNTING GENERAL TABLE --,DB2PM_REL SMALLINT

- - - - - - - - - - - - - - - - 38 Line(s) not Displayed,CLASS1_CPU_TOTAL DECIMAL(13, 6)

- - - - - - - - - - - - - - - 272 Line(s) not Displayed,PAR_ONEDB2_DCLTT INTEGER

)• We are going to preprocess the accounting data with a daily rollup.• Unique index / key on TIMEPERIOD_DT, WORKLOAD_CLASS.• This will provide a tremendous reduction in data volume.• For 50 workload classes this table will have, at most, 50 rows per day.

Roll-up elements(GROUP BY)

Aggregation elements(SUM, MAX)

22

22

Back End - Aggregate / SummarizeSELECT TIMEPERIOD_DT

,WORKLOAD_CLASS ,MAX(TIMEPERIOD_MO) AS TIMEPERIOD_MO ,MAX(TIMEPERIOD_WK) AS TIMEPERIOD_WK ,MAX(TIMEPERIOD_DW) AS TIMEPERIOD_DW ,COUNT(*) AS COUNT ,MAX(DB2PM_REL ) AS DB2PM_REL

- - - - - - - - - - - - - - - 38 Line(s) not Displayed,SUM(CLASS1_CPU_TOTAL ) AS CLASS1_CPU_TOTAL

- - - - - - - - - - - - - - - 9 Line(s) not Displayed,SUM(CLASS2_CPU_TOTAL ) AS CLASS2_CPU_TOTAL

- - - - - - - - - - - - - - - 60 Line(s) not Displayed,SUM(SELECT ) AS SELECT ,SUM(INSERT ) AS INSERT ,SUM(UPDATE ) AS UPDATE ,SUM(DELETE ) AS DELETE

- - - - - - - - - - - - - - 198 Line(s) not Displayed,MAX(PAR_ONEDB2_DCLTT ) AS PAR_ONEDB2_DCLTT

FROM ACTG_GNRL_VX <- use the viewWHERE TIMESTAMP_DATE >= STARTING_DATE GROUP BY TIMEPERIOD_DT, WORKLOAD_CLASS ; This drives the rollup.

Time period / how much do you want to roll up?

• Execute this query with an unload program, (HPU, DSNTIAUL, etc).• To start off, LOAD REPLACE ACT_SUMM_DLY with roll-up of all history from the ACTG_GNRL_VX view.• Going forward, you can run a daily or weekly incremental roll and then run a LOAD RESUME.• We update weekly, so our predicate on TIMESTAMP_DATE looks at the last 7 days only.• Your summary table index will cause your LOAD RESUME to discard any rows that you accidentally load twice.• After the load, execute validation queries (see Appendix 2) to verify that no data is missing. • When you update your workload classification view, you should re-aggregate all of your history and reload.

You have hundreds of raw elements from the accounting table to roll up (with a column function) under the GROUP_BY. For character, date and time elements, your rollup should be MAX or MIN (I use MAX). For numeric elements, your default rollup should be MAX, (not SUM). Before you SUM a numeric, make sure that is a measure of some sort of activity, as opposed to a code or a “high water mark”; do not SUM those. Here is a good rule of thumb to start off: MAX everything except for a handful of numeric elements that you plan to analyze. If you have a problem with a SUM, you will see an overflow message in your processing.

23

23

Back End - Use the Performance Database

• Now you have some useful assets :• You have the ACTG_GNRL table, with thread level information.• You have the ACTG_GNRL_VX view that presents the detailed data

by workload class and time period (month, week, day, hour).• You have summary tables (like ACTG_SUMM_DLY) with rolled up

data. This is the source data for the charts.• You have processes in place to update and maintain those tables.

• Anyone familiar with your applications can use basic SQL skills to query the view to analyze the workload.

• Run ad-hoc queries to understand the data and learn how to generate time series.

24

24

System Architecture

Development DB2 Subsystem (z/OS or LUW)

ETLRoutines

ACTG_GNRL_Vx View

Rollup

ACTG_SUMM_DLY Table

Performance DatabaseDB2 Statistics and Accounting Data

ACTG_GNRL Table

SMF(usually on tape)

ApplicationWorkload

Production System (z/OS)

TraceData

(101, etc.)

Production DB2 Subsystem

-START TRACE ...

AnalyticQueries

BACK

END

FRONT

END

ACTG_CTRL Table

Workbook Graphics Application

Charts

Chart Grids

Time Period Query / Chart Query

SQL

Publish

Do the presentation graphicson the desktop.

Do the “heavy lifting”with SQL in DB2.

25

25

Detail charts “drill down” into the workload.

System Workload

0

100000

200000

300000

400000

500000

600000

02/15

/2004

02/29

/2004

03/14

/2004

03/28

/2004

04/11

/2004

04/25

/2004

05/09

/2004

05/23

/2004

06/06

/2004

06/20

/2004

07/04

/2004

07/18

/2004

08/01

/2004

08/15

/2004

08/29

/2004

09/12

/2004

09/26

/2004

10/10

/2004

10/24

/2004

11/07

/2004

11/21

/2004

12/05

/2004

12/19

/2004

01/02

/2005

01/16

/2005

01/30

/2005

02/13

/2005

Week Beginning On

DB

2 C

lass

1 C

PU S

econ

ds

Business Unit 1OtherBusiness Unit 2Business Unit 3Corporate SystemsSystems Software Business Unit 1

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

02/15

/2004

02/29

/2004

03/14

/2004

03/28

/2004

04/11

/2004

04/25

/2004

05/09

/2004

05/23

/2004

06/06

/2004

06/20

/2004

07/04

/2004

07/18

/2004

08/01

/2004

08/15

/2004

08/29

/2004

09/12

/2004

09/26

/2004

10/10

/2004

10/24

/2004

11/07

/2004

11/21

/2004

12/05

/2004

12/19

/2004

01/02

/2005

01/16

/2005

01/30

/2005

02/13

/2005

Week Beginning On


BU1-ODS

0

10000

20000

30000

40000

50000

60000

02/1

5/20

04

02/2

9/20

04

03/1

4/20

04

03/2

8/20

04

04/1

1/20

04

04/2

5/20

04

05/0

9/20

04

05/2

3/20

04

06/0

6/20

04

06/2

0/20

04

07/0

4/20

04

07/1

8/20

04

08/0

1/20

04

08/1

5/20

04

08/2

9/20

04

09/1

2/20

04

09/2

6/20

04

10/1

0/20

04

10/2

4/20

04

11/0

7/20

04

11/2

1/20

04

12/0

5/20

04

12/1

9/20

04

01/0

2/20

05

01/1

6/20

05

01/3

0/20

05

02/1

3/20

05

Week Beginning On


DB2 accounting records, loaded into a table, through a

“workload classification” view

26

26

Front End - Chart Queries & Desktop Graphics• Your “workbook application” produces the charts.

• It runs two embedded queries.• The “chart query” generates the time series data arrays for each chart.

• A control table defines a hierarchy of charts for drill-down through 3 layers of detail, so that you can generate all of the charts with one query.

• The “time period query” generates time period labels for the charts.• Develop both queries to “roll forward” in time, so you do not have to

update time range predicates each period. • We do our query development on the mainframe with DSNTEP2, and

then we paste the finished query into the workbook application.• It merges the time periods and data arrays into “chart grids, that

organize the data exactly how the graphics engine prefers to see it.• It draws the charts from the chart grids.

27

27

Front End - . . . chart query processing• Pull 53 weeks of data from ACT_SUMM_DLY, roll up by

week and workload class, aggregate Class 1 CPU, and calculate a relative time period.

• “Pivot” the 53 rows of data for each workload class.• This produces one row for each class; a 53 week time series of weekly

Class 1 CPU consumption. You can build charts from here.

• . . . For a hierarchy of nested charts, join to a control table.• Use our query as a model to develop yours.

• “Peel the onion” - start at the innermost query and work out.• When you understand each result set, the next layer will make sense.

28

28

Front End - . . . here is our chart querySELECT <- final select

J2.REPORT_LEVEL ,J2.REPORT_TITLE ,J2.REPORT_CATEGORY,SUM(T2.X01) AS X01, . . . (51 lines not shown) . . .,SUM(T2.X53) AS X53

FROM(SELECT <- middle select

WORKLOAD_CLASS,SUM(CASE WHEN PERIOD = -52 THEN X ELSE 0 END) AS X01, . . . (51 lines not shown) . . .,SUM(CASE WHEN PERIOD = -00 THEN X ELSE 0 END) AS X53

FROM(SELECT <- inner select

( DAYS ( TIMEPERIOD_WK ) -DAYS ( CURRENT_DATE - DAYOFWEEK(CURRENT_DATE) DAYS - 6 DAYS ) )/ 7

AS PERIOD,WORKLOAD_CLASS AS WORKLOAD_CLASS,SUM(CLASS1_CPU_TOTAL) AS X

FROM ACTG_SUMM_DLYWHERE TIMEPERIOD_DT BETWEEN

DATE(DAYS(CURRENT_DATE)-DAYOFWEEK(CURRENT_DATE)-370)AND

DATE(DAYS(CURRENT_DATE)-DAYOFWEEK(CURRENT_DATE) )GROUP BY TIMEPERIOD_WK, WORKLOAD_CLASS) AS T1GROUP BY WORKLOAD_CLASS) AS T2INNER JOIN ACTG_CTRL J2ON T2.WORKLOAD_CLASS = J2.WORKLOAD_CLASSGROUP BY J2.REPORT_LEVEL, J2.REPORT_TITLE, J2.REPORT_CATEGORY

• The inner SELECT is a GROUP BY rollup that generates 53 rows of data for each WORKLOAD_CLASS, sequenced by PERIOD. In this case “X” (the value to be charted) is the CLASS1_CPU_TOTAL, but there are hundreds of accounting elements than can be charted.

• The middle SELECT pivots the 53 data values for each WORKLOAD_CLASS into a single row containing a time series in data elements X01 through X53. So, at this point, you have one row for each WORKLOAD_CLASS, and you can use this result set to feed the next step, the workbook application that makes charts.

• The final SELECT is optional. Add this layer of processing when you have built a control table and you are ready to generate a nested hierarchy of charts.

• The BETWEEN predicate is very important because it automatically “rolls” the chart window forward.

I will be changing this to a RIGHT OUTER JOIN.

29

29

Front End - … how the “pivot” works

SELECTWORKLOAD_CLASS,SUM(CASE WHEN PERIOD = -4 THEN VALUE ELSE 0 END) AS X1SUM(CASE WHEN PERIOD = -3 THEN VALUE ELSE 0 END) AS X2SUM(CASE WHEN PERIOD = -2 THEN VALUE ELSE 0 END) AS X3SUM(CASE WHEN PERIOD = -1 THEN VALUE ELSE 0 END) AS X4SUM(CASE WHEN PERIOD = 0 THEN VALUE ELSE 0 END) AS X5

FROM DATATABLEGROUP BY WORKLOAD_CLASS ;

WORKLOAD_CLASS X1 X2 X3 X4 X5Class 1 24.27 0 0 0 0Class 1 0 23.44 0 0 0Class 1 0 0 23.03 0 0Class 1 0 0 0 21.29 0Class 1 0 0 0 0 23.88Class 2 43.24 0 0 0 0Class 2 0 46.07 0 0 0Class 2 0 0 55.98 0 0Class 2 0 0 0 48.20 0Class 2 0 0 0 0 67.11

WORKLOAD_CLASS X1 X2 X3 X4 X5Class 1 24.27 23.44 23.03 21.29 23.88Class 2 43.24 46.07 55.98 48.20 67.11

WORKLOAD_CLASS PERIOD VALUEClass 1 -4 24.27Class 1 -3 23.44Class 1 -2 23.03Class 1 -1 21.29Class 1 0 23.88Class 2 -4 43.24Class 2 -3 46.07Class 2 -2 55.98Class 2 -1 48.20Class 2 0 67.11

When we roll up our ACT_SUMM_DLY table by PERIOD and WORKLOAD_CLASS, we get a result set that looks like the table on the left; for each WORKLOAD_CLASS there is a set of rows that define a time series. But to plot many WORKLOAD_CLASSestogether on a “stacked line” chart, we need to “pivot” n rows into a single row with n data values. Here is how it works. (This example has 5 time periods, our system has 53.)

30

30

Front End - . . . the time period query

• We need one more query, to produce the time period labels that go on the “category axis” at the bottom of each chart.• The first three columns are dummies to align the result set of the time

period query with the result set of the chart query. REPORT_LEVEL and REPORT_TITLE come from the control table join.

SELECT'REPORT_LEVEL'

,'REPORT_TITLE','REPORT_CATEGORY'

,DATE(DAYS(CURRENT_DATE)-DAYOFWEEK(CURRENT_DATE)-370) AS WEEK01, . . . (51 lines not shown) . . .,DATE(DAYS(CURRENT_DATE)-DAYOFWEEK(CURRENT_DATE)- 6) AS WEEK53

FROM SYSIBM.SYSDUMMY1

31

31

Front End - Develop Chart QueriesLook at the chart query and report query results side by side; they are designed to integrate with each other. For this example, there is a control table join; that’s where REPORT_LEVEL, REPORT_TITLE and REPORT_CATEGORY come from. Without the control table join, the result set would have WORKLOAD_CLASS, followed by the time series data points (X01 through X53). From here, the next step is to build a workbook application to run these queries, organize the results, and generate the charts.

Time Period Queryresults

Chart Queryresults

Elements from control table join

report_level report_title report_category X01 . . . X53L1 System Workoad Business Unit 1 xxx.xx . . . xxx.xxL1 System Workoad Business Unit 2 xxx.xx . . . xxx.xxL1 System Workoad Corporate Systems xxx.xx . . . xxx.xxL2A1 Business Unit 1 Receivables xxx.xx . . . xxx.xxL2A2 Business Unit 1 Marketing xxx.xx . . . xxx.xxL2B1 Business Unit 2 CRM xxx.xx . . . xxx.xxL2B2 Business Unit 2 Supply Mgt xxx.xx . . . xxx.xxL2C1 Corporate Systems General Ledger xxx.xx . . . xxx.xxL2C2 Corporate Systems Payroll xxx.xx . . . xxx.xxL3A1 BU1 - Receivables Batch xxx.xx . . . xxx.xxL3A2 BU1 - Receivables Online xxx.xx . . . xxx.xxL3A3 BU1 - Receivables Maintenance xxx.xx . . . xxx.xxL3B1 BU1 - Marketing Batch xxx.xx . . . xxx.xxL3B2 BU1 - Marketing Online xxx.xx . . . xxx.xxL3B3 BU1 - Marketing Maintenance xxx.xx . . . xxx.xx(and 4 other Level 3 reports)

1 2 3 WEEK01 . . . WEEK53REPORT_LEVEL REPORT_TITLE REPORT_CATEGORY mm/dd/yyyy . . . mm/dd/yyyy

32

32

Front End - Build a Control Table• The control table supports hierarchical “drill down” charts.

• Our workload view has 60 workload classes. But a chart with too many lines (like more than ten) is incomprehensible.

• That is why we produce hierarchical “drill down” charts, with aggregated workloads at the upper levels.

• Our first prototype system did these aggregations in the workbook, but that was difficult to manage.

• The control table shifts the aggregation work back to DB2.• Start without a control table; you will know when you need it.

• The control table:• Specifies how many charts to produce and at how many levels.• Drives aggregation of discrete workload classes into larger classes.• Specifies chart names and workload time series labels.

33

33

Front End - Build a Control Table

In this presentation, the control table is named ACTG_CTRL.Column Name Type Description--------------- -------- ---------------------------------------------------------------------WORKLOAD_CLASS CHAR(30) This is the same WORKLOAD_CLASS that you define in the viewREPORT_LEVEL CHAR(6) Assigns workload classes to reports and controls aggregationREPORT_TITLE CHAR(20) Each report has a title that appears at the topREPORT_CATEGORY CHAR(20) Each time series has a title, that appears in the legend to the right

Don’t use special characters (forward slash, back slash, pipes, etc.) in the REPORT_TITLE. Each report has it’s own worksheet (that’s a tab) in the workbook application, and you will want to use REPORT_TITLE for the worksheet name on the tab. Some spreadsheets do not allow these characters for worksheet names. You probably want to be careful with REPORT_CATEGORY also. Dashes are OK, I use them for my titles and categories in Excel.

34

34

Front End - Build a Control Table

Level 1 has one chart Level 2 has three charts Level 3 has six charts

This represents the entries in the control table in three chunks. Actually, each of the 18 workload classes in this chart has three rows in the table (L1, L2 and L3), for 54 rows all together. It does not matter how you sort them. Here is how the drill down works for the workloads highlighted in yellow:

• The Level 1 chart is titled “System Workload” and it charts “Business Unit 1”, “Business Unit 2” and “Corporate Systems”. Everything that runs in the whole system is accounted for in this chart.

• There is a Level 2 chart titled “Business Unit 1” and it charts “Receivables” and “Marketing”.

• There is a Level 3 chart titled “BU1 - Receivables” and it charts “Batch”, “Online” and “Maintenance”.

workload_class level report_title report_category level report_title report_category level report_title report_category

RCV Sys - Batch L1 System Workoad Business Unit 1 L2A1 Business Unit 1 Receivables L3A1 BU1 - Receivables BatchRCV Sys - Online L1 System Workoad Business Unit 1 L2A1 Business Unit 1 Receivables L3A2 BU1 - Receivables OnlineRCV Sys - Mtce L1 System Workoad Business Unit 1 L2A1 Business Unit 1 Receivables L3A3 BU1 - Receivables MaintenanceMKT Sys - Batch L1 System Workoad Business Unit 1 L2A2 Business Unit 1 Marketing L3B1 BU1 - Marketing BatchMKT Sys - Online L1 System Workoad Business Unit 1 L2A2 Business Unit 1 Marketing L3B2 BU1 - Marketing OnlineMKT Sys - Mtce L1 System Workoad Business Unit 1 L2A2 Business Unit 1 Marketing L3B3 BU1 - Marketing MaintenanceCRM Sys - Batch L1 System Workoad Business Unit 2 L2B1 Business Unit 2 CRM L3C1 BU2 - CRM BatchCRM Sys - Online L1 System Workoad Business Unit 2 L2B1 Business Unit 2 CRM L3C2 BU2 - CRM OnlineCRM Sys - Mtce L1 System Workoad Business Unit 2 L2B1 Business Unit 2 CRM L3C3 BU2 - CRM MaintenanceSUP Sys - Batch L1 System Workoad Business Unit 2 L2B2 Business Unit 2 Supply Mgt L3D1 BU2 - Supply Mgt BatchSUP Sys - Online L1 System Workoad Business Unit 2 L2B2 Business Unit 2 Supply Mgt L3D2 BU2 - Supply Mgt OnlineSUP Sys - Mtce L1 System Workoad Business Unit 2 L2B2 Business Unit 2 Supply Mgt L3D3 BU2 - Supply Mgt MaintenanceGLS Sys - Batch L1 System Workoad Corporate Systems L2C1 Corporate Systems General Ledger L3E1 Corp - GL BatchGLS Sys - Online L1 System Workoad Corporate Systems L2C1 Corporate Systems General Ledger L3E2 Corp - GL OnlineGLS Sys - Mtce L1 System Workoad Corporate Systems L2C1 Corporate Systems General Ledger L3E3 Corp - GL MaintenancePAY Sys - Batch L1 System Workoad Corporate Systems L2C2 Corporate Systems Payroll L3F1 Corp - Payroll BatchPAY Sys - Online L1 System Workoad Corporate Systems L2C2 Corporate Systems Payroll L3F2 Corp - Payroll OnlinePAY Sys - Mtce L1 System Workoad Corporate Systems L2C2 Corporate Systems Payroll L3F3 Corp - Payroll Maintenance

35

35

Front End - Build a Workbook Application• The first three worksheets run queries and organize the data.

• 1 - “Query - Master” runs the chart query.• 2 - “Query - Time Periods” runs the time period query.

• (Appendices 3 and 4 explain how to embed and edit queries in Excel)• 3 - “Chart Grids” - merges the query results into “chart grids”

• An array of data organized to build a chart with the least possible effort.• Pull the raw time series data from “Query - Master”, break it up into

sections, and merge time interval headings from “Query - Time Periods”.• Pull the first cell (A1) from the A2 cell of “Query - Time Periods” (we do

not want the header line from “Query - Master”).• Fill down and right to pick up all of the data from “Query - Master”.• Now you have some shuffling to prepare the grid for each chart.

• (Appendix 5 has detailed instructions for doing this in Excel)

36

36

Front End - Build a Workbook Application

Here is the chart grid forthe “System Workload” chart.

(the “level 1” chart)

Here is the chart grid forthe “Business Unit 1” chart.

(the first “level 2”chart)

report_level report_title report_category X01 . . . X53L1 System Workoad Business Unit 1 xxx.xx . . . xxx.xxL1 System Workoad Business Unit 2 xxx.xx . . . xxx.xxL1 System Workoad Corporate Systems xxx.xx . . . xxx.xxL2A1 Business Unit 1 Receivables xxx.xx . . . xxx.xxL2A2 Business Unit 1 Marketing xxx.xx . . . xxx.xx(and other reports)

“Query - Master”, showing the first two report blocks:

1 2 3 WEEK01 . . . WEEK03REPORT_LEVEL REPORT_TITLE REPORT_CATEGORY mm/dd/yyyy . . . mm/dd/yyyy

“Query - Time Periods” worksheet:

REPORT_LEVEL REPORT_TITLE REPORT_CATEGORY mm/dd/yyyy . . . mm/dd/yyyyL1 System Workoad Business Unit 1 xxx.xx . . . xxx.xxL1 System Workoad Business Unit 2 xxx.xx . . . xxx.xxL1 System Workoad Corporate Systems xxx.xx . . . xxx.xx

REPORT_LEVEL REPORT_TITLE REPORT_CATEGORY mm/dd/yyyy . . . mm/dd/yyyyL2A1 Business Unit 1 Receivables xxx.xx . . . xxx.xxL2A2 Business Unit 1 Marketing xxx.xx . . . xxx.xx

(and other chart grids)

“Chart Grids” worksheet:

37

37

Front End - Build a Workbook Application

• All of the charts are built from the “Chart Grid” worksheet.• We build the chart grids one at a time.

• Our system has 14 charts; the “Chart Grid” worksheet is deliberately organized to make this as easy as possible.

• In the worksheet, select a data range and insert a chart.• The software should pick up the series headings and time labels.• You will have to specify the chart titles.

• “Stacked line graphs” represent a workload as a sum of discrete components. You can adjust the stacking order.

• We usually put the most stable components at the bottom of the stack• Detailed instructions for the building a chart in Excel are in Appendix

7.

38

38

(the chart grids are separated by empty rows)REPORT_LEVEL REPORT_TITLE REPORT_CATEGORY 02/15/04 02/22/04 (49 02/06/05 02/13/05L2A 01 Business Unit 1 Shadow Tables 49169.63 42056.59 cols 46866.09 45189.43L2A 02 Business Unit 1 ODS 43967.09 47288.44 not 21685.25 20992.75L2A 03 Business Unit 1 Data Warehouse 25139.81 25698.19 shown) 14966.45 14372.58L2A 04 Business Unit 1 End User SQL 139844.7 140857.1 22839.75 20367.44L2A 05 Business Unit 1 Support 57304.34 27871.22 824.36 5004.473L2A 06 Business Unit 1 Other 0 0 0 0(the chart grids are separated by empty rows)

Front End . . . chart grid and chart, side by side

Business Unit 1

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

02/15

/2004

02/29

/2004

03/14

/2004

03/28

/2004

04/11

/2004

04/25

/2004

05/09

/2004

05/23

/2004

06/06

/2004

06/20

/2004

07/04

/2004

07/18

/2004

08/01

/2004

08/15

/2004

08/29

/2004

09/12

/2004

09/26

/2004

10/10

/2004

10/24

/2004

11/07

/2004

11/21

/2004

12/05

/2004

12/19

/2004

01/02

/2005

01/16

/2005

01/30

/2005

02/13

/2005

Week Beginning On

DB

2 C

lass

1 C

PU S

econ

ds


The green box shows what part of the chart grid you select to build the chart.

The blue arrows show the three items that you have to input manually.1) “Business Unit 1” is the “Chart Title”.2) “DB2 Class 1 CPU Seconds” is the “Value Axis Title”.3) “Week Beginning On” is the “Category Axis Title”.(these are the terms that Excel uses)

The green arrows show what the the graphics application (Excel) generates automatically from the selected data.

Chart Grid

39

39

Operating and maintaining the system• The system presented here requires a weekly update.

• Two scheduled jobs update ACT_SUMM_DLY (summary table).• Roll up the latest data (7 days) from ACTG_GNRL.• Append rolled up data (LOAD RESUME) to ACTG_SUMM_DLY.

• In the workbook, rerun both queries to refresh the data (Appendix 6).• These queries should both “roll forward” in time.• The charts will update themselves.

• A workload change (like a new system or application) requires some end-to-end work. • Update the ACTG_GNRL_Vx view to identify the new workload.• Complete refresh (roll up all history) of ACTG_SUMM_DLY.• Update the ACTG_CTRL table to pick up the new workload.• Rebuild the “Chart Grid” worksheet and all of the charts.

40

40

Visualizing DB2 Performance for Your ApplicationsSession: A4

Robert H. SchummHBUS / US Technical [email protected]

41

41

ReferencesIBM MANUALS“DB2 UDB for OS/390 and z/OS V7 Administration Guide”, SC26-9931-00•5.2.3 A general approach to problem analysis in DB2 - Information on controlling accounting record generation for a CICS workload, and on correlating DB2 and CICS accounting records.•5.3.3.2.2 Accounting and statistics traces

“DB2 UDB for OS/390 and z/OS V7 Command Reference”, SC26-9934-00• 2.30 –DISPLAY TRACE• 2.71 –START TRACE

“DB2 Performance Expert, Performance Monitor, Report Reference”, SC27-1647-03• 4.2.6.1 Accounting Record Generation• 4.5.5.20 Identification • 7.0 The Performance Database

DB2 INSTALL LIBRARIES•SDSNSAMP Library Member DSNWMSGS - Technical descriptions of all DB2 trace data, including IFCID 003 / accounting record. (In V8 I understand that this member in in the DSNIVPD library).•SDSNMACS Library has “mapping macros” for all of the trace records.

ARTICLES I FOUND ON THE WEB (recently, in January / February 2005)“Interpreting DB2 accounting trace using COBOL”, Pranav Sampat, Mainframe Week online, February 12, 2003, www.mainframeweek.com/journals/articles/0057/574

"Finding Out Who Did It - Correlating Application and Database Processes in DB2 V8: Basis For Enhanced Accounting and Workload Management", Namik Hrle AND Johannes Schuetzner, IDUG Solutions Journal, Volume 11 Number 2 (August 2004), www.idug.org/idug/member/journal/aug04/article03.cfm

42

42

Appendix 1 - The DB2 Accounting Record• The “accounting record” records DB2 activity from an application point of view.• It is also known as the SMF 101 record.•Mapping macros (documenting the exact layout of the 101 record) are in SDSNMACS members DSNDQWA0, DSNDQWHS, DSNDQWAC, DSNDQLAC, and DSNDQXST.• The element by element details that the accounting records contains are not described in detail anywhere in the baseline DB2 libraries.• *****Identify and describe a few of the most important elements.*****• Run a "-DIS TRACE" command to verify that DB2 accounting trace (Classes 1, 2, and 3) is on, with the destination set to SMF. If "PLAN", "AUTHID" or "CUTOFF-CPU-TIME" is specified (in the START TRACE "constraint block"), then the trace is not capturing every thread.• Your system configuration determines when accounting records are produced.• For the system that we are studying (running an OLAP / DSS workload, with minimal CICS transactional work), it is simple. The DB2 trace begins collecting data when a thread allocates / start, and writes a completed accounting record to SMF when it terminates.• For CICS, when threads are reused, it's not so simple. Typically, systems supporting high volume transactional workloads are configured / tuned to reuse threads, so accounting records from reused threads will contain aggregated statistics for any number of transactions.• DB2 V8 promises new capabilities to aggregate / dis-aggregate accounting data in order to manage data volume and to provide more a more meaningful representation of the application workload.• The growing diversity of DB2 workloads (i.e. z/OS DB2 as a distributed server) requires additional elements to tie threads back to applications; so DB2 V8 has promised additional identifying elements.

43

43

Appendix 2 - Summary Table ValidationThese are the queries we use to check the data in the ACTG_SUMM_DLY table, to make sure that all of the data to produce the charts has been correctly loaded.

-- How many rows were loaded each day?

SELECT TIMEPERIOD_DT

,COUNT(*) AS RECORD_COUNT ,SUM(CLASS1_CPU_TOTAL) AS TOTAL_CPU

FROM ACTG_SUMM_DLY GROUP BY TIMEPERIOD_DT ;

-- Has a complete set of data been loaded for each week?

SELECT TIMEPERIOD_WK

,COUNT(*) AS RECORD_COUNT ,SUM(CLASS1_CPU_TOTAL) AS TOTAL_CPU ,MIN(TIMEPERIOD_DT) AS START_WEEK, MAX(TIMEPERIOD_DT) AS END_WEEK,COUNT(DISTINCT(TIMEPERIOD_DT)) AS DAY_COUNT

FROM ACTG_SUMM_DLY GROUP BY TIMEPERIOD_WK ;

44

44

Appendix 3 - Embedding SQL in Excel• For a query developed on the mainframe, download it into a plain text file. "--" to indicate comments is acceptable. The end-of-query marker (semicolon) is not necessary.• Start with a clean worksheet.• Give it a name that identifies the query.• Pull down "Data" / "Get External Data" / "Create New Query".• "Choose Data Source" - select a DB2 subsytem to get a signon screen where you enter your userid and password.• Now you are at a Microsoft Query "Wizard" window that says "Add Tables"; close that window.• Now you are at a Microsoft Query window, click the "SQL" button.• Paste in the query and click the "OK" box.• Click the "OK" box when it says "SQL Query can't be represented …".• Wait for the query to execute.• Now you are at a Microsoft Query window displaying the result set.• Pull down "File" / "Return Data to Microsoft Excel".• The prompt says ""Where do you want to put the data?", select "Existing Worksheet", click "OK".• Wait for the data to load.• Now you have the result set in your worksheet, and the query is saved in the worksheet so that you can edit it or run it again later.

•Connectivity to the DB2 server (z/OS, LUW) is required. Usually this is an install of an ODBC (Open DataBase Connectivity) driver on the desktop.

45

45

Appendix 4 - Editing a Query in Excel

• Editing a query in Excel is difficult because Excel wants to execute the query before you run it. This means that you:• Must have access to your DB2 system (i.e. signed on to network).• Have to wait for it to run every time you want to edit a query.

• Pull down "Data" / "Get External Data" / "Edit Query".• Now you have to wait, whether you want to or not, the query will run.• Now you are at a Microsoft Query window; click the "SQL" button.• Update the query and click the "OK" box.• Click the "OK" box when it says "SQL Query can't be represented …".• Wait for the query to execute.• Now you are at a Microsoft Query window displaying the result set.• Pull down "File" / "Return Data to Microsoft Excel".• The prompt says ""Where do you want to put the data?", select "Existing Worksheet", click "OK".• Wait for the data to load.

46

46

Appendix 5 - Creating the “Chart Grids” in Excel

• Open a new worksheet and give it a name. We use “Chart Grids”.• Link the A1 cell on “Chart Grids” to the A1 cell on “Query – Master”.• Fill right and down to capture all of the data from “Query – Master”.• Erase the first row, leaving it blank. This row contained column titles from the master query.• Link the A1 cell on “Chart Grids” to the A2 cell from “Query – Time Periods”.• Edit the formula in this cell as follows:• ='Query - Time Periods'!A2• changes to• ='Query - Time Periods'!A$2• Fill the first row to the right. Reset date format if necessary.• If you are using the control table to drive multiple reports, the “chart breaks” will be wherever the “REPORT_TITLE”changes. Insert two blank lines between each break. The data should shift down but still “lock” onto the correct numbers from “Query – Master”.• Now that you have two blank lines for each chart break, you can copy the first row (with the dates that label the x-axis of each chart) to the head of each chart. So now you should have a “chart grid” for each chart; with each chart grid separated by a blank line.

47

47

Appendix 6 - Running a Query in Excel

•Go to the worksheet with the query that you want to execute.•Pull down "Data" / "Refresh Data".•If you have not connected to DB2, you will get an error, but it will take you to a panel that will connect you.•Wait for the query to execute and load the data into the worksheet (the "spinning world" indicates query is running).

•For the application described here, you will have to do this twice, once for “Query - Master” and then for “Query - Time Periods”.•If you have written queries that reference the current data to “roll” the time window forward, you should be able to run both queries each cycle, without editing them.

48

48

Appendix 7 - Building a Chart in Excel•If you are building more than one chart (our prototype system builds 14 charts), it will be convenient to open up a plain text “note pad” window (I use the Windows Notepad accessory). Use this to save the “Chart title” (different for every chart), the “Category (X) axis title” and the “Value (Y) axis title” (both of which are the same for every chart).• Under the heading “REPORT_TITLE”, there is the report title as a text element. Clip and save it in your note pad.• Cursor to the heading “REPORT_CATEGORY”, click it, and then hold and drag down and right until you have selected the whole grid for that report. Do not go across the break to the next report.• Select “Insert” – “Chart”.• From “Standard Types”, select “Area” for “Chart type” and “Stacked Area” for “Chart sub-type”.• “Next” takes you to “Chart Wizard – Step 2 of 4 – Chart Source Data”. You should see a miniature of a presentable chart.• “Next” takes you to “Chart Wizard – Step 3 of 4 – Chart Options”

• “Titles” should already be on top.• Fill in “Chart tile”, “Category (X) axis”, and “Value (Y) axis” title. The X-axis is your time period; for a weekly chart, I use “Week Beginning On”. The Y-axis is whatever performance attribute you rolled up from the accounting table; in this example, it is “Estimated Class 1 CPU”.• If you are using the “note pad” to keep the chart title and the axis titles, this is simply a matter of cutting and pasting.

• “Next” takes you to “Chart Wizard – Step 4 of 4 – Chart Location”.• Select “As new sheet”.• Paste the chart title saved in the clipboard into the box on the right, this is the name of the new worksheet with the new chart.

• “Finish” takes you to completed chart.

Documents

Platform: z/OS Visualizing DB2 Performance for Your