Non-Relational Efficiencies

Copyright 2007, Information Builders. Slide 1

Non-Relational Efficiencies

Walter F. BloodTechnical Director

Information Builders, Inc

Non-Relational EfficienciesRelational Efficiencies

Why is Relational Efficiency important?Code optimization for the relational engine Translation of

JOINSAggregation and sortingRecord selectionVirtual fields

Target – reduce answer set returned to minimum to get fastest returnMaking maximum use of the relational engineLimiting data traffic to a minimum

.

Non-Relational EfficienciesNon-Relational Efficiencies

Non-Relational Efficiency has similar target

Target – get answer set in shortest period of timeMake optimum use of WebFOCUS engine functionality

What is Non-relational?How do you recognize efficiency?

.

Non-Relational EfficienciesNon-Relational Efficiencies

Non-relational databases – FOCUS, flatfiles,…Non SQL

Intermediate files created during the development of a report

Combinations of Non-relational datafiles and relational tables

Creating non-relational output – PDF, EXCEL, comma delimited

.

When Are We Non-Relational?

Efficiency ?

Non-Relational EfficienciesRecognizing Efficiency

Efficiency ?

.

RelativeBaseline

IncrementalVary and Test

MeasureSystem

CPUIOs

WebFOCUSRecordsLines

CPU?


From operating system – process based

ps -u username

time programname

CPU?


From WebFOCUS – fine tuning control

-SET &TIME1 = &FOCCPU;

WebFOCUS processing -SET &TIME2 = &FOCCPU;-SET &CPUTIME=&TIME2 - &TIME1;

I/O?


From operating system – process based-device oriented

iostat

I/O?



-TYPE &READS-TYPE &WRITES-IF &LINES GT 0 THEN GOTO CONTINUE;-IF &RECORDS EQ 0 THEN GOTO EXIT;

ELAPSED TIME ?


From operating system – process based

time programname

ELAPSED TIME?



-SET &START = HHMMSS(‘A8’);

DEFINE FILE ABCTIME1/A8 WITH FLD=HHMMSS(TIME1);

Non-Relational EfficienciesWeb-FOCUS Functionality

… in Structures… in Connections … in Expressions… in Selections… in Sorts

Where in Web-FOCUS?


Non-Relational Efficiencies…In Structures

Metadata – Master Files - MFDs

Trim out unused fields Re-Describe data

Index, index, index McGyver

Non-Relational Efficiencies…in Structures

Trim out unused fieldsLess parsingSmaller data buffers

-------------------------------1. Remove from master2. Replace with FILLER in MFD


&TEST = 128.9

Re-Describe data----------------------

Identify repetitive fields in master file


&TEST = 128.9

Re-Describe data----------------------

Combine repetitive fields with OCCURS clause


Re-Describe data----------------------

Combine repetitive fields with OCCURS clause


Re-Describe data-----------------------Look for data that has rectype behavior•Characterizing data•Common position


1. For JOINing – add Internal index on HOLD2. External index – for static data3. Multi-dimensional index – REBUILD 4. Create dimension file to act as index - JOIN

Index, index, index


For homework:

FOCALPOINT or techsupport

McGyver


Non-Relational Efficiencies…In Connections

Two Modes of Connecting

Nested Loop - JOIN

Sort and Merge - MATCH



Nested Loop

A B

C



Sort Merge

A B

C



Equality based JOINMost direct connectionMultiple fields

Conditional JOIN Less efficientProvides additional functionalityExpression controls connection/efficiency

JOIN Efficiency Considerations


Non-Relational Efficiencies…JOINs


Non-Relational Efficiencies…JOINs


Non-Relational EfficienciesConnect Based on Range of Values

• Connection can also be based upon range - • LE, LT, GE, GT, NE or FROM /TO

EMPLOYEE IDLAST NAMEFIRST NAMEHIRE DATECURRENT SALARY…

EMPLOYEETAX YEARTAX RATEMINIMUM SALARYMAXIMUM SALARY

TAX RATE

JOIN FILE file AT field TO ALL FILE file AT field AS nameWHERE condition


Non-Relational EfficienciesConnect Based on Range of Values


Non-Relational EfficienciesConnect Based on Expression

Connect files that have no apparent connection

Employee Salary History

New CarFinance

Packages

JOIN?




New CarFinance

Packages

JOIN?




New CarFinance

Packages

JOIN?


OLD-OR-NEW – all records from first and second files. This is the default if the AFTER MATCH line is omitted. (The UNION of the sets.)

OLD-AND-NEW – only records common to both files. (The INTERSECTION of the sets.)

OLD-NOT-NEW – records from the first file with no match in the second file.

MATCH Efficiency – variety of outputs

Non-Relational EfficienciesMATCH Output


NEW-NOT-OLD - records from second file with no match in the first file.

OLD-NOR-NEW - non-matching records from both files -records from the first file with no match in the second file, and records from the second file with no match in the first file.

OLD – records from the first file with matching records in the second file.

NEW – records from the second file with matching records in the first file.

Non-Relational EfficienciesMATCH Output



1. What types of files? How large?2. Where are the fields you are selecting on?3. Is the connection equality or conditional ?4. If conditional how complex is the expression ?5. Where are short paths expected?

Host File Xref File

JOIN orMATCH


Non-Relational Efficiencies…In Expressions

DATAIN

MATRIX DATAIN

DEFINEIFWHEREBY

COMPUTEIF TOTALWHERE TOTALBY TOTAL



For virtual fields, timing is keyDEFINE

On initial read of data Maximum data volume

COMPUTEOn exit from matrixEqual or smaller volume

Efficiency – evaluate on smaller volume



For expressions, not all subroutines are equalReferencing other data

DECODE functionDBLOOKUP subroutine-READFILE dialogue manager commandJOIN with LOOKUP/FIND

Changing dataCTRAN – 1 characterSTRREP – string of charactersGETTOK/SUBSTR – positional change

Efficiency – check possibilities for optimum



Working with more than one record instance at a time

Writing to multiple files simultaneously



Working with more than one record instance at a time

Available in COMPUTE, DEFINE, WHERE

References the field value in the previous record

Works with real and virtual fields

LASTTHE FUNCTION


Non-Relational Efficiencies…In Expressions LAST

In COMPUTE



Same as DEFINE

Operates on matrixProcesses same or

fewer records than DEFINE

In COMPUTE



In WHERE

LAST Considerations: The order the data is processed is critical

You may need to presort data to a hold file

Not optimizable to relational databases

DEFINES work with WHERE

COMPUTES work with WHERE TOTAL

In DEFINE In COMPUTE


Writing to multiple files simultaneously

To create a log of specific data values readWhich customers placed orders?

To create of record of calculated valuesWill I go out of stock on any these orders?

To create additional outputWhat page number will this item be on?

PUTDDREC


PUTDDREC syntax: PUTDDREC(ddname, dd_len, record_string, record_len,

outfield)

ddname - ddname assigned by filedef to output filedd_len - length of the ddnamerecord_string - string of characters to write to file or field

containing that stringrecord_len- length of the string to be includedoutfield - return code

Output file must be filedef’dDEFINE/COMPUTE field format always I1

Handles open, write and close


FILEDEF PUTDD1 DISK putdd1.datTABLE FILE EMPLOYEEPRINT EMP_ID CURR_JOBCODE AS 'JOB' CURR_SALCOMPUTE SALA/A12 = EDIT(CURR_SAL); NOPRINTCOMPUTE EMP1/A50= LAST_NAME|FIRST_NAME|EMP_ID|CURR_JOBCODE|SALA;NOPRINTCOMPUTE OUT1/I1 = PUTDDREC('PUTDD1',6, EMP1, 50, OUT1);BY LAST_NAME BY FIRST_NAMEEND

PUTDDREC in Action


LAST_NAME FIRST_NAME EMP_ID JOB CURR_SAL OUT1--------- ---------- --------- --- -------- ----BANNING JOHN 119329144 A17 $29,700.00 0BLACKWOOD ROSEMARIE 326179357 B04 $21,780.00 0CROSS BARBARA 818692173 A17 $27,062.00 0GREENSPAN MARY 543729165 A07 $9,000.00 0IRVING JOAN 123764317 A15 $26,862.00 0JONES DIANE 117593129 B03 $18,480.00 0MCCOY JOHN 219984371 B02 $18,480.00 0MCKNIGHT ROGER 451123478 B02 $16,100.00 0ROMANS ANTHONY 126724188 B04 $21,120.00 0SMITH MARY 112847612 B14 $13,200.00 0 RICHARD 119265415 A01 $9,500.00 0STEVENS ALFRED 071382660 A07 $11,000.00 0

Report Created


BANNING JOHN 19329144A17000000029700BLACKWOOD ROSEMARIE 26179357B04000000021780CROSS BARBARA 18692173A17000000027062GREENSPAN MARY 43729165A07000000009000IRVING JOAN 23764317A15000000026862JONES DIANE 17593129B03000000018480MCCOY JOHN 19984371B02000000018480MCKNIGHT ROGER 51123478B02000000016100ROMANS ANTHONY 26724188B04000000021120SMITH MARY 12847612B14000000013200SMITH RICHARD 19265415A01000000009500STEVENS ALFRED 71382660A07000000011000

Sequential File Created


PUTDDREC Considerations:

Create fixed data file to fit a specific Master fileCreate comma-delimited data file for loading System and User &variables are available&MDYY, &FOCCPU, &FOCUSER, etc

Write control at any point or multiple pointsWrite to multiple PUTDDREC files in a request

Non-Relational Efficiencies…in Selections

WHERE vs IFWHERE more expensive than IFIF limited to comparison to values – match arrayWHERE allows “expression operator expression”WHERE DecomposedIFs removedWHEREs converted to DEFINE/IF combinationDEFINE expressions converted to polishes

Non-Relational Efficiencies…in Selections

Selection placement Within the request

Before matrixAfter matrix – TOTAL test

Within the dataIndexed or alternative views in TABLEType of comparison

Selection considerations


Non-Relational Efficiencies

Questions?

Documents

Non-Relational Efficiencies