DB-5: Simulating User Load Tom Bascom VP Technology White Star Software [email protected]

DB-5: Simulating User LoadTom BascomVP TechnologyWhite Star [email protected]

DB-5: Simulating User Load - 2

Sanity Check!

This Presentation is NOT about simulating your user interface.

We are going to discuss a method for creating a database load that mimics a target usage profile.


The Challenge

A Volatile Business Environment System Load expands to consume all

resources… Something is Always changing…

Fighting Misconceptions:– Load testing is too difficult.

– Load simulation tools are expensive.

– Load testing tools don’t support Progress.


Why Simulate User Load?

Regression Testing Finding the Limits / Capacity Planning Root Cause Analysis Minimizing Cost / Optimizing Resources Quality of Service (QoS) Assurances To support other testing efforts more

realistically


Considerations of Fidelity

Assumptions Characterizing the Load Where are the Bottlenecks? Users are fickle… How do we (realistically) drive test scenarios?

(Or: Are we really simulating user load?)


Assumptions

We are primarily interested in the DATABASE. READ activity is, by far, the largest consumer

of database resources. WRITE activity is also important. An ABL session is an ABL session… (in terms

of resource use)

The functionality of the application is not being tested (by these tests).


Characterizing the Load

Historical data collection is a must. Application Metrics

– Orders Taken– Widgets Produced

Baseline db Performance– Logical IO ops/sec– Disk IO ops/sec– Latch & Resource Waits– TRX, Record Locks & Blocked Clients

IO by User TableStats


Where are the Bottlenecks? Database

– Latch contention

– Transaction throughput Operating System

– Disk

– CPU

– Memory

– Network


Users are fickle…

They work on different things… at different rates … pretty much randomly … and sometimes at their leisure ;-)

Our best friends are:– IO by User (PROMON, ProTop)

– _TableStat (VST, ProTop)


IO Requests by User

0

50

100

150

200

250

300

350

0 10 20 30 40 50 60 70


IO Requests by Table

0

1000

2000

3000

4000

5000

Reco

rds/s

ec

Ord

er-

Lin

e

Ord

er

Invoic

e

Custo

mer

Item

Sta

te

Ref-

Call

Local-D

efa

ult

Sale

sR

ep

Table


IO Requests by Table

Create Read Update Delete

5,940,439 1,662,688,907 2,540,165 2,500,292

0.35% 99.34% 0.15% 0.15%

83.8/s 23,454.5/s 35.8/s 35.3/s

Table Name Created % Read % Updated % Deleted %

CHART_ENTRY 0 0.0 1,061,326,658 63.8 0 0.0 0 0.0

BUREAU_QUEUE 0 0.0 241,087,053 14.5 25,150 0.9 0 0.0

APP_INFO 4,711 0.0 82,913,516 4.9 213,944 8.4 26 0.0

BTE_QUEUE 0 0.0 27,612,013 1.6 14,312 0.5 0 0.0

SCORE_EXPWGHT 0 0.0 20,349,269 1.2 0 0.0 0 0.0

APP_REQUEST 4,711 0.0 13,491,724 0.8 45,119 1.7 668 0.0

APP_TRADE 105,260 1.7 11,499,880 0.6 143,465 5.6 20,672 0.8

APP_OPTION 23,791 0.4 11,341,471 0.6 50,809 2.0 9,614 0.3

APP_TRANSTAGG 3,347,503 56.3 8,603,941 0.5 0 0.0 1,919,357 76.7


Data Distribution

0

250

500

750

1000R

eco

rds

Ord

er-L

ine

Ord

er

Invo

ice

Cus

tom

er

Item

Sta

te

Ref

-Cal

l

Loca

l-Def

ault

Sal

esR

epTable


Idle Users?

Current Users: 192 Idle Users: 0:01 18 0:02 8 0:03 10 0:04 4 0:05 4 0:06 2 0:07 4 0:08 4 0:09 1 10m - 1hr: 51 Hour+ old: 32


How do we drive test scenarios?

With some ABL code that reads and writes records as the application does.

Distributed among many sessions. Profiled according to TableStats.


Code !!!


WARNING

Using the CREATE, UPDATE or DELETE routines will modify your database!

Use these destructive options only with test copies of your database.

Use the READ-ONLY option with production databases!


A Simulation Controller

You already have everything that you need… A generic Load driver Tuning the driver to your Load Controlling test runs


Control Program (pacectl.p)

cmd = “mbpro &1 -pf pace.pf -param &2 -U &3".

input from value( "io.usr" ).

do while true:

import io_usr nm_usr.

xcmd = substitute( cmd, pdbname(1), io_usr, nm_usr ).

message xcmd.

if opsys = "unix" then

os-command silent value( xcmd ).

else

run spawn( input xcmd, input "", output pid ).

pause 1.

end.


Config File (io.usr)

0,500,50,0 jami

0,100,10,0 tucker

0,50,5,0 jami

0,40,4,0 julia

0,30,3,0 peter

0,20,2,0 emily


pace.p

do while true:

x = random( 1, pickLimit ).

find first tt_pace where tblPick >= x.

run xread( tblName, r_rate ).

t = time - s.

do while (( i / t ) > r ):

pause 1 no-message.

t = time - s.

end.

end.


xread()

create buffer bh for table tblName.qh:set-buffers( bh ).qh:query-prepare( “for each “ + tblName ). j = random( j / 2, j * 10 ).do while k < j: qh:get-next( no-lock ) no-error. if qh:query-off-end then do: qh:get-first( no-lock ) no-error. next. end. k = k + 1.end.


Demo !!!


Selected Case Studies

Benchmarking OpenEdge 10.0– PSDN Whitepaper:

http://www.psdn.com/library/entry!default.jspa?categoryID=256&externalID=876

Ensuring Disk Array Performance Verifying a Failover Plan Validating Parameter Changes Providing a background load for development

or testing environments. Supporting Demos ;-)


Benchmarking OE10

PSDN Whitepaper:

http://www.psdn.com/library/entry!default.jspa?categoryID=256&externalID=876

Problem:– How Does OE10 Compare to Progress 9?


Ensuring Disk Array Performance

Problem:– Will a new disk array configuration support

the production workload? Several previous attempts to go live had

failed. Load simulation was used to prevent a

repeat of that fiasco. Problems were found and solutions were

verified and confidence was repaired prior to a successful implementation.


Verifying a Failover Plan

Problem:– Redundant (and expensive) systems were

purchased and configured for mutual failover. A method to realistically test a failover with

significant load on the hardware was needed.

The failover worked very well! But the “fail back” failed miserably As a result a serious OS bug was identified

and fixed and downtime avoided.


Validating Parameter Changes

Problem:– Changing many Progress and OS parameters

can result in unanticipated limits being reached.

# of file handles, semaphores, memory size etc…

Solution:– Use the load simulator to start sessions and

stress these limits.


Background Load

Problem:– Developers and testers are often working on

over-powered servers with no competition for resources.

– Which leads to a false sense of good performance when testing.

Solution:– Run load simulator to provide a continuous

background load.


Demos!

Problem:– Single user demos are boring.

Solution:– Create some fake users that do interesting

things.


Summary

Basic Load Simulation is easier than you think!

And doesn’t necessarily require fancy (and expensive) tools.

It does require fore-thought and planning. And can be immediately useful to you!


Questions

http://www.greenfieldtech.com/downloads.shtml

Documents

DB-5: Simulating User Load Tom Bascom VP Technology White Star Software [email protected]