Bea Internals

PeopleTools Application Server & BEA Tuxedo [email protected] Consultancy Ltd.

http://www.go-faster.co.uk

23 January 20022

BEA Tuxedo InternalsGo-Faster Consultancy Ltd.

Who am I?

Independent Consultant

Performance Tuning

PeopleSoft / Oracle

23 January 20023


Rules of Engagement

Interrupt me:— if you think I have got something wrong— if you have a question— if you can’t hear me

23 January 20024


BEA Tuxedo

Historical Overview

Tuxedo Internal Architecture

PeopleSoft 3-tier client implementation

How does PeopleSoft Internet Architecture (PIA) fit in

Configuration issues

Monitoring

23 January 20025


Brief Historical Background

In the beginning— monolithic

PresentationBusiness

Logic

Data Access (DBMS)

23 January 20026


Client Server

Client Server

LAN

Presentation Business Logic

Data Access (DBMS)

SQLRaw Data

Two-tier client-server

23 January 20027


3-Tier Client Server

GUI+

ApplicationLogic

Data Access (DBMS)

Business Logic

Client Server

LAN

Request

Response

“1st Generation”

23 January 20028


3-Tier Client Server

Business Logic Data Management

TUXEDO Servers

PresentationLogic

TUXEDO Clients

Presentation

DBMS 1...DBMS n

Database Servers

Service 1Service 2...Service nTUXEDO

MessagingSQL

“2nd Generation” - Tuxedo

23 January 20029


Where did Tuxedo come from?

Bell Labs - 1993

Transactions for

UniX

Extended for

Distributed

Operations

23 January 200210


What is Tuxedo?

It is the knots on the ends of the string!

23 January 200211


PresentationLogic

PanelLoad

PanelSave

FieldChange

etc...

PeopleSoft 2-Tier Client

23 January 200212



PresentationLogic

PanelLoad

PanelSave

FieldChange

etc...

Server SideClient Side

Tuxedo C

lient

Tuxedo S

erver

23 January 200213



DATABASE

Client Connects Directly to the Database— 1 database connection per window— inactivity timeout

23 January 200214



DATABASE

Client Connects Directly to the Database— 1 database connection per window— inactivity timeout

What happens in 3-Tier?

23 January 200215


PS/Tuxedo Internal Structure

DATABASE

BBL

BB

1st Application Server Process Started— Bulletin Board Liaison (BBL)— Reads PSTUXCFG configuration file

Shared Memory Segment— Bulletin Board (BB)

PSTUXCFG

23 January 200216



DATABASE

WSL

BBL

BB

Workstation Listener (WSL)— Listens for connection request from clients on a specified port— Client initially contacts the WSL

23 January 200217



WSL

BBL

BB

Workstation Listener (WSL)— Client needs to know where to find the WSL— Machine and port specified in Configuration Manager

23 January 200218



DATABASE

WSL WSH

BBL

BB

Workstation Handler (WSH)— handles traffic from client after connection— Client is assigned to a port on the WSH— WSH port range can be constrained to match a firewall— Additional WSH can be spawned if necessary— Compress Messages

23 January 200219



DATABASE

WSL WSH

BBL

BB

PSAUTHAUTHQ

PSAPPSRVAPPQ

PSQCKSRVQCKQ

PSSAMSRVSAMQ

PSQRYSRVQRYQ

PSAPISRVAPIQ

23 January 200220


How does it work?

DATABASE

WSL WSH

BBL

BB

PSAUTHAUTHQ

PSAPPSRVAPPQ

PSQCKSRVQCKQ

PSSAMSRVSAMQ

PSQRYSRVQRYQ

PSAPISRVAPIQWSHQ

23 January 200221


How does PIA work?

Java Client & PIA use Jolt— Java Class Library— Jolt Listener maps Java functions to Tuxedo Messages

23 January 200222


Jolt Internal Structure

DATABASE

WSL WSH

BBL

BB

PSAUTHAUTHQ

PSAPPSRVAPPQ

PSQCKSRVQCKQ

PSSAMSRVSAMQ

PSQRYSRVQRYQ

PSAPISRVAPIQ

JSL JSH JREPSRV

JREPOSITORY

JSHQ

23 January 200223


Java

JVM

New structure for PIA (Apache Webserver)

HTTPD ClientClient

JSL JSH

Servlet EngineServlet Container

Servlet

ServletThread

ServletThread

ServletThread

JSH

Cookie

Disk(static

objects) JSH

http 1.1address+cookie

(stateless)

Servlet thread

IS stateful

mod_jserv

Sessionid = “xy1k…vm8010

Port 8010

Webserver

Tuxedo Application

Server

23 January 200224


What is the difference between the web servers?

Apache— Separate Servlet Engine — Separate encryption module (Raven)

Weblogic— Contains an integral servlet container

23 January 200225


Progressive Connection Concentration

Typical picture

35,000 users

1000 concurrent connections— 1000 users active in the last n minutes— where n is the Java + JSH timeout— therefore need 1000 threads— but maybe only 4 JVMs

100 JSH— assumes 10 clients per JSH

10-20 PSAPPSRV processes— each PSAPPSRV connects to database

1 database

35000 users

1000 connections4 JVMs

1000 Java Threads

100 JSHs

10-20 PSAPPSRVs

1 database

23 January 200226


psappsrv.val

psappsrv.cfg

psappsrv.ubb

PSTUXCFG

psappsrv.ubx

psadmin(configuration

dialogue)

ubbgen(PS Utility)

tmloadcf(tuxedo

process)

BBL(domainstartup)

Configuration

23 January 200227


Configuration Files

psappsrv.cfg— contains configurable parameters

psappsrv.val— validation file for configurable values

psappsrv.ubx— configuration template for Tuxedo domain

psappsrv.ubb— configuration file for Tuxedo domain

PSTUXCFG — compiled configuration file— read by BBL at domain startup

23 January 200228


psappsrv.ubx -v- psappsrv.ubb

psappsrv.ubx & psappsrv.cfg— PeopleSoft file— make changes to these files— difficult to read

psappsrv.ubb— Tuxedo file— never make changes to this file— easier to read— this file is compiled by tmloadcf

to generate domain configuration file PSTUXCFG

— variables fully resolved

23 January 200229


psappsrv.ubb

Five (or Six) Sections— Resources

– resource limits -> BB sizing

— Machine– Resource limits. Environment

— (Network– Connection between Machines)

— Groups– Logical Split (BASE, APPSRV, JREPGRP, JSLGRP)

— Servers– Number of servers, Command line

— Services– Timeout, Load, Priority

23 January 200230


Application Server Configuration Issue

Move quick PSAPPSRV services into a second server (PSQCKSRV) (y/n)?

— <=PT7.58 this question is misleading– services are additionally advertised on PSQCKSRV as well as

PSAPPSRV– so they are advertised on more than one queue

— >=PT7.59 and PT8.1– services are de-advertised from PSAPPSRV– quick services not used in PIA, only IC* services

– so not a problem

23 January 200231


DATABASE

WSL WSH

BBL

BB

PSAPPSRVAPPQ

PSQCKSRVQCKQ

PeopleTools <=7.58

Quick Service

23 January 200232


Inside the Bulletin Board

PSAPPSRV

PSAPPSRV

PSAPPSRV

PSQCKSRV

PSQCKSRV

PSQCKSRV

PSAPPSRV

Without Load Balancing (default)

— Table of servers process where each service is advertised

— Read and populated as a stack— WSL looks for first free server

where request can be enqueued— This can cause problems

23 January 200233


PSAPPSRVAPPQ

PSQCKSRVQCKQ

Load Balancing

Load Balancing is about choosing the queue upon which to place the message

— by default all requests have the same load

5050 50 50

5050 50 505050 50 50

50

23 January 200234


Load Balancing Trace

Add -r to application server command line

— CLOPT="-r -e APPQ.stderr -A -- -C psappsrv.cfg -D GP81O81 -S PSAPPSRV”

— -r enables trace to file ‘stderr’ (not channel 2)— -e overrides the output filename

— process trace file with Tuxedo utility ‘txrpt’— obtain average service time

— make load proportional to average service time

23 January 200235


Load Balancing Trace

— Output from TXRPT

SVCNAME 18p-19p TOTALS

Num/Avg Num/Avg

--------------- -------- -------

SqlRequest 16/0.10 16/0.10

MgrClear 5/0.04 5/0.04

RamList 1/0.43 1/0.43

SamGetParmsSvc 1/0.50 1/0.50

--------------- ------- -------

TOTALS 23/0.12 23/0.12

23 January 200236


PSAPPSRVAPPQ

PSQCKSRVQCKQ

Load Balancing

load = execution time (seconds) x 100

7878 386 135

6363 3 766 30 3

50599

116

23 January 200237


Multiple Queues on Busy System

You can have many server processes on a single queue

BEA recommend no more that 10 server processes per queue

If you have queue contention it is likely to impede the WSH/JSH processes enqueuing requests. You are unlikely to observe queuing with PQ.

So, if you want more than 10 PSAPPSRV processes you should use two APPQ queues

In, which case you should to enable load balancing and you must calculate the load.

— Less significant with PIA because services different.

23 January 200238


DATABASE

WSL WSH

BBL

BB

PSAPPSRVAPPQ1

PSAPPSRVAPPQ2


23 January 200239



psadmin will not correctly calculate— MAXACCESSERS— MAXSERVICES

tmadmin->bbs— will report maximum number of services

– count of all services advertised on each server process

and if you have multiple work station listeners— MAXWSCLIENTS

23 January 200240


PSQCKSRVQCKQ

Service Priority

Priority is about queue jumping— servers de-queue the request— 9 out of 10 de-queued on the basis of priority— 1 out 10 is taken from the front of the queue

8080 5050 50 5050 50

23 January 200241


Service Priority

All PeopleSoft Services have same priority— specified in SERVICES section in psappsrv.ubb— therefore there is no queue jumping

— no reason to believe that it would be advantageous

23 January 200242


Tuning Tip

Financials— On-line journal edit & post - remote call— Service initiates a Cobol process— Process can take minutes— RemoteCall services can block APPQ— Move RemoteCall service to a separate queue

23 January 200243


1 domain on many nodes

DATABASE

PSAPPSRVAPPQWSL WSH

BBL

BB

TLISTEN

BRIDGE TLISTEN

BRIDGE

WSL WSH

BBL

BB PSAPPSRVAPPQ

Multi-Node Domain

DBBL

23 January 200244


Multi-Node Domain

Distinct Bulletin Board Liaison process— Master Node

TLISTEN process— Listens for incoming requests

BRIDGE— Sends & Receives messages between nodes— in a load balancing environment

– configurable bridge load add to remote queues– limiting number of messages sent across the bridge

23 January 200245


Multi-Node Domain

DO ATTEMPT THIS WITHOUT BEA CONSULTANCY

At UBS— PT 7.5x— susceptible to network errors

— integrated into HP service guard— ultimately this was removed from production because of

transient network errors between the nodes causing the domain to ‘partition’

23 January 200246


So what is going on?

tmadmin— Tuxedo command line interface— also available within psadmin

23 January 200247


tmadmin commands

pq - printqueues— amount of work queued and waiting— not always accurate

23 January 200248


tmadmin commands

psr - printservers— reports activity of individual server processes

23 January 200249


tmadmin commands

psc - printservices— each service on each server

23 January 200250


tmadmin commands

Lots of other tmadmin commands

documented in the Tuxedo documentation

Use them to— dynamically reconfigure the domain— dynamically resize the domain

– start/shutdown extra servers— monitoring scripts

– tmadmin -r for read only mode

23 January 200251


Java Administrative Applet

23 January 200252


Conclusion

So now you know— BEA Tuxedo/Jolt is a messaging protocol— How the messages are moved around— How PeopleSoft has used it— How the PIA has been constructed— Options for Tuning and Sizing— Load Balancing— Multi-Node domains— Monitoring Scripts

BEA Documentation— is on your CD— http://e-docs.bea.com

23 January 200253


More Information about PT8 Application Server

PeopleSoft 2000 Conference: Americas— 8006: PeopleTools 8.1 Application Server— 8010: Inside the Application Server

– Buzz Voelker– PeopleTools Application Server Team

23 January 200254


Final Caveat

PeopleSoft will not support any of the changes that I have described in this presentation

Everything described in this presentation has been done, in production, at UBS

If you want to use a multi-node domain you are strongly recommended to arrange support/consultancy direct from BEA.

23 January 200255


Questions?

PeopleTools Application Server & BEA Tuxedo [email protected] Consultancy Ltd.

http://www.go-faster.co.uk

Documents

Bea Internals