PIA Load Balancing, JVMs and Garbage Collection

David KurtzGo-Faster Consultancy Ltd.

david.kurtz@go-faster.co.uk

www.go-faster.co.uk

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 2

Who am I?

• DBA– Independent consultant

• System Performance tuning– PeopleSoft ERP

– Oracle RDBMS

• Training Course

• UK Oracle User Group – Unix SIG

• Oak Table– www.oaktable.net

• Book– www.psftdba.com

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 3

Resources

• If you can’t hear me say so now.

• Please feel free to ask questions as we go along.

• The presentation will be available from– www.go-faster.co.uk

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 4

A War Story

• Two forms of load balancing– Windows Network

Load Balancing

– IIS Proxy

• PSFT Recommendations:– Red Paper HA & Clustering

WEB1PIA1

50.0.0.7:7001

PIA250.0.0.7:17001

IIS50.0.0.7:443

WEB2PIA1

50.0.0.8:7001

PIA250.0.0.8:17001

IIS50.0.0.8:443

Win

do

ws

Netw

ork

Lo

ad B

alancin

g

50.0.0.50:443

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 5

Problems

• Strange Pauses– Mostly to do with garbage collection

• IIS Proxy wouldn’t load balance– All of load on one PIA or the other

• Pauses cause IIS Proxy failover

• On Failover users loose their sessions

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 6

IIS Load Balancing

• iisproxy.ini

WebLogicCluster=50.0.0.8:7001,50.0.0.8:17001DynamicServerList=OFF…

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 7

IIS Load Balancing

• Uneven loading of PIAs caused overload of one PIA.

• Duration of garbage collection with 100% of load on single PIA much higher

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 8

How do I know the load is balanced?

• Web server access log

• PeopleSoft Performance Monitor• PT>=8.44

– Event 150: JVM Status– Event 152: Web Site Status– Event 153: Web Servlet Status

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 9

How do I know the load is balanced?

PIA Activity

0

50

100

150

200

250

300

Wed 29.06.05 12:00 Wed 29.06.05 13:00 Wed 29.06.05 14:00 Wed 29.06.05 15:00

Time

Ses

sio

n i

n W

eb-A

pp

hrmsweb001 17001:17002 hrmsweb001 7001:7002 hrmsweb002 17001:17002 hrmsweb002 7001:7002

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 10

JVM Garbage Collection

• Java Command Line Options– verbosegc

• Writes to weblogic.log

– Xloggc:<filename>

0.000: [GC 46591K->3553K(518464K), 0.1262603 secs]19.604: [GC 50145K->6662K(518464K), 0.3389070 secs]70.052: [Full_GC 46565K->10901K(518464K), 0.9507536 secs]90.967: [Full_GC 55273K->16571K(518464K), 1.1008306 secs]

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 11

Garbage Collector Log

90.967: [Full_GC 55273K->16571K(518464K), 1.1008306 secs]

– Elapsed time since JVM started– Collection type– Size of live objects before and after GC – Total space available, less permanent objects– Duration of collection

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 12

Timestamps on GC log

-XX:+PrintGCTimeStamps

• But I couldn’t get it to work!

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 13

Other Garbage Collection Options

• -XX:+UseConcMarkSweepGC

• -XX:+UseParNewGC

• -XX:+CMSParallelRemarkEnabled

• Caveat: These options are not presented as being any kind of panacea, but something to consider. You need to do your own testing.

• BTW: I got the idea for this from a UKOUG Oracle Application Server SIG!

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 14

-XX:+UseConcMarkSweepGC

• Concurrent Low Pause Collector

• GC is, as far as possible, concurrent with the execution of the application.– Uses a separate thread

• App is paused for shorter periods during GC

• Only effective on multi-processor machines

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 15

-XX:+UseParNewGC

• Parallel young generation GC with concurrent GC– for multiprocessor machines, – enables multi threaded young generation

collection– Application is not paused

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 16

-XX:+CMSParallelRemarkEnabled

• Reduces remark pauses

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 17

Default Garbage Collector

GC Duration

0.001

0.01

0.1

1

10

0 20000 40000 60000 80000 100000 120000

Offset Seconds

Du

rati

on

(s

)

DURATION

• Individual GC duration

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 18

Parallel Garbage Collector

GC Duration

0.001

0.01

0.1

1

10

0 20000 40000 60000 80000 100000 120000

Offset Seconds

Du

rati

on

(s

)

DURATION

• Individual GC duration

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 19

Default Garbage Collector

GC Duration

0.01

0.1

1

10

0 20000 40000 60000 80000 100000 120000

Offset Seconds

GC

Du

rati

on

se

co

nd

s/m

inu

te

DURATION

• Cumulative GC duration / minute

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 20

Parallel Garbage Collection

GC Duration

0.001

0.01

0.1

1

10

0 20000 40000 60000 80000 100000 120000

Offset Seconds

GC

Du

rati

on

se

co

nd

s/m

inu

te

DURATION

• Cumulative GC duration / minute

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 21

Conclusion

• Improved GC performance does not have a great effect on Component Performance

• It does reduce the ‘strange pause’ giving the user a better experience

• Parallel GC seems to slowly leak memory.– Possible bug in a particular JVM– I have only tested on Windows

PSFT Tech PUG 5.9.05 www.go-faster.co.uk 22

Questions?

PIA Load Balancing, JVMs and Garbage Collection

David KurtzGo-Faster Consultancy Ltd.

david.kurtz@go-faster.co.uk

www.go-faster.co.uk