Scaling Up theJBoss Application ServerPeter JohnsonDavid StrongLinuxWorld Open Solutions SummitNew York, 14 February, 2007

14 Feb 2007 Page 2

Agenda

Introduction

Methodology

Hardware Tuning

JVM Tuning

Application Server Tuning

Database Tuning

Miscellaneous Tuning

Conclusion

14 Feb 2007 Page 3

16-CPU ES7000(32-bit)

Configuration

Unisys JVM1.4.1_07/1.4.2_08

RHEL 4, SLES 9

JBoss ApplicationServer 3.2.6/4.0.2

SPECjAppServer2002

DriverDriver2-CPU2-CPU

Oracle 9i/10gMySQL 4.1

PostgreSQL 8.1

14 Feb 2007 Page 4

SPECjAppServer2002

Industry standard client/server benchmark– Measures J2EE application server performance– Complete end-to-end web application

• EJB 2.0• JDBC to relational database• Web services

Metrics:– TOPS – Total Operations Per Second– $ per TOPS – price/performance

http://www.spec.org/jAppServer2002/docs/FAQ.html

14 Feb 2007 Page 5

JBoss Performance Increase

113 14

30

48

05

101520253035404550

3.2.3 3.2.6 4.0.2 3.2.6 3.2.6

NOTE: These are relative performance results, not actual SPECjAppServer numbers

App serverinstances

MySQLPostgreSQL

14 Feb 2007 Page 6

Agenda

Introduction

Methodology

Hardware Tuning

JVM Tuning

Application Server Tuning

Database Tuning

Miscellaneous Tuning

Conclusion

14 Feb 2007 Page 7

Methodology

1. Baseline

2. Observe

3. Evaluate

4. Modify

14 Feb 2007 Page 8

SPECjAppServer2002 Statistics

Injection Rate (IR) – benchmark driver parameter– Identifies pressure placed on the application– Starts at 1 and goes up from there– Recommendation: start at 1, increment by 5 or 10.

Total Operations Per Second (TOPS)– Benchmark result– Should be ~1.7 TOPS/IR

Example– IR=50, TOPS=60, TOPS/IR=1.1 evaluate & modify– IR=50, TOPS=83, TOPS/IR=~1.7 increase IR to 60

http://www.spec.org/jAppServer2002

14 Feb 2007 Page 9

Tuning Philosophy

There is always another bottleneck

Sometimes, reducing the bottleneck is sufficient

Everything is fair game (almost):– Add more hardware (CPUs, NICs, memory, RAID disk, etc.)– Tweak the JVM options– Tweak the database settings– Tweak the application server settings– Tweak the application settings– Modify the application

Not allowed withstandard benchmarks

Cost is taken into account inSPECjAppServer 2002 results

14 Feb 2007 Page 10

Agenda

Introduction

Methodology

Hardware Tuning

JVM Tuning

Application Server Tuning

Database Tuning

Miscellaneous Tuning

Conclusion

14 Feb 2007 Page 11

Hardware Tuning

Plan for growth

Memory – you never have enough– 2GB RAM per application server

Monitor system usage– Top (and other tools that come with the OS)– Commercial tools

CPU usage– If over 80% => add another CPU

14 Feb 2007 Page 12

More Hardware Tuning

Network considerations– Private network(s)– Fast NICs, multiple NICs

• Set speed on NICs, don’t use auto-detect

– Routers and network configuration

Fast hard drives on database server– RAID array, 15,000 RPM SCSI

Watch out for domino effect– If app server host now has more capacity, beware the db host

Don’t forget the driver system

14 Feb 2007 Page 13

Agenda

Introduction

Methodology

Hardware Tuning

JVM Tuning

Application Server Tuning

Database Tuning

Miscellaneous Tuning

Conclusion

14 Feb 2007 Page 14

HotSpot™ Heap Primer

Young Generation– Eden

• Where new objects are created

– Survivor spaces• Where garbage collector places objects that are still in use

Old Generation– Where tenured objects are placed

References– http://java.sun.com/docs/hotspot

14 Feb 2007 Page 15

JVM Heap Tuning

Set the heap sizes– Set min and max sizes to same value– Set young generation to 1/3 size of heap– Ex: -Xms1200m –Xmx1200m

–XX:newSize=400m –XX:MaxNewSize=400m

Monitor heap usage– -verbose:gc

• Basic before/after garbage collection heap sizes

– -XX:PrintHeapAtGC• Detailed before/after garbage collection heap size

Set the heap sizes appropriately

14 Feb 2007 Page 16

Other JVM Options

Use thread-local allocation blocks– Section of young generation allocated to each thread– -XX:+UseTLAB– Watch the young generation size

• # threads * TLAB size < young generation size

Delay the distributed garbage collection (DGC)– DGC performs full garbage collection every 60 seconds– This sets it to every hour:

-Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

14 Feb 2007 Page 17

Agenda

Introduction

Methodology

Hardware Tuning

JVM Tuning

Application Server Tuning

Database Tuning

Miscellaneous Tuning

Conclusion

14 Feb 2007 Page 18

Turn off Hot Deployment

The SPECjAppServer application doesn’t change

No other applications will be deployed

<attribute name="ScanPeriod">5000</attribute>

jboss-service.xml

<attribute name="ScanEnabled">false</attribute>

14 Feb 2007 Page 19

Remove Unused Services

Deploy SPECjAppServer to the ‘default’ configuration

Remove the following services:– hsqldb-ds.xml– scheduler-service.xml– snmp-adaptor.sar– monitoring-service.xml– mail-service.xml– schedule-manager-service.xml– jms

14 Feb 2007 Page 20

Database Connection Pool Size

Size of database connection pool related to injection rate

For IR up to 50– Size = (8 * IR) + 10%– Example: IR = 10, Size = (8 * 10) + 10% = 88

For IR over 50– Size = ( (8 * IR) + 10% ) / 10– Example: IR = 100

• Size = ( (8 * 100) + 10% ) / 10 = (880) / 10 = 88

Set <min-pool-size> and <max-pool-size> to same value

14 Feb 2007 Page 21

Other Data Source Configuration

Increase <blocking-timeout-millis>– Prevents failed transactions– Recommended starting setting: 30000

<prepared-statement-cache-size>– Allows for prepared statement reuse– Recommended size: 200

Set to number of prepared statements in your application

14 Feb 2007 Page 22

New Persistence Manager

standardjboss.xml– Container name: cmp2.x jdbc2 pm– Persistence manager: JDBCStoreManager2

Two-level cache– Entity beans stored in a transaction-level cache– Global cache holds “database rows”

Optimistic locking recommendedhttp://www.jboss.org/wiki/Wiki.jsp?page=CMP2xJDBC2PM

14 Feb 2007 Page 23

Other Configuration

Remove metrics collector interceptor– Standard Stateful Session Bean– Standard Stateless Session Bean

<interceptor transaction="Bean" metricsEnabled="true"> org.jboss.ejb.plugins.MetricsInterceptor</interceptor> <interceptor transaction=“Container" metricsEnabled="true"> org.jboss.ejb.plugins.MetricsInterceptor</interceptor>

standardjboss.xml

14 Feb 2007 Page 24

Multiple Application Servers

Symptoms– Low system utilization on all tiers– No obvious bottlenecks– Fairly high injection rate

Add a second application server instance– Separate hardware if app server system maxed out– On ES7000 – added to same system due to scale-up architecture

• Beware of port conflicts

Performance boost expectation– 2 app servers => about 2 * single app server– N app servers => not quite N * single app server

2103

1502

801

IR*#

* - These are hypothetical injection rates and should not be interpreted as actual or observed rates.

14 Feb 2007 Page 25

Agenda

Introduction

Methodology

Hardware Tuning

JVM Tuning

Application Server Tuning

Database Tuning

Miscellaneous Tuning

Conclusion

14 Feb 2007 Page 26

PostgreSQL Configuration

Settings in postgres.conf:– shared_buffers=256MB

• This is not total memory used, rather RAM dedicated to requests– wal_buffers=4096 (x 8K = 32MB)

• Number of disk-page buffers used for write-ahead log (uses some of shared_buffers)

– checkpoint_segments=64• Maximum distance between write-ahead log checkpoints (x16MB)• Prevents “Checkpoints occurring too quickly” errors in log

– stats_start_collector=off• Turn off to get better results, turn back on if you need to analyze a

database bottleneck– max_connections – set to same as JDBC connection pool max

http://www.powerpostgresql.com/PerfList/

14 Feb 2007 Page 27

MySQL Configuration

Used my-innodb-heavy-4G.ini

Placed transaction logs and data on different disks– innodb_data_home_dir– innodb_data_file_path– innodb_log_group_home_dir

max_connections - set to same as JDBC connection pool max

14 Feb 2007 Page 28

Oracle 9i/10g Tuning

Buffer cache– Use Oracle Enterprise Manager (OEM) to access the Memory

configuration for the database– Click on the Suggestion button for suggested changes– Refer to Oracle tuning guide for more info

Processes– Symptom: exceptions stating Oracle out of processes– Use OEM to change Processes parameter: 1000-1500– Also change Sessions and Transactions

14 Feb 2007 Page 29

Oracle 9i/10g Tuning, cont.

Symptom: Insufficient connectionsCause: stack size too bigFix (Windows):1. Shutdown TNS and database services2. Find out size of existing stacks:

- orastack tnslsnr.exe- orastack oracle.exeShould see size of 1 MB

3. Change stack size to 0.5 MB:- orastack tnslsnr.exe 524288- orastack oracle.exe 524288

4. Restart TNS and database services

WARNINGDo not go lower

than 0.5 MB

14 Feb 2007 Page 30

Agenda

Introduction

Methodology

Hardware Tuning

JVM Tuning

Application Server Tuning

Database Tuning

Miscellaneous Tuning

Conclusion

14 Feb 2007 Page 31

OS

Process and Thread Affinity

Contain process/threads to run on certain processors– Keep threads within processor group if external

cache

Process affinity– Useful if threads share many common objects

• J2EE-based application servers

Thread affinity– Useful if threads are independent– Useful in scale-up scenario

Recommended: 2-4 CPUs/app server

AS2AS1

CPU

CPU

CPU

CPU

AS3

CPU

CPU

CPU

CPU

14 Feb 2007 Page 32

Agenda

Introduction

Methodology

Hardware Tuning

JVM Tuning

Application Server Tuning

Database Tuning

Miscellaneous Tuning

Conclusion

14 Feb 2007 Page 33

Concluding Thoughts

Significant performance and scalability improvements went into JBoss Application Server 3.2.6 and 4.0.2

Performance testing is an iterative process – you don’t run the benchmark just once

There is always another bottleneck – stop when you meet your SLA objectives

You too can tweak your system to improve its performancehttp://www.jboss.org/wiki/Wiki.jsp?page=SPECjAppServer2002Tuning

14 Feb 2007 Page 34

Thank you!

Questions?