Upload
raczgyorgy
View
145
Download
1
Embed Size (px)
Citation preview
www.jboss.com
2 In search of hIgh-performance applIcatIons
2 performance tunIng prIncIples
2 Why tune for performance?
3 Performance priorities
3 A word about performance benchmarks
5 JBoss eap 4.3 tunIng
5 Connection pooling
6 Thread pooling
8 Object and component pools
9 Logging
9 Caching
10 JBoss EAP performance tuning summary
11 Linux-specifictuning:Largememorypages
11 Linux-specifictuning:Tuningthevirtualmemorymanager
11 Database tuning update
12 Case study
13 applIcatIon performance tunIng: a contInuous process
13 appendIx: usIng large-page memory (lInux-specIfIc InstructIons)
Best practices for performance tuning JBoss enterprIse applIcatIon platform 4.3 tIps and trIcks for optImIzIng your applIcatIon’s performance
2 www.jboss.com
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
in search of high-performance applications
Overall,performancetuningisaveryimportantpartofcreating,maintaining,anddeployingasuccessfulbusinessapplication.Whetheryouarebuildingcustomapplicationsordeployingcommercial,off-the-shelfsolutions,youwilllikelyneedtotunetheapplication,thedatabase,themiddleware,orallthree.Infact,75%of performance issues originate with the application itself.
Whenorganizationsselectapplicationmiddleware,performanceisalwaysoneoftheirmostimportantselectioncriteria,ifnotthemostimportant.InmanycasesourcustomerstellustheychoseJBossEnterprise Application Platform (EAP) because of its superior performance. They know that many users ofJBossEAPareachievingsuperiorapplicationperformancedayafterday.
Togetthemostfromyourcompany’sinvestmentinmiddleware,developersandarchitectsneedtoknow thespecificwaystheycanachievesuperiorperformancewithJBossEAP.Whilewewouldallliketothink|thatanapplicationcouldperformwellstraightoutofthebox,thisisnotusuallythecase.Applications canhavewidelyvaryingcharacteristics,andwhilesomeapplicationsmightperformwellwithdefaultmiddlewaresettings,otherswillnot.
IfyouarenewtoJBossEAPorperformancetuning,thispaperwillintroduceyoutobestpracticesthatcanhelpyouavoidcommonperformancepitfallsasyouprepareyourapplicationforproduction.Ifyou’reanoldhandatapplicationperformanceissues,youknowthattechnologyisconstantlychanging.Youmaybenefitfrom an update on best practices in JBoss EAP performance tuning.
performance tuning principles
Why tune for performance?
Performance was once considered just another feature of an application. Today it is frequently considered themostimportantcharacteristicoftheapplication—onethatcanhaveasignificantimpactonyourbusinessandyourproductivity.Consideryourreactiontoaslowwebsite.Ifyou’relikemostpeople,youbecomefrustrated,losepatience,andgoelsewhere.Ifthatcompanyiscountingonrevenuefromwebsales,ithasnotonlylostyourattention—ithaslostbusiness.Evenforinternalapplications,poorperformancecanaffectproductivityifusershavetowaitordealwithunpredictablebehavior.Atbest,theymightfeelannoyed,losealittletime,orformanegativeopinionoftheirITdepartments.Atworst,businesstransactionsmaybelost,orcustomersmaygowithoutimportantserviceifusersmustworkaround a poorly performing application.
But user experience is not the only reason to tune for performance. A well-performing application will generallyusefewerhardwareandsoftwareresources.Acompanycanoptimizeitsinvestmentinhardwarewhenapplicationsaretunedappropriately,whetherthatmeansusingoldersystemslonger,purchasingnewsystemsthataremoremodestlysized,orusingfewersystemsoverall.Onthesoftwareside,awell-performingapplicationwillgenerallyneedtousefewerCPUcountsorsoftwarelicenses,nomatterwhattypeofsoftwareisinvolved.Reducingsoftwarecostscansavethecompanysignificantmoneyovertime.
www.jboss.com 3
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
performance prIorItIes
Superiorperformancecomesfrommanylayersoftheapplicationstack,notjusttheapplicationserver.Inmanycases,thewaytheapplicationisdesignedandhowitconnectstothedatabaseandothersoftwarecomponentscanhavealargeimpactonoverallapplicationperformance.Manyorganizationsspendmoretimetuningtheircustom-builtapplicationsanddatabasesthantheirunderlyingapplicationservers.Sokeepinmindthatasuperiorlyperformingapplicationservermayhaveonlyaminorimpactontheoverallperformance of your applications.
a Word aBout performance Benchmarks
Some organizations rely on industry-standard performance benchmarks when selecting middleware. While benchmarkscanhelpvendors,theycandeceiveyouasanevaluatorbecausebenchmarkapplicationsareusuallyverydifferentfromtheapplicationsyouwillruninproduction.Thecurrentbenchmarkforapplicationservers,forexample,SPECjAppServer2004,isa2004applicationthatdoesn’ttakeadvantageofmanyrecentdevelopmentsinJava.Butasidefromthat,everyapplicationisdifferentandthesystemsonwhichbenchmarkapplicationsarerunmaybeverydifferentfromyours.Understandinghowagivenbenchmarkrunsmaytellyouverylittleabouthowyourapplicationwillrunonyourhardwarewithyoursettings.Beforeyourelyonabenchmark,youneedtoinvestigatethesoftwareandhardwareconfigurationsused,theserverandnetworkconfigurations,thesettingsused,thearchitectureoftheapplication,andhowallofthesecomparewithyourownenvironment.Werecommendthatbeforeselectingasystem,youtestitwithanapplicationandhardwareconfigurationasclosetoyoursaspossible.1
performance tuning principle #1: understand your performance requirements
Thefirststepintuningyourapplicationforperformanceistounderstandtheconditionsunderwhichitwillneedtoperform.Ifyourapplicationisareplacementforanexistingsolution,thenyourorganizationalreadyhassignificantexperiencewiththatsolution.Chancesareyouhavemetricsavailablesuchasthenumberofusers,thenumberoftransactionsperday,thevariationsintransactionloadortypeoverthecourseofaday,week,month,oryear,andsoon.
Ifyouaredeployingacompletelynewsolution,youwillneedtostudythatapplication’sbusinesscontextverycarefully.Themoreyouunderstandexactlyhowyourapplicationwillbeused,themoresuccessfulyourperformancetuningwillbe.Sometimesassumptionsdonotreflectreality.Inonecase,adevelopmentteamcreatedandtesteditsapplicationbasedonanassumedworkloadof60,000transactionsperday.Thefirstdayinproduction,6milliontransactionsoccurred.
performance tuning principle #2: plan for peaks, not averages
Asyouexamineperformancerequirements,oneofyourgoalsshouldbetodevelopaprofileofyourapplication’sworkloadwithspecialattentiontothepeaks.Forexample,manybusinessapplicationsexperiencedailypeaksinthemorningandafternoonwithavalleyduringthemiddleofthedaywhen
1 Foramorein-depthexplanationoftheprosandconsofperformancebenchmarks,seeDoPerformance Benchmarks&ComparisonsMatter?—AGuidetoAssessingApplicationServerPerformanceResults.
4 www.jboss.com
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
employeesareeatinglunch.Someapplicationsexperiencepeaksattheendofthemonth,quarter,or season.Yourapplication’sworkloadprofilewilldependonthespecificsofyourbusiness,butyoushouldalwayspayparticularattentiontoperiodsofpeakworkload.Oneofthebiggestmistakesdevelopersmake istorelyonanaverage(averagedailyworkload,forexample).Averagesarenotsufficienttoensurethatyour application will perform during periods of peak load.
performance tuning principle #3: always instrument your application
Allapplicationsshouldbeinstrumentedtoprovideinformationforperformanceanalysis.Businessconditions,includingcustomerbehaviorandworkloadcurves,canchangedramaticallyovertime,soevenifanapplicationonceperformedwell,itmaynotperformwellundertoday’sconditions.Ifyourunintotrouble,andyourapplicationhasnotbeeninstrumentedforperformance,thenyouhavenoeasywaytoknowwhereyourproblemsare.Butifyourapplicationisappropriatelyinstrumented,thenyou’llbeabletomonitorchanges in business conditions easily and tune your applications to match before problems occur.
Inthepast,inordertoinstrumentanapplication,adeveloperneededtoembedcodewithintheapplicationitself.Todaymanysolutionsprovideinformationwithoutrequiringdeveloperstocode.WithJBossEAP,thecallstatisticsinthecontainerprovideyouthenumberofcalls,concurrentcalls,mincalltime,maxcalltime,averagecalltime,andthestandarddeviationofcalltimes.Hibernatestatisticsprovidequeryexecutiontimes,andJBossONMonitoringcandisplayallofthisinformationandgraphtheresultsinrealtime.Additionally,numerousthird-partyapplicationperformancemanagement(APM)productsareavailable fromcertifiedRedHatpartnerstoprovideapplicationperformanceinformationaswell.
Finally,forperformance-criticalsituationswhereoneoftheothertoolsdoesn’tprovidetheinformationyouneed,youcanwriteyourowninstrumentationusingtheJBossAOPframework.Thisframeworkoffersquitea few features that enable you to see exactly what is happening at runtime. Whether you choose to write yourowninstrumentationwilldependontheapplication,yourskillsandavailabletime,andtheimportanceoftheapplication’sperformancecharacteristicstoitsoverallsuccess.
performance tuning principle #4: understand where your application spends its time
One important reason to instrument your application is to understand where time is being spent. While you wanttoknowabouttimeacrosseachlayerofyourapplicationstack,thetoolswithinJBossEAPwillhelp youunderstandonlypartofthisequation.Ifyourapplicationisspendingtoomuchtimeinthedatabase, forexample,thenyoumayneedtofocusonyourdatabasestatisticstopindowntheproblem.
Byunderstandingwhereyourapplicationspendsitstime,youwillbeabletoavoidthe“shotgunmethod”of performance tuning — trying multiple solutions to common problems without knowing whether any of themarerelevanttoyourproblem.Youmightsolveyourproblemthisway,buttheprobabilityislow.Manydeveloperswhoassumetheyknowwhatiscausingtheirperformanceproblemaremistaken,resultinginproblemsthatpersistformonthsorsometimesyears.Thatiswhyhavinganobjectivewaytoknowwhereyourapplicationisspendingtimewillhelpyouavoidspendingtoomuchtimesolvingperformanceproblems.
www.jboss.com 5
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
performance tuning principle #5 replicate or model your production environment
Asyouworktounderstandexactlyhowyourapplicationwillperforminproduction,youwillneedtorunitinatestenvironment.Somecompaniesmakeitstandardpracticetoimplementatestenvironmentthatisanexactreplicaoftheirproductionenvironment.Thisisanidealapproachinmanyrespects,becausedevelopersdon’thavetowonderhowtheirapplicationswillscaleforlargersystems.
Formanyorganizations,though,thecostofreplicatingtheproductionenvironmentisasignificantobstacle.Theirtestenvironmentstypicallyemploysmallersystems.Ifthisisyoursituation,thenyouwillneedtocreateamodelthatdefinestherelationshipbetweenperformanceinyourtestenvironmentandperformanceintheproductionenvironment.Thatis,youneedtomodelhowtheapplicationwillscale.Asyoudoso,keepthefollowinginmind:
do not assume a linear relationship.• Performance doesn’t often scale in a linearly. Vendors may tellyoutheyexperiencelinearresults,butforrealproductionapplications,thisisrarelythecase.
Be conservative in creating your model, and use actual historical data wherever possible.• Ifyoucanbaseyourmodelondatafrompastexperiencewithotherapplications,youwillbeabletorefineitovertimeandincreaseyourconfidenceinitsaccuracy.
JBoss eap 4.3 tuning
While75%ofallperformanceproblemsaretheresultoftheapplication,notthemiddlewareorthe operatingsystem,youstillneedtoknowhowtotunesettingsinthemiddlewaretoimproveperformance andthroughput.Dependingonyourapplication,oneormoreofthefollowingmayneedsomeattention:
Connection pooling•
Thread pooling•
Object and component pools•
Logging•
Caching•
Thefollowingsectionsprovideanoverviewofeachoftheseareas.
connectIon poolIng
Connection pooling and thread pooling are the most important areas to consider when you want to maximize throughputonmodernhardware.Intermsofsystemresources,databaseconnectionsareexpensivetosetupandteardown.Butinspiteofthis,someapplicationscreateanewconnectiontothedatabasewitheveryqueryortransactionandthenclosethatconnectionimmediately.Thispracticeaddsagreatdealofoverheadtotransactionprocessingandcanleadtopoorperformance.
6 www.jboss.com
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
TotakeadvantageoftherobustconnectionpoolinginJBossEnterpriseApplicationPlatform,startbyadjustingconnectionpoolsettingsonthedatasourcedefinitionsthatyoucansetupinthedeploydirectory.Settheminimumpoolsizetothelevelyouwanttotunefor,thensetthemaximumatleast25-30%higher.Don’tbeconcernedaboutsettingthemaximumtoohigh,asthepoolwillshrinkautomaticallyyoudon’tneedthat many connections.
Todeterminethepropersizing,youcanmonitoryourconnectionusage.Toosmallapoolwillalsothrottletheapplication,astheEAPwillqueuetherequestforadefaultof30,000milliseconds(or30seconds)beforegivingupandthrowinganexception.Ifyoustartseeingalotof30-secondtimeouts,thatisastrongcluethatyouneedtolookatyourconnectionpooling.YoucanmonitortheconnectionpoolutilizationfromtheEAPJMXconsole,fromJBossON,orfromdatabase-specifictools.
Hereisanexampleofconnectionpoolsettingsforadatasource:
<datasources> <local-tx-datasource> <jndi-name>MySQLDS</jndi-name> <connection-url>jdbc:mysql://<host>:3306/Schema</connection-url> <driver-class>com.mysql.jdbc.Driver</driver-class> <user-name>someuser</user-name> <password>somepassword</password> <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.MySQLExceptionSorter</exception-sorter-class-name> <min-pool-size>75</min-pool-size> <max-pool-size>100/max-pool-size> ... <!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml --> <metadata> <type-mapping>mySQL</type-mapping> </metadata> </local-tx-datasource> </datasources>
thread poolIng
Thread pooling is the next most important area to consider as you tune your application for performance. JBossEAPhasrobustthreadpooling,butbeforeyoucansizethethreadpoolsappropriately,youneedto know how they are used and which ones might be affecting your application’s performance. The characteristicsofyourspecificapplicationwilldeterminewhichthreadpoolsareusedandwhichones mightbecomebottlenecks.Thiscanvarysignificantlyfromapplicationtoapplication.Thetablebelowprovidesasummaryofhoweachthreadpoolisused.
www.jboss.com 7
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
removing connectors
Ifyouarecertainthataparticularconnectorwillnotbeusedinyourapplication,werecommendremoving it.Forexample,manyapplicationsuseeithertheHTTPdthreadpoolorthemod_jkthreadpoolbutnotboth.Soyoucanremovetheoneyoudon’tneed.
monitoring and tuning thread pools
MonitoryourthreadpoolsviatheEAP4.3JMXConsole,whichdisplaysnotonlythenumberofactivethreads for each pool but also the queue size. We recommend that you adjust thread pool settings through JBossOperationsNetwork,asitallowsyoutodefinesettingsthatpersist.SettingsadjustedviatheJMXconsolewillnotsurviveareboot.IfyouusetheJMXconsole,pleaseremembertogobackandeditthefile if you want the adjustments to be permanent.
thread pool Where Is It defIned? hoW Is It used?
System thread pool Injboss-service.xmlintheconfdirectory ForJNDInaming—thedefaultsetting isfineformostcases
HTTPDthreadpool in JBoss Web
Intheserver.xmlfileunder<server>/deploy/jboss-web.deployer
WhenmakingHTTPrequestsdirectly to EAP
AJP thread pool Intheconnectorsectionofserver.xml WhenmakingHTTPrequeststhroughmod_jk
JCA thread pool (also called the Work Managerthreadpool)
<server>/deploy/jbossjca- service.xml
InconjunctionwithJMS,asJBossMessagingusesJCAinflowastheintegration into EAP
JBossMessagingthreadpool (for remote clients)
<server>deploy/jboss-messaging.sar/remoting-bisocket-service.xml
Pools the TCP sockets
JBossMessagingthreadpool(inJVMclients)
Notdirectlyconfigurable Pool is bounded by the number of MDBsyouhavedefined
EJB3(sameJVM) ClientsinthesameJVMwillrun onwhateverthreadpooltheyare already using
Forexample,awebrequestcomes in through the AJP connector. When itcallsanEJB3bean,itwillcontinueexecuting on the AJP connector thread pool.
EJB (remote clients) <server>/ejb3/deployer/META-INF/ jboss-service.xml
8 www.jboss.com
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
example: http thread pool
<Connector port=”8080” address=”${jboss.bind.address}” maxThreads=”250” maxHttpHeaderSize=”8192” emptySessionPath=”true” protocol=”HTTP/1.1” enableLookups=”false” redirectPort=”8443” acceptCount=”100” connectionTimeout=”20000” disableUploadTimeout=”true” />
example: mod_jk or aJp thread pool
<!-- Define an AJP 1.3 Connector on port 8009 --> <Connector port=”8009” address=”${jboss.bind.address}” protocol=”AJP/1.3” emptySessionPath=”true” enableLookups=”false” redirectPort=”8443” maxThreads=”200” />
example: Jca thread pool
<mbean code=”org.jboss.util.threadpool.BasicThreadPool” name=”jboss.jca:service=WorkManagerThreadPool”> <!-- The name that appears in thread names --> <attribute name=”Name”>WorkManager</attribute> <!-- The maximum amount of work in the queue --> <attribute name=”MaximumQueueSize”>1024</attribute> <!-- The maximum number of active threads --> <attribute name=”MaximumPoolSize”>100</attribute> <!-- How long to keep threads alive after their last work (default one minute) --> <attribute name=”KeepAliveTime”>60000</attribute> </mbean> <mbean code=”org.jboss.resource.work.JBossWorkManager” name=”jboss.jca:service=WorkManager”> <depends optional-attribute-name=”ThreadPoolName”>jboss.jca:service= WorkManagerThreadPool</depends> <depends optional-attribute-name=”XATerminatorName”>jboss:service= TransactionManager</depends> </mbean>
oBJect and component pools
Object pools and component pools are essentially the same thing. Their settings represent the number of objectinstances.ForEJB3twotypesofpoolsaredefinedin<server>/deploy/ejb3-interceptors-aop.xml.ThesearetheThreadLocalPoolandtheStrictMaxPool.Bydefault,StatelessSessionandStatefulSessionBeansusetheThreadLocalPool,whichisbackedbyanInfinitePoolwithnomaximumsize.Therefore,itgrowsaccordingtovolumeinyourapplication.Thishasthedistinctadvantageofnotneedingtobetuned.Bydefault,MessageDrivenBeans(MDBs)usetheStrictMaxPool.Thispoolactuallyobeysamaximum,will
www.jboss.com 9
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
queueuprequestswhenthatmaximumhasbeenreached,andwilltimeoutanythinginthequeueifthereisnotanavailablereferencefromthepool.Inthiscase,thesystemwillthrowanexceptionandiftheproblemoccurredinmid-transaction,youwillexperienceatransactionrollback.Giventheimpactfailedtransactionscanhaveonyourbusiness,werecommendthatyoumonitortheStrictMaxPoolcloselyviatheJMXconsole.
loggIng
Developersshouldtakefulladvantageoflogginginthedevelopmentandtestingphasesoftheapplicationlifecycle.Inproduction,however,loggingcancausebottlenecks.Youwanttobesurethatloggingprovidesyou with useful information without hurting application throughput. Consider making the following changes asyoupromoteyourapplicationintoproduction:
turn off console logging in production.• InEAP’sdefaultconfiguration,consoleloggingisenabled,whichmeansyouhavetheopportunitytoseeallthelogsfromyourIDE.Inproduction,thisisanexpensiveprocesswithunbufferedI/O.Whilesomeapplicationsmaybefinewithconsolelogging,high-volumeapplicationsbenefitfromturningitoff.JBossEAP4.3offersanewconfigurationset,namedProduction,whichgivesdevelopersabetterstartingpointforcreatingaproductionenvironment.IntheProductionconfiguration,consoleloggingisturnedoff.
turn down logging verbosity.• Thelessyoulog,thelessI/Owilloccur,andthebetteryouroverallapplicationthroughputwillbe.Loggingisalwaysatradeoff,sothinkcarefullyabouthowmuchloggingyou really need in production.
use asynchronous logging.• This can make a big difference for high-throughput applications. With asynchronouslogging,logmessageswillgointoaqueueandcontrolreturnstotheapplicationasif the logging had been completed. Then a separate thread executes the log operations from the queue.
Wrap debug log statements with• If(debugEnabled()). This simple practice can make a huge differenceifyourapplicationcontainsalotofdebuglogstatements.Withoutthisconditionset,yourapplicationcreatesallofthestringobjectsforeachofthelogstatements,andLog4jcreatestheLoggingEventobjectforeachlogstatementregardlessoftheloglevelthatissetbecausetheloglevelischeckedonlyafteralloftheseobjectshavebeencreated.InsomecasesthiscanleadtocreationofthousandsandthousandsoftemporaryStringandLoggingEventobjects,resultinginmemoryandgarbage collection issues and reducing throughput dramatically. By placing a conditional wrapper aroundyourdebuglogstatements,youcanensurethatunnecessarylogprocessingdoesnotaffect your throughput.
cachIng
Caching,whileoftenveryhelpful,maybeoneofthetrickiestareastotunecorrectly.JBossCacheisanintegralpartoftheEAPinfrastructure,andyourapplicationcanuseittocacheanythingyoulike.Cachingisespeciallyvaluableforapplicationsthatperformalotofreadoperationsagainstdatathatiseithercompletelystaticordoesn’tchangefrequently,suchasreferencedata.Forapplicationswithheavyuseofwriteoperations,cachingmaysimplyaddoverheadwithoutprovidinganyrealbenefit.Ifnotconfiguredproperly,Cachingmayactuallyreducethroughput.
10www.jboss.com
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
One of the easiest potential performance enhancements you can make is to cache EJB 3 entities. To definewhichentitiesyouwantcached,modifythefilepersistence.xml that you deploy with your EJB 3application(anexampleisshownbelow).Youdefinethecachesizeandevictionpolicyinthefileejb3-entity-cache-service.xmlfoundinthedeploydirectory.Thesedefinitionsmayrequiresometrial anderrortogetright.Rememberthatyouareworkingwithlimitedheapspace,andamisguidedcachingstrategy can be worse than none at all.
Evictionpolicywilldependonthespecificsofyourapplication.Youcandefinethecacheastransactional orread-write.Withverylargedatasets,notethatcachingmaynotprovidenoticeableperformancebenefits.Inthiscase,shrinkingthecachescanimproveperformance.Also,ifyourapplicationisverywrite-heavy,itmaynotbenefitfromcaching.Testingvariouscachingandnon-cachingconfigurationswillhelpyoudetermine this.
example: settings for caching in persistence.xml
<persistence> <persistence-unit name=”services” transaction-type=”JTA”> <provider>org.hibernate.ejb.HibernatePersistence</provider> <jta-data-source>java:/MySQLDS</jta-data-source> <properties> <property name=”hibernate.default_catalog” value=”EJB3”/> ... <property name=”hibernate.cache.provider_class” value=”org.jboss.ejb3.entity.TreeCacheProviderHook”/> <property name=”hibernate.treecache.mbean.object_name” value=”jboss.cache:service=EJB3EntityTreeCache”/> <property name=”hibernate.ejb.classcache.services.entities.Customer” value=”read-only”/> <property name=”hibernate.ejb.classcache.services.entities.Inventory” value=”transactional”/> ... </properties> </persistence-unit> </persistence>
JBoss eap performance tunIng summary
Tosummarize,keepthefollowingperformancerecommendationsinmindwhentuningJBossEAP4.3.
DefinedatasourcesinthedeploydirectoryandtakeadvantageofEAP’srobustconnectionpooling.•
Understandwhichthreadpoolsareactuallyusedbyyourapplication,removepoolsthatarenot •needed,monitorthenumberofactivethreadsandthequeuesize,andadjustpoolsizeifneeded.
Ifyouareusingmessage-drivenbeans,monitortheStrictMaxPoolcloselytoensurethatthemaximum•object pool size is not a bottleneck.
www.jboss.com 11
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
Employdifferentloggingstrategiesfordevelopmentandproduction.Especiallyforhigh-volume•applications,turnconsoleloggingoff,reduceloggingverbosity,useasynchronouslogging,andalwayswrap debug log statements with If (debugEnabled ()).
Takeadvantageofcachingifyourapplicationisread-heavyormakessignificantuseofreferencedata.•Avoidcachingforverylargedatasetsorwrite-heavyapplications.
lInux-specIfIc tunIng: large memory pages2
Fortoday’s64-bitsystems,useoflargememorypagescanimproveperformancesignificantly.Thedefaultmemorypagesizeistypically4KB.Whenyouareaddressinglargeamountsofmemory,thisquicklyadds uptoalargenumberofmemorypages—justonegigabyterequires262,144memorypages.That’salotfor asystemtokeeptrackof,whichtranslatestoalotofsystemoverhead.
Asidefromhelpingyouavoidtheoverheadofmappingsomanymemorypages,largememorypagesonLinuxcannotbeswappedtodisk.Thisisamajoradvantagebecausehavingyourheapspaceswaptodiskcanwreckhavocontheperformanceofyourapplication.
Largepagesupportbeginsat2MBandcanrunashighas256MBonsomehardwarearchitectures.Thesenumberswillvary,andyouwillneedtofindoutthevaluesforyourspecificserver.AllthemajorJVMsystemssupportlargememorypagesonLinux.Becauseitcanbetrickytosetup,weprovidesomespecificinstructions in the appendix.
Whenyouuselarge-pagememory,keepinmindthatthememoryisnotavailabletoapplicationsingeneral.Tootherapplications,yoursystemwilllookasthoughithashadmemoryremovedfromit,becausethismemorywillbededicatedtoyourspecificapplication.Refertotheappendixformorespecificinformationaboutconfiguringyourapplicationtouselargememorypages.
lInux-specIfIc tunIng: tunIng the vIrtual memory manager
InLinux,thevirtualmemorymanageristunable.Insomecases,youcanachieveperformancebenefitsbychangingthesettings.Youwon’tbeusingthefilesystemcachemuch,andyoudon’twantthesystemtofavorthefilesystemcacheoveryourapplications,whichcansometimesoccurwiththedefaultsettings.
In/etc/sysctl.conf you can set vm.swappinessto1topreventapplicationsfrombeingswappedto disk when there is memory pressure.
dataBase tunIng update
Moderndatabases,especially64-bit,areextremelyefficientatcachingdata.Intheearlydaysof64-bitdatabases,largebuffersizeswouldslowperformanceduetoelongatedsearchtimes,butthisisnolongertrue.Inourtesting,wehaveexperiencedOLTPapplicationswithverylargebuffercachesthatshowverygood results.
Consideryourapplication’scharacteristics—especiallyitsread/writeratios—whendecidingwhetherandhowtousedatabasecaching.Themajorityofapplicationsweseetodayareread-intensive,asmostapplicationsdrivetheirbusinesslogicbyreadingdatafromthedatabase.Themoreread-intensivetheapplicationis,themorecachingcanhelp.Iftheapplicationiswrite-heavy,orthedatasetisverylarge(fromadatawarehouse,forexample),thenalargecachewon’thelpandmayslowdowntheapplication.
2 WhilethissectionprovidesLinux-specificsettings,theprinciplesoflargememorypageuseareapplicable to any 64-bit system.
12 www.jboss.com
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
YoushoulduseDirectI/Oifyourdatabasesupportsit.Especiallywithalargecacheorawrite-intensiveworkload,youshouldavoiddoublebufferingwiththefilesystembuffercache.NotethatMySQL5documentationindicatesthatqueriesmayslowdownbyuptoafactorof3whenusingDIRECTI/O,butwehavenotexperiencedthisincaseswhereaproperlysizedbufferpoolisused.IfyouareusingDIRECTI/O,youshouldbesuretousethevirtualmemorysettingmentionedearlier.WealsosuggestusingasynchronousI/Oifyourdatabasesupportsit.
case study
TodemonstratethedramaticresultsthatcanbeachievedwithJBossEAPperformancetuning,weperformed an experiment using a sample application.
WebeganwithJBossEAP’sdefaultconfiguration(withoneminorexception),alongwithallLinuxparametersattheirdefaults.UsingGrinder,anopensourceJavaload-testingframework,wemeasuredthehighestthroughputwecouldachievewithallofthesesettings.Usingthesameapplication,wethenappliedmanyoftheoptimizationsdiscussedhere,bothtoJBossEAPandtotheoperatingsystem(inthiscase,Linux),andmeasured throughput again.
TheapplicationwasanEJB3applicationwithtwoservletsfortheUI,statelessandstatefulsessionbeans formostofthebusinesslogic,amessagedrivenPOJOforsomeasynchronousprocessing,andentitiesforthepersistence.Alltestswereperformedwith3.5GBheapandadatasourcewithsufficientconnections in the pool.
Thefollowinggraphfirstillustratesasinglevirtualuserasabaselineformeasuringscalabilityasweaddedusers.Asonewouldexpect,thefirsttwobarsarestatisticallyequalregardlessoftheperformanceoptimizationsmade.Thenexttwobarsshowhowmanyvirtualuserswecouldachievebeforewewerenolongerat85%oflinearscalability,asmeasuredbyatransactionspersecond(TPS)meanfromthebaseline(thebarsontheleft).Youcanseethatintheun-optimizedcase,wewereabletoreach25virtualusers, andintheoptimizedcase,wewereabletoreach75virtualusers.Weachievedathreefoldincreaseinthenumberofvirtualusersthatthesamesystemcouldsupport,simplybyoptimizingsystemperformance.
80
70
60
50
40
30
20
10
0
Baseline EJB 3 application 1 virtual userTPS (mean)
No optimizations
Baseline EJB 3 application 1 virtual userTPS (mean)Optimized
Top throughput test 25 virtual users
TPS (mean)No optimizations
Top throughput test 75 virtual users
TPS (mean)Optimized
JBoss eap performance tunIng
www.jboss.com 13
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
application performance tuning: a continuous process
Tuninganapplicationforoptimalperformancecanensureapositiveuserexperience,promotebusinessproductivity,andhelptooptimizeuseofhardwareandsoftwareresources.Performancetuningisnota one-timetask,butanongoingprocessthatensuresawell-performingapplicationasbusinessconditions andsystemtechnologychangeovertime.Applicationdevelopersandarchitectsshouldalwaysbepreparedtotunetheirapplicationsforperformancebothbeforeandaftertheyareputintoproduction.Asalways, themorebusiness-criticalanapplicationisandthehigherthevolumeoftransactionsitmustsupport,themore important performance tuning will be to your business.
Whentunedinaccordancewiththecharacteristicsofyourapplication,JBossEnterpriseApplicationPlatform(EAP)canprovidesuperiorapplicationperformance.Thispaperhasgivenyouanoverviewof basic performance principles as well as an introduction to performance tuning best practices for JBoss EAP.KeepinmindthatthesetechniquesapplytoanyoftheRedHatplatformsthatJBossEAPsupports:JBossEnterprisePortalPlatform,JBossEnterpriseSOAPlatform,andJBossEnterpriseBRMS.
ForadditionalinformationonJBossEAPperformancetuning,pleasereferto:
Managingapplicationperformance:JBossOperationsNetwork•
JBossEAP4.3ServerConfigurationGuide•
JBossEAPOptimizationandTuningAssistanceServices•
JBoss EAP 4.3 Product Documentation•
appendix: using large-page memory (linux-specific instructions)
Thisappendixcontainsaprocedureandexampledesignedtohelpyoutakeadvantageoflarge-pagememoryforyourapplications.Seethesection“Linux-specifictuning:Largememorypages”earlierinthispaperforanoverview.
1. tell the Jvm to use large-page memory.
SunJVMandOpenJDKrequirethefollowingoption,passedonthecommandline,touselargepages:
-XX:+UseLargePages
TheSuninstructionsleaveitatthat,butifyoudonothingelseyouwillmostlikelygetthefollowingerror:Failed to reserve shared memory (error-no=12)
The following sections describe additional steps you should complete.
14 www.jboss.com
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
2. set kernel parameters.
Set three kernel parameters in /etc/sysctl.confasfollows:
kernel.shmmax = n • Where n is equal to the number of bytes of the maximum shared memory segment allowed on the system.YoushouldsetittoatleastthesizeofthelargestheapsizeyouwanttousefortheJVM.Alternatively,youcansetittothetotalamountofmemoryinthesystem,andyouwillneverhave torevisitit.
vm.nr_hugepages = n• Wherenisequaltothenumberoflargepages.Youwillneedtolookupthelargepagesizein /proc/meminfo.
vm.huge_tlb_shm_group = gid• WheregidistheIDofasharedgroupIDfortheusersyouwanttohaveaccesstothelargepages. This setting enables you to limit access to the large memory segment.
3. set limits.conf parameters.
Next,setthesememlocklimitsin/etc/security/limits.conf:
<username> soft memlock n <username> hard memlock n
where <username>istheruntimeuseroftheJVM,andnisthenumberofpagesfromvm.nr_hugepages multiplied by the page size in KB from /proc/meminfo.
4. persist your settings.
Enterthecommand:
sysctl –p
Thisensuresthatthesettingsyoucreatedinstep3willsurviveareboot.
5. reboot.
WhentheOSallocatesthesepages,itmustfindcontiguousmemoryforthemortheoperationwillfail. Arebootwillpreventthisproblem.
6.Confirmpageallocation.
Whenlargepagesareallocated,/proc/meminfo will display a non-zero number for HugePages_Total. Ifyoudonotseeanon-zeronumber,thenyouarenotusingthelargepages,andsomethingisconfiguredincorrectly.
www.jboss.com 15
Best practices for performance tuning JBoss Enterprise Application Platform 4.3
WehavesometimesseenaproblemwithMYSQLwheretheSELinuxpolicywaspreventingitfromaccessingthe large pages. Check /var/log/messages for avc_denied messages (error-no=13 “permission denied”) in the mysqld.log.
example: setting large memory pages
Consideraserverwith8GBofmemory.Wewillallocate6GBtobesharedbytheJBossEAPJVMand aMySQLdatabase.
Thepagesizeis2MB,asshownin/proc/meminfo (Hugepagesize: 2048KB)
Configuration:/etc/sysctl.conf
Changemaximumsharedmemorysegmentsizeto8GB. •kernel.shmmax = 8589934592
Add the gid to the • hugetlb_shm_grouptogiveaccesstotheusers. vm.hugetlb_shm_group = 501
Add6GBin2MBpagestobesharedbetweentheJVMandMySQL. •vm.nr_hugepages = 3072
Calculations:
1024*1024*1024*8=8589934592
(1024*1024*1024*6)/(1024*1024*2)or6GB/2MB=3072pages
Configuration:/etc/security/limits.conf
AddthelimitsformemlocktoallowtheJVMandMySQLtoaccessthelarge-pagememory.•
jboss soft memlock 6291456 jboss hard memlock 6291456 mysql soft memlock 6291456 mysql hard memlock 6291456 root soft memlock 6291456 root hard memlock 6291456
Calculations:
3072largepages*2048KBpagesize–3072*2048=6291456
Configuration:/etc/group
AddJBossandMySQLuserstothe501(• hugetlb) group in /etc/grouptogiveuserspermission to attach to the shared memory segment.
JBoss sales and InquIrIes NORTHAMERICA
1–888–REDHAT1 www.jboss.com
Copyright©2009RedHat,Inc.RedHat,RedHatEnterpriseLinux,theShadowmanlogo,JBoss,MetaMatrix,and RHCEaretrademarksofRedHat,Inc.,registeredintheU.S.andothercountries.Linux® is the registered trademark ofLinusTorvaldsintheU.S.andothercountries.
www.jboss.com # 1388350_1109