Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 1
© Copyright 2007, Dancing Cloud Services
Management, Monitoring, and Deployment of Tomcat and JBoss
Simon [email protected]
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 2
© Copyright 2007, Dancing Cloud Services
Agenda
Installation and startup
Management
Web console elements
System and use of MBeans for monitoring/control
J2EE Domain for application monitoring
JMX Command-line access
Monitoring
Graphs, snapshots, and monitors
Connecting JConsole
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 3
© Copyright 2007, Dancing Cloud Services
Agenda
Deployment
Removal/reconfiguration of some services:
Web-console/JMX
HTTP Invokers
Class downloader
Basic clustering of JBoss
Basic load balancing with Apache
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 4
© Copyright 2007, Dancing Cloud Services
Install & Startup
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 5
© Copyright 2007, Dancing Cloud Services
Download/Install
Jboss.com -> Downloads -> Community downloads -> JBoss
application server -> 4.2.1.GA Download -> (sourceforge) jboss-4.2.1.GA.zip
Unzip
(Older versions require JAVA_HOME)
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 6
© Copyright 2007, Dancing Cloud Services
Execute
Execute jboss-4.2.1.GA/bin/run.sh (run.bat)
bind to appropriate IP address
Browse to MyHost:8080/
Verify inbound firewall access to
HTTP/HTTPS (or 8080 by default)
Might require other firewall holes for other
services
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 7
© Copyright 2007, Dancing Cloud Services
Default Homepage
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 8
© Copyright 2007, Dancing Cloud Services
Management
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 9
© Copyright 2007, Dancing Cloud Services
Web Console
Java applet at:myserver:port/web-console
Install java plugin in your browser
Typically link or copy:
$JAVA_HOME/jre/plugin/i386/ns7/libjavaplugin_oji.so
to your plugins directory, perhaps:
/usr/lib/browser-plugins
.../firefox/plugins
(or whatever)
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 10
© Copyright 2007, Dancing Cloud Services
Web Console
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 11
© Copyright 2007, Dancing Cloud Services
Web Console Views
System
[Monitoring]
J2EE Domains
[AOP]
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 12
© Copyright 2007, Dancing Cloud Services
System View
JMX Beans
Provide information or control
Attributes may be read/write
Operations may be invoked
Grouped by “Domain Name”
e.g. jboss.system
Bean has a “type” name
e.g. type=Server and type=ServerInfo
Subset of this information and control is provided by the JMX console
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 13
© Copyright 2007, Dancing Cloud Services
ServerInfo JMX Bean
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 14
© Copyright 2007, Dancing Cloud Services
Server JMX Bean
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 15
© Copyright 2007, Dancing Cloud Services
Useful MBeans and Elements
jboss.system:type=Server
JVM, Build, Version info
shutdown, runGarbageCollector operations
jboss.system:type=ServerInfo
Thread count, IP address, processors, memory/free memory, JVM, OS version
listThreadCpuUtilization, listThreadDump,
listMemoryPools
jboss:service=JNDIView
list
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 16
© Copyright 2007, Dancing Cloud Services
Useful MBeans and Elements
jboss.jca:service=ManagedConnectorPool,
name=DataSourceName
InUseConnectionCount
MinSize
ConnectionCreatedCount
MaxSize
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 17
© Copyright 2007, Dancing Cloud Services
J2EE Domains
Observe individual applications
Deployment descriptor
Number of calls to servlet/jsp
Calls to EJB methods
Max/min/count of EJBs method-ready
Minimum/maximum response time
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 18
© Copyright 2007, Dancing Cloud Services
Command Line Access
bin/twiddle.sh \--server=MyServer:1099 get \jboss.system:type=ServerInfo
Assumes port 1099 is open
Can also invoke operations:
bin/twiddle.sh -s Christine invoke \ jboss.system:type=Server shutdown
Clearly, none of this is safe for remote access
Block port 1099 by firewall
But HTTP Invokers bypass this
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 19
© Copyright 2007, Dancing Cloud Services
Connecting Jconsole
JConsole might provide advantages over web console
e.g. more detailed memory investigation
Enable this in the bin/run.conf file or edit run.bat
Add:
-Dcom.sun.management.jmxremote
Note: in run.conf, be sure to get this inside the quote marks
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 20
© Copyright 2007, Dancing Cloud Services
Monitoring
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 21
© Copyright 2007, Dancing Cloud Services
Graphing an MBean Property
Find the MBean's domain in:
Web Console
System
JMX Mbeans
Expand the domain, then expand the bean
Right click for context menu
Select graph
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 22
© Copyright 2007, Dancing Cloud Services
MBean Graph
Select jboss.system:type=ServerInfo and graph TotalMemory
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 23
© Copyright 2007, Dancing Cloud Services
Graphables
Graphs can be drawn for numeric bean attributes
e.g. memory use/free, connection count
Even silly ones like port numbers
You can add your own MBeans and graph them too
Be careful interpreting Free Memory
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 24
© Copyright 2007, Dancing Cloud Services
Snapshots
Graphs are great, but they go away
Snapshots collect data for later use
Right click on a numeric data item in an MBean
Select Create Snapshot
Fill in the time interval in milliseconds
Press Create
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 25
© Copyright 2007, Dancing Cloud Services
Snapshot Preparation
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 26
© Copyright 2007, Dancing Cloud Services
Snapshot Use
After creation Start Snapshot
Later, Graph Dataset
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 27
© Copyright 2007, Dancing Cloud Services
Snapshot Recovery
Can also control snapshot and retrieve data using the MBean created for the
snapshot
Click on JMXBeans, search for
jboss.snapshot:*
Select relevant bean to start, stop, and
retrieve data as text
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 28
© Copyright 2007, Dancing Cloud Services
Monitoring
In addition to graphs, you can configure
notifications that will advise if a numeric
attribute changes in particular ways
Right click on the attribute, Create Monitor
Provide a name, threshold value, period, and
enable the monitor
Ensure the period won't miss a spike
Don't forget to enable it
Console Alert sends message to log
You can arrange to send email
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 29
© Copyright 2007, Dancing Cloud Services
Create Monitor
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 30
© Copyright 2007, Dancing Cloud Services
Deployment
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 31
© Copyright 2007, Dancing Cloud Services
Protecting Services
Some services should be password protected
Others should be shut down or reconfigured
Web console, JMX services
HTTP Invoker
Class download service
Database choice
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 32
© Copyright 2007, Dancing Cloud Services
Remove Console Services
JMX console:
Remove (default)/deploy/jmx-console.war
Also disables JMX/MBean functions in Web
console.
For web console, remove: (default)/deploy/management/console-mgr.sar
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 33
© Copyright 2007, Dancing Cloud Services
Password Protection
Alternative to removing, require login
edit (default)/deploy/jmx-console.war/WEB-INF/web.xml
uncomment provided <security-
constraint> element
<login-config> and <security-role> elements are already provided
edit (default)/deploy/jmx-console.war/WEB-INF/jboss-web.xml
uncomment provided <security-domain>
element
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 34
© Copyright 2007, Dancing Cloud Services
Security Constraint
<security-constraint> <web-resource-collection> <web-resource-name>HtmlAdaptor</web-resource-name> <description> An example security config that only allows users with the role JBossAdmin to access the HTML JMX console web application </description> <url-pattern>/*</url-pattern> <http-method>GET</http-method> <http-method>POST</http-method> </web-resource-collection> <auth-constraint> <role-name>JBossAdmin</role-name> </auth-constraint></security-constraint>
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 35
© Copyright 2007, Dancing Cloud Services
Password Change
By default, security is referred, by jboss-web.xml, to a policy named
“jmx-console”
Defined in (default)/conf/login-config.xml
as property file based
Inevitably, default user/password are well known (admin/admin!)
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 36
© Copyright 2007, Dancing Cloud Services
Username Password Files
Create/change user=password entries in the file:
(default)/conf/props/jmx-console-users.properties
Ensure matching user=role entry exists in file:
(default)/conf/props/jmx-console-roles.properties
Role required in security constraint was
JBossAdmin
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 37
© Copyright 2007, Dancing Cloud Services
HTTP Invoker
Used for tunneling when poor firewall administration forces it
Best removed if not needed
Simply remove http-invoker.sar from the deployment directory
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 38
© Copyright 2007, Dancing Cloud Services
Class Download Service
Used for dynamic loading of stubs,
polymorphic argument types, etc.
Powerful RMI feature permits object oriented
distributed computing
Not just object-based
But largely unnecessary if classes are pre-
installed on all participating systems
Allowing attackers to download your classes,
or worse, resources, is bad
Resources include the username/password files!
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 39
© Copyright 2007, Dancing Cloud Services
Removing Class Download Service
Edit conf/jboss-service.xml
Remove the entry:
<mbean
code=”org.jboss.web.WebService”name=”jboss:service=WebService>
...
</mbean>
Editing the contents can provide more
control if you don't want to disable class downloading completely
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 40
© Copyright 2007, Dancing Cloud Services
Basic JBoss Cluster
Essentially provided out of the box
Run the “all” configuration
Multiple machines will cluster
For demo, run on single machine
Create duplicate configuration called “all2” or similar
Arrange to use alternate port numbers (base + 100)
In conf/jboss-service.xml uncomment the element<mbean code=”org.[...].ServiceBindingManager”[...]
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 41
© Copyright 2007, Dancing Cloud Services
Session Replication
Provided by default
Be sure to mark application as
<distributable/> to enable replication
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 42
© Copyright 2007, Dancing Cloud Services
Load Balancing
Use Apache as front end load balancer
Don't forget that LB is single point of
failure — monitor it carefully and treat it well!
Install mod_jk-apache.?.??.?.so into apache2/modules
Configure the jk module
LoadModule jk_module modules/mod_jk-apache-2.0.55.so
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 43
© Copyright 2007, Dancing Cloud Services
Configure jk module
# Locate workers.propertiesJkWorkersFile conf/workers.properties# Location of jk logsJkLogFile logs/mod_jk.log# jk log levelJkLogLevel info# log formatJkLogStampFormat "[%a %b %d %H:%M:%S %Y] "# JkOptions indicate to send SSL KEY SIZE, JkOptions +ForwardKeySize +ForwardURICompat\ -ForwardDirectories# JkRequestLogFormat set the request format JkRequestLogFormat "%w %V %T"JkMount /mycluster/* mybalancerJkMount /mycluster mybalancer
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 44
© Copyright 2007, Dancing Cloud Services
Configure Workers
ajp13worker.list=mybalancerworker.mybalancer.type=lbworker.mybalancer.balance_workers=worker1,worker2worker.mybalancer.sticky_session=0(ajp13)worker.worker1.type=ajp13worker.worker1.host=localhostworker.worker1.port=8009worker.worker1.lbfactor=1(ajp13)worker.worker2.type=ajp13worker.worker2.host=localhostworker.worker2.port=8109worker.worker2.lbfactor=1
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 45
© Copyright 2007, Dancing Cloud Services
Configure Apache
Edit apache2/conf/httpd.conf
In the LoadModule block, insert:
Include conf/jk.conf
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 46
© Copyright 2007, Dancing Cloud Services
Summary
Installation and startup
MBeans via console and JMX command line
J2EE Domain
Graphs, snapshots, and monitors
Connecting JConsole
Removal/reconfiguration of services
Basic clustering and load balancing
Colorado Software Summit: October 21 – 26, 2007
Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 47
© Copyright 2007, Dancing Cloud Services
Management, Monitoring, and Deployment of Tomcat and JBoss
Simon [email protected]