Embed Size (px)
Citation preview
Genic Data Manager
Configuration, Monitoring and Support Guide
2
Information in this document is subject to change without notice. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without the express written permission of DataGenic Limited. © Copyright 2013 DataGenic, all rights reserved.
VERSIONING Date Version Author Changes 26-April-13 V1.0 Juan Pedraja Initial Version 29-April-13 V1.1 Juan Pedraja Added for Max Num of Cursors errors in Oracle 30-April-13 V1.2 Juan Pedraja Corrected DEFAULT_TIMEOUT default setting.
Added details on GDM_SYSTEM_PROPERTY Added on rolling server logs
2-May-13 V1.3 Juan Pedraja Added line for JMS Heuristic Exceptions in JBoss 21-May-13 V1.4 Juan Pedraja Added details on Kerberos logging 07-June-13 V1.5 Juan Pedraja Added details on JMS queue configuration parameters 08-Oct-13 V1.6 Juan Pedraja Added details on thread dumps on Linux. And disable
auditing for gdmxdataloading 25-Nov-14 V1.7 Juan Pedraja Added details on Twiddle , logging transaction activity , and
EJB set up
CONTACT DETAILS DataGenic Ltd Juan Pedraja
46 New Broad Street London Mobile: +34 654 26 01 65 London Office: +44 (0)203 651 5567 Email: [email protected]
3
Contents 1 Introduction .................................................................................................................................... 5
2 Configuration .................................................................................................................................. 5
2.1 Oracle Settings ........................................................................................................................ 5
2.2 JVM settings ............................................................................................................................ 5
2.3 JDBC Connection Pools ........................................................................................................... 7
2.4 JMS configuration ................................................................................................................... 8
2.5 JTA settings ........................................................................................................................... 11
2.6 GDM settings ......................................................................................................................... 11
2.6.1 Disable Auditing for Technical Users ............................................................................ 12
2.6.2 Server Settings .............................................................................................................. 12
2.6.3 SERVER Locations .......................................................................................................... 13
2.7 Logging .................................................................................................................................. 14
2.7.1 Add Thread to log file .................................................................................................... 15
2.7.2 Debugging Transaction related exceptions in JBoss ..................................................... 16
3 Exposing EJBs in JBoss ................................................................................................................... 17
4 Monitoring .................................................................................................................................... 18
4.1 JConsole ................................................................................................................................ 18
4.1.1 Overview ....................................................................................................................... 18
4.1.2 Memory ......................................................................................................................... 19
4.1.3 Threads .......................................................................................................................... 20
4.1.4 Thread Dumps ............................................................................................................... 20
4.1.5 VM Summary ................................................................................................................. 21
4.1.6 MBeans ......................................................................................................................... 21
4.1.7 Heap Analysis ................................................................................................................ 22
4.2 JRockit ................................................................................................................................... 23
4.3 Thread dumps using the JBoss console ................................................................................ 23
4.4 Thread dumps in Linux .......................................................................................................... 25
5 JMS Message monitoring .............................................................................................................. 26
5.1 Listing JMS messages using the JBoss Console ..................................................................... 26
5.2 Pausing JMS Queues ............................................................................................................. 28
6 Diagnosing a problem ................................................................................................................... 28
6.1 Initial Investigation ................................................................................................................ 28
6.1.1 ORA-01000: maximum open cursors exceeded ............................................................ 28
6.2 Reporting a problem ............................................................................................................. 29
4
6.2.1 Performing an Oracle export ........................................................................................ 29
5
1 Introduction The purpose of this document is to provide an overview of the main configuration parameters that affect a Genic Data Manager (GDM 4.x) server installation. It covers the main JVM, Application and GDM settings to verify when investigating performance and stability in GDM and it includes overview of the tools that can be used to diagnose issues. It also outlines the different steps that need to be performed before contacting DataGenic support and the information that needs to be collected and sent as a part of a support case. The focus of this document is the GDM server running inside a single application server instance. Configuration and Support for database, network, clustering or load balancing, DataHub connectivity or data processing are out of scope for this document. For more details on other GDM server topics check out the wiki pages at http://www.datagenicgroup.com/wiki/index.php/Genic_DataManager_-_Server_Installation_Guide
2 Configuration The configuration parameters outlined here are just generic indications that will be affected by a number of customer-site specific items including; no. of users, speed of network, hardware used, data feeds taken and should be scaled accordingly by the Database and System Administrators of your own network. These settings are targeted to test/integration/production systems. GDM instances used for development or basic testing can be configured to used much fewer resources.
2.1 Oracle Settings The most important Oracle settings are listed below Setting Description ALTER SYSTEM SET open_cursors = 1000 SCOPE=BOTH; It sets the OPEN_CURSORS parameters to 1000
which is the minimum for a GDM installation
2.2 JVM settings The most important parameters to specify are listed below JVM setting Description JVM -server It indicates that the server version
of the JVM should be used. Only x64 JVM should be used.
Sun only
-Xms<nn>g -Xmx<nn>g
Minimum and maximum heap sizes. While a GDM instance used for development can possibly work with 1-2GB of RAM. Any integration or production instance should have at least 4GB and
Sun and JRockit
6
ideally 8 GB of RAM. Both minimum and maximum settings should be set to the same size to ensure all the memory is allocated at start up. Heap sizes should be sensible to ensure that there is enough RAM in the system to avoid OS swapping. These settings cover only the heap size, the total memory footprint of the java process will be about 400-500 MB larger.
-XX:PermSize=256m -XX:MaxPermSize=512m
Minimum and Maximum memory space allocated to store class files.
Sun and JRockit
-XXtlasize:128k -XXlargeobjectlimit:128k It controls the Thread local area size used for object allocation
JRockit only
-XX:-OmitStackTraceInFastThrow It ensures that the stack trace is not removed from an exception. It helps in diagnosing production problems
Sun only
-Xmanagement:port=8456 It enables the JVM monitoring options and listens on port 8456. See Monitoring section for more details.
JRockit only
-XX:+HeapDumpOnOutOfMemoryError It produces a dump of the contents of the heap in case there is an OutOfMemory exception thrown by the JVM. The file is called java_<pid>.hprof in the working directory of the VM. You can specify an alternative file name or directory with the -XX:HeapDumpPath=
Sun only
-Djava.net.preferIPv4Stack=true For compatibility Sun and JRockit -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8456 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.remote.ssl=false
It enables JVM remote monitoring options and listens on port 8456. See Monitoring section for more details.
Sun only
-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
It enables the JVM in debug mode. Not to be used for production.
Sun and JRockit
-Dcom.datagenicgroup.gdm.DEFAULT_TIMEOUT=3600000 This property configures the amount of time before timing out when making a call to a web service. Default is 5 minutes but for some long running operations it might have to be increased
Sun and JRockit
7
2.3 JDBC Connection Pools GDM uses extensively the database and this is one of the main candidates for tuning depending on the usage of the application. There are 2 pools configured in the application server for use for GDM. JDBC Pool Description Connections datagenic.jdbc.gdmDS This is the transactional
datasource in GDM used in most of the operations.
It should have about 200 connections per instance and minimum pool size should be set to 100 and maximum sizes should be set to 200 to avoid increasing and shrinking the pool.
datagenic.jdbc.templateDS This is the non-transactional datasource in GDM used in most of the operations.
It should have about 150 connections per instance and minimum pool size should be set to 100 and maximum size should be set to 200 to avoid increasing and shrinking the pool.
The configuration for these 2 pools Oracle and JBoss and Weblogic is shown below Pool JBoss WLS datagenic.jdbc.gdmDS
<local-tx-datasource> <jndi-name>datagenic.jdbc.gdmDS</jndi-name> <connection-url>jdbc:oracle:thin:@hostname:1521:SID</connection-url> <driver-class>oracle.jdbc.driver.OracleDriver</driver-class> <user-name>gdm</user-name> <password>gdm</password> <min-pool-size>200</min-pool-size> <max-pool-size>200</max-pool-size> <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter </exception-sorter-class-name> <new-connection-sql>select sysdate from dual</new-connection-sql> <check-valid-connection-sql>select count(*) from dual</check-valid-connection-sql> <metadata> <type-mapping>Oracle9i</type-mapping> </metadata> </local-tx-datasource>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/jdbc-data-source http://xmlns.oracle.com/weblogic/jdbc-data-source/1.0/jdbc-data-source.xsd"> <name>datagenic.jdbc.gdmDS</name> <jdbc-driver-params> <url>jdbc:oracle:thin:@hostname:1521:GDM</url> <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name> <properties> <property> <name>user</name> <value>rwe4</value> </property> </properties> <password-encrypted>{AES}iFsGn+UZ8a0N9mIlGwIo9bREYrodSuutdeb175wMiN8=</password-encrypted> <use-xa-data-source-interface>true</use-xa-data-source-interface> </jdbc-driver-params> <jdbc-connection-pool-params> <initial-capacity>200</initial-capacity> <max-capacity>200</max-capacity> <capacity-increment>1</capacity-increment> <shrink-frequency-seconds>0</shrink-frequency-seconds> <highest-num-waiters>2147483647</highest-num-waiters> <connection-creation-retry-frequency-seconds>0</connection-creation-retry-frequency-seconds> <connection-reserve-timeout-seconds>10</connection-reserve-timeout-seconds> <test-frequency-seconds>120</test-frequency-seconds> <test-connections-on-reserve>true</test-connections-on-reserve> <ignore-in-use-connections-enabled>true</ignore-in-use-connections-enabled> <inactive-connection-timeout-seconds>0</inactive-connection-timeout-seconds> <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name> <login-delay-seconds>0</login-delay-seconds> <init-sql>SQL SELECT 1 FROM DUAL</init-sql> <statement-cache-size>10</statement-cache-size> <statement-cache-type>LRU</statement-cache-type> <remove-infected-connections>true</remove-infected-connections> <seconds-to-trust-an-idle-pool-connection>10</seconds-to-trust-an-idle-pool-connection> <statement-timeout>-1</statement-timeout> <jdbc-xa-debug-level>10</jdbc-xa-debug-level> <pinned-to-thread>false</pinned-to-thread> </jdbc-connection-pool-params> <jdbc-data-source-params> <jndi-name>datagenic.jdbc.gdmDS</jndi-name> <global-transactions-protocol>TwoPhaseCommit</global-transactions-protocol> </jdbc-data-source-params> <jdbc-xa-params> <keep-xa-conn-till-tx-complete>true</keep-xa-conn-till-tx-complete> <need-tx-ctx-on-close>false</need-tx-ctx-on-close> <xa-end-only-once>false</xa-end-only-once> <keep-logical-conn-open-on-release>false</keep-logical-conn-open-on-release> <resource-health-monitoring>true</resource-health-monitoring> <recover-only-once>false</recover-only-once> <xa-set-transaction-timeout>false</xa-set-transaction-timeout> <xa-transaction-timeout>0</xa-transaction-timeout>
8
<rollback-local-tx-upon-conn-close>true</rollback-local-tx-upon-conn-close> <xa-retry-duration-seconds>0</xa-retry-duration-seconds> <xa-retry-interval-seconds>60</xa-retry-interval-seconds> </jdbc-xa-params> </jdbc-data-source>
datagenic.jdbc.templateDS <no-tx-datasource> <jndi-name>datagenic.jdbc.templateDS</jndi-name> <connection-url>jdbc:oracle:thin:@hostname:1521:GDM</connection-url> <driver-class>oracle.jdbc.driver.OracleDriver</driver-class> <user-name>gdm</user-name> <password>gdm</password> <min-pool-size>150</min-pool-size> <max-pool-size>150</max-pool-size> <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter </exception-sorter-class-name> <new-connection-sql>select sysdate from dual</new-connection-sql> <check-valid-connection-sql>select count(*) from dual</check-valid-connection-sql> <metadata> <type-mapping>Oracle9i</type-mapping> </metadata> </no-tx-datasource>
<jdbc-data-source xmlns="http://www.bea.com/ns/weblogic/90" xmlns:sec="http://www.bea.com/ns/weblogic/90/security" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wls="http://www.bea.com/ns/weblogic/90/security/wls" xsi:schemaLocation="http://www.bea.com/ns/weblogic/920 http://www.bea.com/ns/weblogic/920.xsd"> <name>datagenic.jdbc.templateDS</name> <jdbc-driver-params> <url>jdbc:oracle:thin:@hostname:1521:GDM</url> <driver-name>oracle.jdbc.OracleDriver</driver-name> <properties> <property> <name>user</name> <value>gdm</value> </property> </properties> <password-encrypted>{3DES}VM20XjWN8Hs=</password-encrypted> </jdbc-driver-params> <jdbc-connection-pool-params> <initial-capacity>150</initial-capacity> <max-capacity>150</max-capacity> <test-connections-on-reserve>true</test-connections-on-reserve> <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name> <init-sql>SQL SELECT 1 FROM DUAL</init-sql> </jdbc-connection-pool-params> <jdbc-data-source-params> <jndi-name>datagenic.jdbc.templateDS</jndi-name> <global-transactions-protocol>None</global-transactions-protocol> </jdbc-data-source-params>
2.4 JMS configuration GDM uses extensively JMS as a way of decoupling the processing of some of the services such as auditing, quality management rules and notifications from the main client calling threads. Queue Description datagenic.jms.AUDIT_QUEUE It stores all the auditing
messages recording GDM activity. These messages are eventually inserted to the AUDIT_INFO table
datagenic.jms.EVENT_QUEUE All notifications produced by GDM activity are stored in this queue before being set to the subscribers
datagenic.jms.CURVE_BUILDER All curves scheduled to be built are queued for processing
datagenic.jms.DATA_CONTENT All imports through ImportExportActions result in entries for the Data Monitoring dashboard
datagenic.jms.DATA_QUALITY_MANUAL All manual quality checks are stored in this queue for asynchronous execution
datagenic.jms.DATA_QUALITY_AUTO All post update quality checks are stored in this queue for asynchronous execution
datagenic.jms.QUERY All queries are stored in this queue for asynchronous execution
datagenic.jms.DefaultQueue This is the default Queue used to store any messages that failed to be processed. It
9
requires manual intervention These JMS queues are persistent and store their en-queued messages in the database. If a message fails to be processed, it is put back in the queue for and reprocessed. The time between re-processing and the number of times it attempts to process it are highlighted in yellow below. The number of messages you want to keep in memory can be configured for each Queue using the following parameters: Parameter name Description Default value FullSize the maximum number of messages held by the
queue or topic subscriptions in memory at any one time. The actual queue or subscription can hold many more messages than this but these are paged to and from storage as necessary as messages are added or consumed.
PageSize
When loading messages from the queue or subscription this is the maximum number of messages to pre-load in one operation.
DownCacheSize When paging messages to storage from the queue they first go into a "Down Cache" before being written to storage. This enables the write to occur as a single operation thus aiding performance. This setting determines the max number of messages that the Down Cache will hold before they are flushed to storage.
So for instance , if you want to ,limit the size of the dlq, you could alter the %JBOSS_HOME%/deploy/jms/jbossmq-service.xml and add <!-- Dead Letter Queue --> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=DLQ"> <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="fullSize">10</attribute> <attribute name="pageSize">10</attribute> <attribute name="downCacheSize">10</attribute> </mbean> It would mean that the Jboss start-up should be a lot faster since it won’t load any dead messages in memory The configuration and database storage for these queues in JBoss and Weblogic is shown below. Application Server Configuration Database storage JBoss <mbean code="org.jboss.mq.server.jmx.Queue"
name="jboss.mq.destination:service=Queue,name=datagenic.jms.DATA_QUALITY_AUTO"> <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="JNDIName">datagenic.jms.DATA_QUALITY_AUTO</attribute> <attribute name="RedeliveryDelay">10000</attribute> <attribute name="RedeliveryLimit">3</attribute> </mbean> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=datagenic.jms.DATA_QUALITY_MANUAL"> <depends optional-attribute-
JMS_MESSAGES
10
name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="JNDIName">datagenic.jms.DATA_QUALITY_MANUAL</attribute> <attribute name="RedeliveryDelay">10000</attribute> <attribute name="RedeliveryLimit">3</attribute> </mbean> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=datagenic.jms.AUDIT_QUEUE"> <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="JNDIName">datagenic.jms.AUDIT_QUEUE</attribute> <attribute name="RedeliveryDelay">10000</attribute> <attribute name="RedeliveryLimit">3</attribute> </mbean> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=datagenic.jms.EVENT_QUEUE"> <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="JNDIName">datagenic.jms.EVENT_QUEUE</attribute> <attribute name="RedeliveryDelay">10000</attribute> <attribute name="RedeliveryLimit">3</attribute> </mbean> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=datagenic.jms.CURVE_BUILDER"> <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="JNDIName">datagenic.jms.CURVE_BUILDER</attribute> <attribute name="RedeliveryDelay">10000</attribute> <attribute name="RedeliveryLimit">3</attribute> </mbean> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=datagenic.jms.DATA_CONTENT"> <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="JNDIName">datagenic.jms.DATA_CONTENT</attribute> <attribute name="RedeliveryDelay">10000</attribute> <attribute name="RedeliveryLimit">3</attribute> </mbean> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=datagenic.jms.QUERY"> <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="JNDIName">datagenic.jms.QUERY</attribute> <attribute name="RedeliveryDelay">10000</attribute> <attribute name="RedeliveryLimit">3</attribute> </mbean> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=datagenic.jms.DefaultQueue"> <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends> <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager</depends> <attribute name="JNDIName">datagenic.jms.DefaultQueue</attribute> </mbean>
WLS <weblogic-jms xmlns="http://xmlns.oracle.com/weblogic/weblogic-jms" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-jms http://xmlns.oracle.com/weblogic/weblogic-jms/1.0/weblogic-jms.xsd"> <queue name="datagenic.jms.DATA_QUALITY_AUTO"> <sub-deployment-name>GDMQueues</sub-deployment-name> <delivery-params-overrides> <redelivery-delay>10000</redelivery-delay> </delivery-params-overrides> <delivery-failure-params> <error-destination>datagenic.jms.DefaultQueue</error-destination> <redelivery-limit>10</redelivery-limit> <expiration-policy>Discard</expiration-policy> </delivery-failure-params> <message-logging-params> <message-logging-enabled>true</message-logging-enabled> <message-logging-format>%header%,%properties%,%body%</message-logging-format> </message-logging-params> <jndi-name>datagenic.jms.DATA_QUALITY_AUTO</jndi-name> </queue> <queue name="datagenic.jms.DATA_QUALITY_MANUAL"> <sub-deployment-name>GDMQueues</sub-deployment-name> <jndi-name>datagenic.jms.DATA_QUALITY_MANUAL</jndi-name> </queue> <queue name="datagenic.jms.AUDIT_QUEUE"> <sub-deployment-name>GDMQueues</sub-deployment-name> <jndi-name>datagenic.jms.AUDIT_QUEUE</jndi-name> </queue> <queue name="datagenic.jms.EVENT_QUEUE"> <sub-deployment-name>GDMQueues</sub-deployment-name> <jndi-name>datagenic.jms.EVENT_QUEUE</jndi-name> </queue> <queue name="datagenic.jms.CURVE_BUILDER"> <sub-deployment-name>GDMQueues</sub-deployment-name> <jndi-name>datagenic.jms.CURVE_BUILDER</jndi-name> </queue> <queue name="datagenic.jms.DATA_CONTENT"> <sub-deployment-name>GDMQueues</sub-deployment-name> <jndi-name>datagenic.jms.DATA_CONTENT</jndi-name> </queue> <queue name="datagenic.jms.QUERY"> <sub-deployment-name>GDMQueues</sub-deployment-name> <jndi-name>datagenic.jms.QUERY</jndi-name> </queue> <queue name="datagenic.jms.DefaultQueue"> <sub-deployment-name>GDMQueues</sub-deployment-name> <jndi-name>datagenic.jms.DefaultQueue</jndi-name>
WLSTORE
11
</queue> </weblogic-jms>
For JBoss installations and in order to avoid the noisy HeuristicMixedException messages, the following line has to be added to %JBOSS_HOME%/server/<SERVER_NAME>/jms/jms-ds.xml
2.5 JTA settings The transaction timeout needs to be increased to 7200 seconds to cater for some long running transactions in GDM. AppServer Configuration File WLS <jta>
<timeout-seconds>7200</timeout-seconds> <abandon-timeout-seconds>86400</abandon-timeout-seconds> <forget-heuristics>true</forget-heuristics> <before-completion-iteration-limit>10</before-completion-iteration-limit> <max-transactions>10000</max-transactions> <max-unique-name-statistics>1000</max-unique-name-statistics> <checkpoint-interval-seconds>300</checkpoint-interval-seconds> <unregister-resource-grace-period>30</unregister-resource-grace-period> </jta>
$DOMAIN_HOME/config/config.xml
JBoss <!-- JBoss Transactions JTA --> <mbean code="com.arjuna.ats.jbossatx.jta.TransactionManagerService" name="jboss:service=TransactionManager"> <attribute name="TransactionTimeout">7200</attribute> <attribute name="ObjectStoreDir">${jboss.server.data.dir}/tx-object-store</attribute> </mbean>
$JBOSS_HOME/server/gdm/conf/ jboss-service.xml
Notice that in a JBoss cluster environment %JBOSS_HOME%/server/gdm/cnf/ jbossjta-properties.xml needs to uniquely identify each node cluster node. That is in a cluster environment, the value of com.arjuna.ats.arjuna.xa.nodeIdentifier must be unique. By default all nodes in the cluster have the same “1” value. It is recommended that one is set to “2” (to indicate the second node), “3” for the third node etc.
2.6 GDM settings There are certain parameters within the GDM application that also can affect the behaviour of the GDM server:
12
2.6.1 Disable Auditing for Technical Users By default all user actions in GDM are audited automatically, imposing an overhead that becomes quite significant in the case of technical users. There are certain tasks in GDM such as data loading from the data hub or internal source or export jobs that are quite performed by a specially configured GDM user. DataGenic recommends that those users have auditing disables by going to Server Administration and selecting the technical user and clicking on disable Auditing.
2.6.2 Server Settings The Server Administration Server settings allows configuration of some of the manual maintenance tasks such as data portioning, number of quality checks to store, etc.
13
2.6.3 SERVER Locations GDM has the option to deploy all components (workflow, curve builder) in the same application server or split across instances to distribute the los. The GDM_SYSTEM_PROPERTY in the database points to what all the components are available. If the URLs are Name Description Example GDM_SERVER_URL The listening port and address
where the genicDM.ear. It is used for internal DG product communication.
http://localhost:19003
GIQ_SERVER_URL The listening port and address where the genicDM.ear is available. It is used for internal DG product communication
http://localhost:19003
GCB_SERVER_URL The listening port and address where the genicCB.ear is available. It is used for internal DG product communication
http://localhost:19003
GWF_SERVER_URL The listening port and address where the genicWorkflow.ear is available. It is used for internal DG product communication
http://localhost:19003
GWS_SERVER_URL The listening port and address where the genicWebSocket.ear is available. It is used for internal
http://localhost:19003
14
DG product communication WEBSOCKET_SERVER_URL The listening port and address of
the where the genicWebSocket.ear is available. It is used for internal DG web socket publishing. Notice the ws for the protocol
ws://localhost:19003
GWF_CLIENT_URL The listening port and address where the genicWorkflow.ear is available to the client. If a hardware load balancer or web server or a DNS address is used for client access. That is what should be configured here
http:// gdm.intranet.customer.com
GIQ_CLIENT_URL The listening port and address where the genicDM.ear is available to the client. If a hardware load balancer or web server or a DNS address is used for client access. That is what should be configured here
http://gdm.intranet.customer.com
GCB_CLIENT_URL The listening port and address where the genicCB.ear is available to the client. If a hardware load balancer or web server or a DNS address is used for client access. That is what should be configured here
http://localhost:19003
WEBSOCKET_CLIENT_URL The listening port and address where the genicWebsocket.ear is available to the client. If a hardware load balancer or web server or a DNS address is used for client access. That is what should be configured here
ws://ws. gdm.intranet.customer.com
2.7 Logging GDM utilises Apache Log4j, a Java-based logging library (http://log4j.apache.com) for all logs. The main configuration file is log4j.properties in weblogic or %JBOSS_HOME%/server/gdm/conf/jboss-log4j.xml and allows to configure the location of the files, the log level for each categories. The main categories for GDM are com.datagenic and org.hibernate and DataGenic recommends that are set to INFO for production use. They can be enabled to DEBUG for a short period of time to help diagnose a problem App Server Location WLS servername.log, access.log, servername.out at
\\APPSERVER/$DOMAIN/servers/<servername>/logs/ JBoss log files are boot.log, server.log or run.log at
15
\\APPSERVER/$JBOSS_HOME/servers/gdm/logs/ JBoss ships with some classes that generate a lot of log entries with GDM. DG recommends that these loggers are silenced by adding the following entries to the jboss-log4j.xml <!-- ======================= --> <!-- Silence some noisy Loggers --> <!-- ======================= --> <appender name="NULL" class="org.apache.log4j.varia.NullAppender"/> <category name="org.hibernate.cache.ReadWriteCache"> <priority value="ERROR"/> </category> <category name="org.jboss.wsf.stack.jbws.RequestHandlerImpl"> <priority value="FATAL"/> <appender-ref ref="NULL"></appender-ref> </category> <category name="org.jboss.ws.core.soap.EnvelopeBuilderDOM"> <priority value="ERROR"/> </category> <category name="org.hibernate.event.def.DefaultDeleteEventListener"> <priority value="WARN"/> </category> <category name="com.arjuna.ats.arjuna.logging.arjLoggerI18N "> <priority value="FAT"/> <appender-ref ref="NULL"></appender-ref> </category>
In order to avoid large log files, DG recommends that log4j is configured to roll the log file on a daily basis. See below for Jboss configuration <!-- A time/date based rolling appender --> <appender name="FILE" class="org.jboss.logging.appender.DailyRollingFileAppender"> <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/> <param name="File" value="${jboss.server.log.dir}/server.log"/> <param name="Append" value="true"/> <!-- Rollover at midnight each day --> <param name="DatePattern" value="'.'yyyy-MM-dd"/> <layout class="org.apache.log4j.PatternLayout"> <!-- The default pattern: Date Priority [Category] Message\n --> <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/> </layout> </appender>
If there are problems with the SingleSignOn (SSO) configuration, you can edit the %JBOSS_HOME%\server\gdm\krb5login.conf or %WLS_HOME%\user_projects\domains\myDomain\krb5login.conf to enable debug (see highlighted in bold) com.sun.security.jgss.krb5.accept { com.sun.security.auth.module.Krb5LoginModule required principal="HTTP/[email protected]" useKeyTab=true keyTab=webserver.keytab storeKey=true isInitiator=false debug=true; };
2.7.1 Add Thread to log file Adding the thread name to the logs makes it easier to tell which log entries belong to what operations when there are multiple executions running at one time. You can enable logging of the current thread by configuring as follows (notice highlight in yellow)
16
2.7.2 Debugging Transaction related exceptions in JBoss If there are lot of Heuristic Exceptions in the server logs. 2014-11-14 14:26:04,498 WARN [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.resources.arjunacore.preparefailed] [com.arjuna.ats.internal.jta.resources.arjunacore.preparefailed] XAResourceRecord.prepare - prepare failed with exception XAException.XAER_RMFAIL 2014-11-14 14:26:04,498 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_50] - Prepare phase of action -60173daf:c072:5465fdf6:1fbf received heuristic decision: TwoPhaseOutcome.HEURISTIC_HAZARD 2014-11-14 14:26:04,499 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_36] - BasicAction.End() - prepare phase of action-id -60173daf:c072:5465fdf6:1fbf failed. 2014-11-14 14:26:04,499 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_37] - Received heuristic: TwoPhaseOutcome.HEURISTIC_HAZARD . 2014-11-14 14:26:04,499 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_38] - Action Aborting
You can enable more logging during the transaction commit phase to work out what the reason for an heuristic exception might. Perform the following changes need to be done
• Enable debug level in jbossjta-properties.xml. As shown below, increase the debug level from 0x00000000 (OFF) to 0xffffffff (ON)
17
• Enable logging in jboss-log4j.xml by adding the following categories
• Add a new Appender to place all the transactions logging in a separate log (transactions log)
3 Exposing EJBs in JBoss In order to expose the GDM EJBs to the outside world, the following steps needs to be implemented.
• Bind the server to the host name and not to localhost or multiple address. That is, modify the %JBOSS_HOME%/bin/service.cmd so it has the –b flag followed by the hostname (see highlighted below) where hostname should be replaced by the IP or DNS name
• Open ports in windows or linux firewalls: 1099 : Naming service port, 1098 : Rmi service port and 3873 : EJB3 Remote
Invoker Port. See screenshot for firewall rule in T01
18
4 Monitoring There are some different tools to monitor the state of a running GDM server
4.1 JConsole Java 6 includes a JMX monitoring console called $JAVA_HOME/bin/JConsole that at start up show you a list of running local JVMs and allow you to attach to one of these by selecting it. Or to any other remote JVMs as long as that remote JVM has been started with the remote JMX options enabled (see section above for JVM settings). Attaching JConsole to a JVM activates the jmx module but the performance overhead is pretty low (about 5%) and it can be used in a production system to diagnose problems.
Once connected you are presented with the following Tabs
4.1.1 Overview The tab showing an overview of the heap memory, threads, classes loaded and CPU usage
19
4.1.2 Memory The memory tab will give an overview of the usage of the heap space and the different areas (Heap, Eden, Non-Heap, Perm Gen, etc.). It also gives the chance to force a full Garbage Collection A normal profile of GDM is shown below, where heap usage increases over time until it reaches a peak that forces a garbage collection to happen. Giving the graph that distinct peaks and valleys profile. Notice that after a full garbage collection the valleys or memory watermark should remain fairly constant. Any increasing over time watermark can point to a memory leak problem
Java 6 allows developers to dump the entire contents of a JVM's heap to a file in a binary (HPROF) format that can be exported and analysed offline
20
4.1.3 Threads The threads tab gives an overview of the activity of the threads in the JVM. Giving you access to the stack trace of each of them
4.1.4 Thread Dumps A different and informative monitoring is thread dumps, especially when the application experiences locking problems or long running operations. By default JConsole does not come with the ability to perform thread dumps. But there is an open source plug-in TDA available at https://java.net/projects/tda/downloads. Download and install in your local machine (e.g. C:\apps\tda) where JConsole is running. This tool is both a standalone analyser, allowing you to analyse thread dumps offline, as well as a plugin for your JDK's JConsole tool. The plugin allows you to take full snapshots of your JVM's thread state. This includes the full stack traces, as well as information about which java monitors a thread has taken, or is waiting for. In order to start JConsole with this new plugin, you must tell it where to find TDA on the command line when you launch JConsole: %JAVA_HOME%/bin/jconsole -pluginpath C:\apps\tda \tda.jar You will notice that JConsole will now have an extra tab called Thread Dumps from where you can perform thread dumps, find long running jobs, save dumps to a log file, etc.
21
4.1.5 VM Summary This TAB will give you an overview of the main parameters of the running JVM
4.1.6 MBeans The tab allows you to interact with the JMX beans installed in the GDM server. Depending on whether you are running inside Weblogic or JBoss you will have access to different beans. One of the standard ones is com.sun.management.HotspotDiagnostic that offers the dumpHeap operation. Now fill in the two fields p0 (this is the full filename of the target dump file - use extension .hprof), and leave p1 set to true (this tells the JVM to only dump referenced objects, and no garbage). Then click on the dumpHeap button. Depending of the size of your heap this may take a couple of seconds. Eventually JConsole will respond by telling you that the method was invoked correctly.
22
4.1.7 Heap Analysis There are many available tools on the web to analyse binary heap dump. One good option is the Eclipse Memory Analyzer (http://www.eclipse.org/mat/). A standalone application based on Eclipse's RCP platform that analyses heap dumps, writes out disk-based indexes on the heap and offers a user interface to rapidly search and analyse the content of the heap dump. Once the heap dump has been loaded the first time, all the necessary indexes are retained, so opening a heap dump a second time is instant.
The Memory Analyzer will present you with an overview of the system, a thread overview, a report of possible leak suspects. You can also drill down into the list of classes, the number of objects in the heap for this type, the amount of bytes taken by these objects, the number of bytes held on the heap from this and only this object type.
23
Check out the MAT web site http://wiki.eclipse.org/index.php/MemoryAnalyzer for more in-depth explanations on how to analyse heap dumps
4.2 JRockit The Oracle JRockit Mission control is a very similar application to JConsole . It is available at as part of the JRockit JDK (http://www.oracle.com/technetwork/middleware/jrockit/downloads/index.html)
4.3 Thread dumps using the JBoss console See online article at https://community.jboss.org/wiki/GenerateAThreadDumpUsingJMXConsoleJBossAS3456 for details on how to generate a thread dump using the built in JBoss jmx console.
24
The JMX console also be accessed through a command line option, so commands can be scripted. The command line utility is called Twiddle (https://developer.jboss.org/wiki/Twiddle) and it is available at %JBOSS_HOME%/bin/twiddle.cmd Please notice that twiddle can be executed from any machine since it uses the http port to access the JBoss server. In order to generate a thread dump of the JVM running the JBoss server follow these steps:
1. Open the JMXConsole (for example: http://localhost:8080/jmx-console) 2. Navigate to jboss.system:type=ServerInfo mbean (hint: you can probably just CTRL-F and enter type=ServerInfo
in the dialog box) 3. Click on the link for the Server Info mbean.
4. Navigate to the bottom where it says "listThreadDump"
5. Click it and get your thread dump
25
Alternatively you can use twiddle to execute the listThreadDump method and pipe the returned HTML directly to file i.e.
%JBOSS_HOME%/bin/twiddle –s localhost:1099 invoke "jboss.system:type=ServerInfo" listThreadDump > threads.html
4.4 Thread dumps in Linux If the JConsole is not available and only access to the command line in Linux is possible. There is an way to analyse thread activity in the JVM. In Linux, every process has its own PID, which is what it is shown with top/ps -ef, etc. In the case of a JVM process, you get one bid PID for the whole GDM server which might tell you that it is consuming 99% CPU but it actually does not break it to which processes within GDM are actually doing work. Fortunately, in Linux/Unix, every Java thread is implemented as lightweight process, which in effect it means that it gets its own PID. So when you do TOP, you get: • By default, all java threads aggregated into a single entry • Press CTRL +H (once in top), you get individual entries for all threads inside that WLS process which allows you to identify whether the CPU load is spread evenly across all running threads OR for instance that 99% CPU is really down to the one thread (i.e. one transaction)
26
If you note down the PID of the individual thread (e.g. 123456 below) in TOP PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND ........... 123456 weblogic 9 0 86616 234M 26780 S 0.0 78.5 0:00 java You can convert into its hexadecimal equivalent (e.g. 1e240 above) using any of the many utilities out there to convert integers into hexadecimal (e.g. http://int.darkbyte.ru ) If you do a thread dump of the JVM process (e.g. kill –QUIT <PID> or kill -3 PID ), each thread gives its PID, simply searching for “nid=<pid in hex>” will help you to match it "[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=1 tid=0x83da550 nid=1e240 RUNNABLE java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:129) oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:718) oracle.net.ns.Packet.receive(Packet.java:295) oracle.net.ns.DataPacket.receive(DataPacket.java:106) oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:317) oracle.net.ns.NetInputStream.read(NetInputStream.java:262) oracle.net.ns.NetInputStream.read(NetInputStream.java:187) ............................................ Although it is convoluted that is one way of finding out in Linux which exact thread is actually using CPU/MEMORY etc.
5 JMS Message monitoring
5.1 Listing JMS messages using the JBoss Console In order to retrieve the list of messages stored in each of the JMS queues follow these steps:
1. Open the JMXConsole (for example: http://localhost:8080/jmx-console)
27
2. Navigate to jboss.mq:type=DestinationManager mbean (hint: you can probably just CTRL-F and enter type=DestinationManager in the dialog box)
3. Click on the link for the DestinationManager mbean.
4. Navigate to the bottom where it says “listMessageCounter”
5. Click it and get you have the count of messages in all destinations
Alternatively you can use twiddle to execute the listMessageCounter method and pipe the returned HTML directly to file i.e.
%JBOSS_HOME%/bin/twiddle –s localhost:1099 invoke "jboss.mq:service=DestinationManager" listMessageCounter > messages.html
28
5.2 Pausing JMS Queues If you want to stop GDM from processing JMS messages, for instance, prior to stoptting the JBoss instance, follow these steps
1. Open the JMXConsole (for example: http://localhost:8080/jmx-console) 2. Navigate to jboss.mq:type=DestinationManager mbean 3. Click on the link for the DestinationManager mbean. 4. Navigate to the bottom where it says “Stop” 5. Click it and all JMS destinations will be unavailable for processing
Alternatively you can use twiddle to execute the listMessageCounter method %JBOSS_HOME%/bin/twiddle –s localhost:1099 invoke "jboss.mq:service=DestinationManager" stop
6 Diagnosing a problem This section outlines a set of steps to perform by the System Administration responsible for a GDM installation when diagnosing an issue.
6.1 Initial Investigation If stability or performance problems are reported for a GDM instance, there are series of things to check Symptom Action The application is too slow Check TaskManager or TOP in the HOST machine to
check out the load on that machine CPU is peaking at 100% Use JConsole to retrieve thread dumps and get an
overview of what the system is doing at that time OutOfMemory errors in the logs Check out the JVM settings particularly Xms and Xmx.
Confirm from the logs that they are set. Use JConsole to perform a memory analysis. If necessary do a heap dump
The application hangs Check out if there are enough JDBC connections available for the application. Either by using the WLS or JBoss consoles. Review the
TimeOut errors or SocketTimeoutExceptions Confirm that the -Dcom.datagenicgroup.gdm.DEFAULT_TIMEOUT and JTA settings are correct.
Gdmx loading or data loading is very slow taking hours for large files
Check that the GdmxDistributionService is configured to run with a technical user that has disableAuditing set to true
There are quite a lot of Heuristic Exceptions in the server logs.
Enable JTA logging in either JBoss or Weblogic and review the looks to check what resources involved
6.1.1 ORA-01000: maximum open cursors exceeded If you find this exception in the logs and you are using ORACLE, it is caused by the number of CURSORS left open by one single SESSIONS/CONNECTION exceeding the configured parameter.
29
Log on to the database as Sysdba and you can use the following statements to diagnose the problem Instruction Description Actions select max(a.value) as highest_open_cur, p.value as max_open_cur from v$sesstat a, v$statname b, v$parameter p where a.statistic# = b.statistic# and b.name = 'opened cursors current' and p.name= 'open_cursors' group by p.value;
It gives you the highest open number of cursors and the configured maxim open of cursors
Oracle has a default of 10 which is way too small for GDM. Make sure that the configured value is at least 1000. If necessary increase.
select sum(a.value) total_cur, avg(a.value) avg_cur, max(a.value) max_cur, s.username, s.machine from v$sesstat a, v$statname b, v$session s where a.statistic# = b.statistic# and s.sid=a.sid and b.name = 'opened cursors current' group by s.username, s.machine order by 1 desc;
It gives a general summary of all open cursors opened by username , machine
Identify the user that is causing the problem,
select a.value, s.username, s.sid, s.serial# from v$sesstat a, v$statname b, v$session s where a.statistic# = b.statistic# and s.sid=a.sid and b.name = 'opened cursors current' order by a.value desc;
It returns the cursors currently open BY all database connections.
Identify the SIS (session ID) that is currently over the limit.
select c.user_name, c.sid, sql.sql_text from v$open_cursor c, v$sql sql where c.sql_id=sql.sql_id -- for 9i and earlier use: c.address=sql.address and c.sid=&sid;
It returns ALL instructions (open cursors) open by the SID session
Review the instructions to ensure that there are no strange statements or indication of problems or to send to DG Support
See http://www.orafaq.com/node/758 for more details on monitoring open and cached cursors
6.2 Reporting a problem DataGenic support can be contacted at [email protected] and will be most effective in helping you with a system problem if you can provide the following information:
• Exact windows, Linux, GDM, JBoss, JDK, WLS and Database versions. • A description of what the observed problem is, what other users/activities were being performed at the time • If repeatable , provide a list of steps to reproduce the problem • If available a test case that exposes the issue • If the problem is CPU bound, provide few Thread dumps with few minutes interval • If the problem is memory related, provide a heap dump • In all cases, collect all server logs not just a snippet, and upload to DG support • If it is the first problem with a particular environment, it helps to provide a description, JBoss/WLS configuration,
particularly of the JVM and JDBC settings. • If possible, upload a database dump so DG support can replicate your environment.
6.2.1 Performing an Oracle export Expdp is an oracle utility that enables to export oracle databases into a dump file that can be sent to DG support. For instance, execute the following instructions to export the database contents for the gdm user (assuming ddefault gdm_data tablespace) into the $ORA_HOME/admin/dpdump/ GDM_01_05_2013.dmp $ expdp gdm/gdm DIRECTORY=DATA_PUMP_DIR DUMPFILE=GDM_01_05_2013.dmp The dmp file size can be greatly reduced by 7zip before uploading it to ftp.datagenic .net See http://www.orafaq.com/wiki/Datapump for more details
