Upload
nico-meisenzahl
View
1.024
Download
0
Embed Size (px)
Citation preview
Social Connections 11 Chicago, June 1-2 2017
Two wrongs don’t make a right –Troubleshooting Connections
Terri Warren, panagenda
@TerriLLBean
Nico Meisenzahl, panagenda
@nmeisenzahl
PLATINUM SPONSORS
GOLD SPONSORS
SILVER SPONSORS
Social Connections 11 Chicago, June 1-2 2017
Nico Meisenzahl
• Consultant at panagenda
• IBM Connections since version 3.0 / 2010
• IBM Notes / Domino since 2008
• Focusing in ICS
• Deployment & consulting
• Optimization and migration
• “panagendian” since 2016
• IBM Champion
@nmeisenzahl
linkedin.com/in/nicomeisenzahl
meisenzahl.org
nico.meisenzahl
+49 170 7355081
Social Connections 11 Chicago, June 1-2 2017
Terri Warren
@TerriLLBean
linkedin.com/in/terrillbean
live:TerriLLBean
• Consultant at panagenda (since 2016)
• In my recent past I was a developer on IBM
Connections, IBM Domino
(Directory/Security) and an SRE on Watson
• Frequent presenter at IBM conferences
focusing in ICS
• Deployment & consulting
• Optimization and migration
Social Connections 11 Chicago, June 1-2 2017
Agenda
• Howto: Troubleshooting
• Troubleshooting…• Connections itself
• Backend like Spectrum CfC, DB2, TDI, WebSphere, Directory, SSO
• Optional add-ons like Orient Me, Docs, FEB, Cognos, CCM
• Tools
Social Connections 11 Chicago, June 1-2 2017
Howto: Troubleshooting
Social Connections 11 Chicago, June 1-2 2017
Reproduce the error
• Reproducible and/or periodically?
• A sequence error?
• Client-side or server-side problem?
• Analyze the root cause
Social Connections 11 Chicago, June 1-2 2017
Be aware of the big picture
• Client-side problems• Debug in different Browsers (IE, FF, Chrome)
• Do not use a server IE
• Server-side: IBM Connections is based on many components• debug on “high level” first
• get an overview which backend service is causing the error
Social Connections 11 Chicago, June 1-2 2017
Configuration changes
• Changes in…
• Connections configuration
• Backend (WebSphere, Database, HTTP, CfC)
• Firewall or network
• OS, hardware or VM
Tip: Even the smallest configuration change can have big consequences!
Social Connections 11 Chicago, June 1-2 2017
Analyze log files and browser
• Analyse log files• Atom.io, Notepad++ or less/tail
• Baretail or tail –f
• kubectl logs
• ELK stack
• Tools for client-side problems• Firebug or Developer-Tools
• BurpSuite or Fiddler
Social Connections 11 Chicago, June 1-2 2017
Analyze root cause
• Find a hint inside the log
• Network timeout or DNS
• SQL errors
• LDAP errors
• Syntax errors in configuration files „xxx-config.xml“
• Error stack
Tip: In a clustered environment, start and analyze only one
Node (if possible)
Social Connections 11 Chicago, June 1-2 2017
Find support
• Knowledge Center https://goo.gl/up6cxG
Troubleshooting Tips https://goo.gl/IaVinx
• IBM Connections Forum http://goo.gl/CVvQCU
• Community Blogs and/or Chats
• Fix Central
• PMR
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting
Social Connections 11 Chicago, June 1-2 2017
WebSphere Application Server logs
• SystemOut.log
• SystemErr.log
• trace.log (if tracing is enabled)
• Log path:• <wasroot>/profiles/<profilename>/logs/<servername>/
Social Connections 11 Chicago, June 1-2 2017
Analyze WAS log files
• Time stap: 24h time stamp with milli-seconds
• Thread id: eight character hexadecimal value
• Short name: typically java class name
• Event type: one character only (E, W, I,…)
• Message identifier: String based on component
• Message: Some information
Social Connections 11 Chicago, June 1-2 2017
WAS Event types
• F - Fatal message
• E - Error message
• W - Warning message
• A - Audit message
• I - Informational message
• C - Configuration message
• D - Detail message
• O - Messages that are written directly to System.out by an application
• R - Messages that are written directly to System.err by an application
• Z - Place holder to indicate type was not recognized
Social Connections 11 Chicago, June 1-2 2017
WAS Message identifier
• Prefix by Application or Server (CLFRW)
• Specific application code (0042)
• Event Type (I)
Social Connections 11 Chicago, June 1-2 2017
Read trace stack
• First line displays key information
• “Caused by” displays root cause
Social Connections 11 Chicago, June 1-2 2017
Environment information
• First log lines on server startup
• WebSphere version
• OS version, Process id
• Installation path
• Java version
Social Connections 11 Chicago, June 1-2 2017
Enable tracing
• Enable tracing using ISC
• Runtime or configuration
only
• Define tracing based on
• App prefix / error stack
• Must gather (PMR)
Social Connections 11 Chicago, June 1-2 2017
Search issues
• http(s)://<fqdn>/search/serverStatus
• Display index, seedlists, log information
• Data is displayed for one node only• Using node fqdn to access different nodes
Tip: This will create persistent files on your hard disk!
Social Connections 11 Chicago, June 1-2 2017
Debug Search
• Search queries (runtime)
• com.ibm.connections.search.index.searching.*=all
• Crawling & seedlists
• com.ibm.connections.search.index.indexing.*=all:
com.ibm.connections.search.seedlist.*=all:
com.ibm.connections.httpClient.*=all
Social Connections 11 Chicago, June 1-2 2017
Recreate Search Index
• SearchService.startBackgroundIndex()
• Crawls seedlists
• Extracts the file content
• Create index
Tip: Use „all_configured“ to index all apps
Social Connections 11 Chicago, June 1-2 2017
CLFRW0394E: Search indexing of services
...
• Search index not ready
• interruption at index creation• CLFRW0283E: Search has encountered a problem
while crawling
• CLFRW0027E: Error Indexing component <app> for search
• INDEX.READY file not present
• Recreate and enable tracing
Social Connections 11 Chicago, June 1-2 2017
Database connections
• Check datasouces• ISC – Resources – JDBC – Data sources
• Check logs for more information
• DB2 server log• <instanceroot>/sqllib/db2dump/
• db2diag.log
• db2diag.xxx.log (log rotation, you should enable this!)
Tip: Oracle users have password expiration enabled by default!
Social Connections 11 Chicago, June 1-2 2017
HTTP Server (IHS & Plugins) logs
• IBM HTTP Server
• <installroot>/logs/
• error_log
• access_log
• based on configuration
• WebSphere AppServer Plugins
• <installroot>/logs/<webserver>/http_plugin.log
Social Connections 11 Chicago, June 1-2 2017
HTTP 404 not found
• Outdated Plugin configuration
• Restart IHS
• WAS Plugin configuration issue
• http_plugin.log
Social Connections 11 Chicago, June 1-2 2017
HTTP 404 not found
• AppServer or App down
• Network issue
• http_plugin.log
Social Connections 11 Chicago, June 1-2 2017
HTTP 500 Internal Server Error
• Unexpected error
• http_plugin.log
• Configuration issue
• WAS Root certificate not trusted or missing
• SSL certificate expired
Social Connections 11 Chicago, June 1-2 2017
Debug wsadmin
• Enable trace within wsadmin session
• AdminControl.trace(‘com.ibm.*=all’)• <wasroot>/profiles/<profilename>/logs/wsadmin.traceout
Social Connections 11 Chicago, June 1-2 2017
Directory/Troubleshooting/ Tuning
• Waltz and Sonata overview
• Enabling Log parameters…
• Cache Tuning
• SSO Config Glitches
• CCM Troubleshooting
Social Connections 11 Chicago, June 1-2 2017
Connections Troubleshooting
• Waltz: "common directory services"
• Directory Service eXtension (DSX) REST API.
• Sonata: common HttpClient
services through a RESTful service
• configurable authenticators that support
• SSO security tokens and cookies for
secure server-to-server communication.
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting:
DSOutOfServiceExceptions
• Symptoms:• Access Connections not possible
• DSX not working
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting: Enable Logging
• Track all basic Waltz configuration settings and transactions.• com.ibm.connections.directory.services.*=all
• Track all LDAP transactions in between Waltz & LDAP server(s).
• WaltzLDAPUsage=all
• Track all LDAP entry to be cached and hit from cache upon DN of LDAP entries.
• WaltzDNEntryCache=all
• Track all LDAP entry to be cached and hit from cache upon ID of LDAP entries.
• WaltzExactIDMatchCache=all• Track all group membership (a list of groups) for a given user upon ID.
• WaltzGroupMembershipCache=all
• Track all members (a list of users) for a given group upon ID.
• WaltzMemberExpansionCache=all
TIP: Save runtime changes to make changes persistent
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting: Enable Logging
• WAS VMM/WIM:
• com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all
• WAS SPNEGO:
• com.ibm.ws.security.spnego.*=all
• Apache commons HttpClient:
• org.apache.commons.httpclient.*=all
• WAS security:
• com.imb.ws.security.*=all:com.ibm.ws.security.policy.*=off;
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting: Enable Logging
• Basic Sonata configuration settings and transactions:• com.ibm.connections.httpClient.*=all"
• Track the amount of HTTP(S) transactions.• SonataHttpUsage=all
• Track headers received for all HTTP(S) transactions.• SonataHttpHeader=all
• Track all bodies received for all HTTP(S) transactions.• SonataHttpBody=all
Social Connections 11 Chicago, June 1-2 2017
EJPVJ9284E: Unable to get the groups
from the directory for the user…
• User was not able to access Connections
anymore
• WAS LDAP bind user had no read access
to one of the groups the user was member
Social Connections 11 Chicago, June 1-2 2017
SSO Configuration Glitches:
• Warning indicating SSO config is missing or
incomplete enabling application security:
• Ensure you filled
in your domain
Tip: Connections CRs sometimes
resets SSO domain
Social Connections 11 Chicago, June 1-2 2017
SSO Configuration Glitches:
• LtpaToken Cookies: • More secure- recommended to use version 2 LTPA cookies
In other words- interoperability mode is left as disabled by default.
Only enable interoperability mode ifsome app servers are still relying on LTPA version 1 and need to interoperate.
Connections Security supports custom LTPA cookie names since 4.5 CR1
Sets com.ibm.websphere.security.customSSOCookieName custom properties (verify 3rd party products Portal/Domino can handle it)
Social Connections 11 Chicago, June 1-2 2017
Waltz Cache: Changing Group
Membership/permissions remains in Waltz Cache
• Issue: Administrator changes access or Group membership for a user. That user does not get immediate access.
• Cache: Connection's Directory Services keeps a “list” of directory objects it has already searched for performance reasons
● Improves performance of connections services
● Reduces load on remote directory servers
● Cache is based on a timing mechanism
● Cache is "flushed and renewed" on a 12 hour time schedule
Social Connections 11 Chicago, June 1-2 2017
Tuning the Cache: Continued
• How does this affect Group Membership?● Group membership exists in the cache for 12 hours
● Operations such as renaming, deleting, or updating groups remain in the cache for that time
• Configuration parameter for cache timing mechanism
● Enables Connections administrators to flush the cache in configurable time increments
● Set cache parameter via JVM arguments to flush them (we'll get to that!)
● Default remains 12 hours
Social Connections 11 Chicago, June 1-2 2017
Cache Configuration: Setting the cache
flush parameters• Setting the cache parameters via JVM configured in
WebSphere● Google Setting generic JVM arguments in WebSphere Application
Server
• Time Values must be the same for all cache settings
• Adjust with care! Flushing the cache has implications!
Social Connections 11 Chicago, June 1-2 2017
IBM Connections Content Manager
(CCM): Troubleshooting• During library creation, retrieving or modifying content:
• Calls are made between Communities, Profiles, and FileNet Issues with Filenet / Connections Profile or Community configuration can
adversely affect access to Libraries and CCM
Logging from Connections components (previous slides) and FileNet essential
• SystemOut.log & trace.log – these are your friend
• FileNet core logging: • <wasprofile>/<servername>/p8_server_error.log
• <wasprofile>/<servername>/p8_server_trace.log
• <wasprofile>/<servername>/pesvr_system.log
• <wasprofile>/<servername>/pesvr_trace.log
Social Connections 11 Chicago, June 1-2 2017
CCM Troubleshooting: Filenet URLS
• Status about FileNet Collaboration Services:
• check the status page at http://<fqdn>/dm
Navigator version and configuration
• Status about FileNet Content Engine
• check the status page at http://<fqdn>/FileNet/Engine
Server status, version, sonata/waltz version
• Status about FileNet Content Engine Domain and Object Store
• check the status page at http://<fqdn>/P8CE/Health
Health checks for authentication, stores and database
Social Connections 11 Chicago, June 1-2 2017
Debug CCM Widget
• Widget issues
• com.ibm.quickr.communitylibrary.*=all:
com.ibm.lconn.widgets.service.*=all:
com.ibm.lconn.widgets.actions.*=all
• Authentication issues (Remember those cool tips a few
slides back? )
• com.ibm.connections.directory.services.*=all:
com.ibm.connections.httpClient.*=all
Social Connections 11 Chicago, June 1-2 2017
Debug FileNet using ACCE
• ACCE tool: check if FileNet Content Engine can
authenticate using Connections and inspect other data
for errors:
http(s)://<fqdn>/acce
Social Connections 11 Chicago, June 1-2 2017
CCM: Enable logging for Connections and FileNet
using JVM Properties
• Download and copy the sample waltz.sonata.log4j. as log4j.xml:• Linux: /<absolute_path_for_log4j_file>/log4j.xml
• Windows:\<absolute_path_for_log4j_file>\log4j.xml
• Add generic JVM properties
• -Dlog4j.configuration=file:<path>/log4j.xml -DskipTLC=true
• Tells the Content Engine (CE) server to skip it’s tracing configuration and writes to waltz.sonata.trace.log Windows: c:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01Linux: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
Social Connections 11 Chicago, June 1-2 2017
CCM: Enable logging for Connections and FileNet
using JVM Properties
• Copy & customize sample log4j.xml
• <ContentEngineRoot>/config/samples
• 20 subsystems (db, engine, security, search,…)
Social Connections 11 Chicago, June 1-2 2017
TDI logs
• <tdisol>/log/ibmdi.log• TDI log file
• <tdisol>/employee.*• Files include all changed users (adds, update, delete, error, skip)
• <tdisol>/syncupdates/*• Temporary files within the sync
• Including database dump and ldiff
• sync_updates_clean_temp_files=false (default: true)• profiles_tdi.properties
Tip: Check the lock file
Social Connections 11 Chicago, June 1-2 2017
Analyze TDI logs
• Error code prefix• CLFRN: Profile & User synchronization
• CTGDIS: TDI itself
• Error code suffix
• I, E, W, …
Social Connections 11 Chicago, June 1-2 2017
Debug TDI
• Profile & User synchronization
(<tdisol>/etc/profiles_tdi.properties)
• source_ldap_debug=true
• debug_update_profile=true
• debug_collect=true
• TDI issues (<tdisol>/etc/log4j.properties)
• log4j.rootCategory=DEBUG, Default
Social Connections 11 Chicago, June 1-2 2017
Cognos BI logs
• Cognos BI• SystemOut.log & trace.log
• <installroot>/logs/cogserver.log
• Cognos Transformer• <installroot>/logs/cogserver.log
• PowerCube build• <installroot>/metricsmodel/trxschelog.log
• <userhome>/Transformer/Logs/*.log (win only)
Social Connections 11 Chicago, June 1-2 2017
BMT-MD-6003 No connection to the
data source …
• PowerCubes not created yet
• Check cronjobs or scheduled jobs
• trxschelog.log
Social Connections 11 Chicago, June 1-2 2017
Debug Cognos BI & Metrics
• Communication between Cognos BI and Connections Metrics
• SonataHttpUsage=all:SonataHttpHeader=all:SonataHttpBody=all: com.ibm.connections.httpClient.*=all: com.ibm.connections.metrics.*=all
• Connections Metrics Servlet
• com.ibm.connections.metrics.cognos.servlet.*=all
Social Connections 11 Chicago, June 1-2 2017
Docs/Viewer logs & urls
• SystemOut.log & trace.log
• http(s):<fqdn>/vsanity/check
• http(s):<fqdn>/sanity/check?app=all&querytype=report
• http(s):<fqdn>/*/version
Social Connections 11 Chicago, June 1-2 2017
Debug LTPA between Domino & WAS
• Debug on Domino side (notes.ini)
• Debug_SSO_Trace_Level=2
• Webauth_verbose_trace=1
• WebSess_verbose_trace=1
• Debug_outfile=<logfilepath>
• Debug on WebSphere
• com.ibm.ws.security.ltpa.*=all
Social Connections 11 Chicago, June 1-2 2017
Debug Kerberos
• Configuration• com.ibm.ws.security.spnego.*=all:
com.ibm.ws.security.*=all: com.ibm.issw.spnegoTAI.*=all: com.ibm.security.krb5.*=all
• Runtime• com.ibm.connections.httpClient.*=all:
com.ibm.connections.directory.services.*=all:com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all
• Fiddler & BurpSuite
Social Connections 11 Chicago, June 1-2 2017
IBM Spectrum CfC stats
• https://fqdn:8443
Social Connections 11 Chicago, June 1-2 2017
Docker & Kubernetes status
• kubectl get pods (--all-namespaces)
• kubectl get services
• kubectl get deployment
• kubectl describe pods xxx
• docker ps
• docker info
Social Connections 11 Chicago, June 1-2 2017
Docker & Kubernetes status
• Persistent Volume status
• kubectl get pv,pvc
Social Connections 11 Chicago, June 1-2 2017
Orient Me logs
• Display application logs
• kubectl logs xxx• -f
• --tail=20
• --since=1h
• Special tasks for MongoDB and Solr
Social Connections 11 Chicago, June 1-2 2017
MongoDB Troubleshooting
• Check persistent volumes
• Access logs
• kubectl logs xxx [mongo|mongo-sidecar]
• Get replica status
• kubectl exec -it mongo-0 -- mongo mongo-0.mongo:27017 --eval
'rs.status()’
• Check “health” status for all 3
members
Social Connections 11 Chicago, June 1-2 2017
Solr logs
• kubectl exec -it xxx -- cat /var/solr/logs/solr.log
• kubectl logs xxx
• only display deploment logs
Social Connections 11 Chicago, June 1-2 2017
Pods startup issues
• kubectl describe pod xxx
Social Connections 11 Chicago, June 1-2 2017
CfC logs
• docker logs -f kubelet
• docker logs -f calico
Social Connections 11 Chicago, June 1-2 2017
Docker tracing
• service docker stop
• docker daemon –debug
• journalctl -u docker.service
Social Connections 11 Chicago, June 1-2 2017
What to get more insights on Orient Me?
Social Connections 11 Chicago, June 1-2 2017
Tools
Social Connections 11 Chicago, June 1-2 2017
Analyze logs
• Analyze logs live• Baretail
• tail –f
• View logs • Atom.io, Notepad++
• less, tail
• ELK Stack• Elasticsearch, Logstash, Kibana
• Small Docker deployment
• Kubectl (local client)
Social Connections 11 Chicago, June 1-2 2017
Analyze logs
• Kubectl (local client)
Social Connections 11 Chicago, June 1-2 2017
Analyze Client-side
• Browser
• Firebug / Developer Tools
• Intercepting proxies
• Fiddler
• Burp Suite
• VMs with different IE versions
• Without GPO
• https://www.modern.ie/en-us/virtualization-tools
Social Connections 11 Chicago, June 1-2 2017
Database Clients
• db2 command
• Dbeaver or IBM Datastudio
• Robomongo (MongoDB)
Tip: Database write access is not supported!
Social Connections 11 Chicago, June 1-2 2017
LDAP & Search
• LDAP
• ldapsearch command
• Softerra LDAP Browser
• Apache Directory Studio
• Search Index
• Luke (Lucence Index Toolbox)
Social Connections 11 Chicago, June 1-2 2017
Analyze Network
• Wireshark
• tcpdump
Social Connections 11 Chicago, June 1-2 2017
Q&A
PLATINUM SPONSORS
GOLD SPONSORS
SILVER SPONSORS