Upload
anthony-beardsmore
View
618
Download
10
Embed Size (px)
Citation preview
Anthony BeardsmoreIBM MQ Appliance Architect
Session #3458 IBM MQ ApplianceAdministration Simplified
Agenda• Getting started
• Day to day tasks
• Access and security
• Configuring and managing High Availability
• Monitoring, troubleshooting
• Hands on
– Hardware config
– Startup Wizard
– Creating a queue manager – differences from software MQ
– MQ commands
– MQSC scripts
– Working with queue manager logs
– Which interface is right for me?
– Defining and managing users
– Defining an HA group
– Working with HA queue managers
– Monitoring APIs
– Application monitoring
– Hardware monitoring
– WebUI charts
– Web UI live demo
Reminder – what is the MQ Appliance?
• The scalability and security of IBM MQ V8– Integrates seamlessly into MQ networks and clusters– Familiar administration model for administrators with MQ skills
• The convenience, fast time-to-value and low total cost of ownership of an appliance
• Ideal for use as a messaging hub running queue managers accessed by clients, or to extend MQ connectivity to a remote location
• Familiar feel for existing MQ users – application interfaces, administration, networking/clustering, security….
• Plus new appliance specific features – e.g. built in high availability© 2015 IBM Corporation
Getting Started
Physical configuration
2x Management 1GB Ethernet(also IPMI)
Display (includes capacity and version information)
Serial / KVM connectionN.B. REQUIRED for initial setup
General purpose 1GB Ethernet(MQ Application traffic)
General purpose 10GB Ethernet(SPF+)(MQ Application traffic)
2x 1GB Ethernet: HA or general purpose
10 GB Ethernet (SPF+): HA or general purpose
HBA (unused)
Notes: Physical configuration
• When racking/cabling a new appliance consider the following things:• Who will be allowed to manage? From where – private network(s)?
– Best practice is probably to limit management traffic (SSH and WebUI) to mgt0/mgt1 interface(s). However on a secure internal network may choose to enable management via all interfaces for simplicity. No technical reason either mgt interface HAS to be cabled unless exploiting IPMI.
• Where will MQ traffic come from? To? Via multiple networks? Is this system going to act as an MQ ‘router’?– The appliance has 8x1GB ‘general’ Ethernet interfaces and 2x10GB.– MQ Traffic can be locked to particular interfaces by IP using listener configuration
• Are you (will you be) using HA?– Will need to configure/reserve one of the 10GB networks and 2x1GB for this purpose– These should be configured on dedicated, independent, and ideally directly connected (no
routing) private subnets.
First time setup
• Initial power on presents a fairly straightforward wizard via serial connection. See the Knowledgecenter getting started section for support in completing this.
• Some particular things to think about:– Setting up user for password recovery really matters… No factory reset
mechanism without at least one working login!– Are you going to use DNS (probably)? DHCP (probably not)?– Remember to use CIDR format for IP/subnet– MUST assign a ‘unique identifier’ (similar to hostname) if planning to use HA– MUST configure at least one IP address and enable the WebUI service on it to
accept license. Probably easiest to just set up one management interface for now and complete the rest later (e.g. through the WebUI)
Everyday tasks
Creating your first queue manager
M2000(mqcli)# crtmqm -fs 1 -p 1414 QMA
Please wait while 1024 MB file system is initialized for 'QMA'.
IBM MQ Appliance queue manager created.
Setup completed.
M2000(mqcli)# strmqm QMA
IBM MQ Appliance queue manager 'QMA' starting.
IBM MQ Appliance queue manager 'QMA' started using V8.0.0.4
Starting MQSC for queue manager QMA.
dis lsstatus(*)
AMQ8631: Display listener status details.
LISTENER(SYSTEM.LISTENER.TCP.1) STATUS(RUNNING)
PID(16441)
Storage allocation
Default listener
Creating first queue managers - Notes
• Essentially identical to software MQ• Major difference: pre-allocate disk space at create time to give
separation between QM data– total: i.e. log files plus queue files– Trace, FFSTs are separate
• Convenience option added to set up TCP listener at create time – -p <PORTNUM>
• Configuring logs – usual considerations. – Only big difference here no linear logging
Everyday activities – general commands
• All the usual MQ ‘distributed’ commands work as you’d expect within the CLI ‘mqcli’ sub shell– Just type ‘mqcli’ at the top level to enter, and ‘exit’ to go back to top– ‘help’ is surprisingly helpful!
• Includes interactive runmqsc• Exception is that where there are duplicated routes to some commands
in software MQ (because of the way it has evolved over the years) we have removed some to simplify the appliance interface– E.g. auth recs now only managed through MQSC
Remote Scripting
For some tasks (e.g. automatic configuration of users or creation of queue managers) it is critical to be able to remotely manage aspects of the appliance through a scriptable interface.
A common approach to this today is to use tools such as ‘expect’ and configure these aspects over SSH connections.
To see (and if you wish, modify, collaborate on, and share) examples of this approach visit our GitHub repository at https://github.com/ibm-messaging/mq-appliance
Working with files on the appliance
TLS Certificates/requests
Queue Manager error Logs
TraceCollected RAS (includes FFSTs, trace, logs)
Backup information
Firmware updates
INOUT
IN/OUT
mqbackup://
mqtrace://mqdiag://
mqerr://
image://
mqpubcert://
CLI Config‐>copy command (supports FTP, SCP, HTTP servers)
Can also be gathered through WebUI or accessed through ‘dspmqerr’ command
No general access to file system –everything needed is available through dedicated URIs or tooling (examples below)
File Management (UI) – new in 8.0.0.4
13
CLI Diagnostic tools
14
Administration – which interface for which tasks?
M2000(mqcli)# crtmqm testPlease wait while 64 GB file system is initialized for queue manager 'test'.IBM MQ Appliance queue manager created.The queue manager is associated with installation 'MQAppliance'.Creating or replacing default objects for queue manager 'test'.Default objects statistics : 83 created. 0 replaced. 0 failed.Completing setup.Setup completed.
M2000(mqcli)# strmqm testIBM MQ Appliance queue manager 'test' starting.The queue manager is associated with installation 'MQAppliance'.5 log records accessed on queue manager 'test' during the log replay phase.Log replay for queue manager 'test' complete.Transaction manager state recovered for queue manager 'test'.IBM MQ Appliance queue manager 'test' started using V8.0.0.4.
M2000(mqcli)# runmqsc test5724-H72 (C) Copyright IBM Corp. 1994, 2014.
Starting MQSC for queue manager test.
Serial/SSHHTTP
M2000(mqcli)# crtmqm testPlease wait while 64 GB file system is initialized for queue manager 'test'.IBM MQ Appliance queue manager created.The queue manager is associated with installation 'MQAppliance'.Creating or replacing default objects for queue manager 'test'.Default objects statistics : 83 created. 0 replaced. 0 failed.Completing setup.Setup completed.
M2000(mqcli)# strmqm testIBM MQ Appliance queue manager 'test' starting.The queue manager is associated with installation 'MQAppliance'.5 log records accessed on queue manager 'test' during the log replay phase.Log replay for queue manager 'test' complete.Transaction manager state recovered for queue manager 'test'.IBM MQ Appliance queue manager 'test' started using V8.0.0.4.
M2000(mqcli)# runmqsc test5724-H72 (C) Copyright IBM Corp. 1994, 2014.
Starting MQSC for queue manager test.
MQ Channel (PCF)
Notes: Which interface is right for me?
• Serial: First time setup and ‘emergency’ use. Various forms of serial-over-lan KVM exist which may be useful for remote recovery. Probably not an everyday tool.
• SSH – everyday interface for ‘power users’, full system administrators. CLI control over everything from network settings to individual object definitions– Remember – not a full OS shell!
• WebUI – currently best seen as graphical equivalent to SSH – highly privileged power users. Particularly useful for ‘overview’ dashboards and monitoring widgets.– Note: HTTP interfaces used by the WebUI are currently not published and considered subject to
change. Interested in RFEs!• Remote MQSC – granular authority for remote access (e.g. power user for a particular queue
manager, or lower privileges). Also the simplest way to script MQ object maniplulation(definition and display), including migration or restore from backups. – Requires MQ V8 Client.
• MQ Explorer – ideal for users experienced with this tooling, and again for granularity of access.• Other PCF based tools – many possibilities and niche specialities. As for MQSC/Explorer, does
required initial channel setup.
User Administration
Managing Administrative users: Privileged
Simplest option – keep administrative users to a minimum and make them all privileged.
Managing Administrative users: ‘group’
Allows some level of granularity, but complicated to configure, and all users will have approximately ‘mqm’
Managing Administrative users: ‘User’
NOT recommended - deprecated function from common DataPower base (creates a user which cannot perform any ‘system’ administration)
Managing messaging/application users
• Object model as for software MQ – so with appropriate authority records you can allow remote applications as much or as little administrative access as required to a queue manager
• For large deployments, consider use of new LDAP facilities and appropriate external tooling. Useful overview here:
https://www.ibm.com/developerworks/community/blogs/messaging/entry/bite_size_blogging_mq_v8_setting_up_a_qmgr_to_use_ldap_authentication?lang=en
User Store
QM1
QM2
QM3
QM4
QM5
QM6
Managing messaging/application users
• Alternatively you can choose to manage these on a per device level– IMPORTANT: remember that if you are configuring HA, both appliances
will need to define users required by any ‘grouped’ queue managers.
Note that by default no user specific group created (all users are added to ‘users’ for convenience). Appliance queue managers default to ‘per user’ authority model.
Network ConfigurationAdvanced Options and considerations
Link aggregation
• The appliance can support various forms of link aggregation for availability and increased throughput– Options include simple ‘standby’ aggregation, or LACP based depending on
your infrastructure and requirements.– Practical ‘gotcha’ – must mark interfaces as ‘available’ for aggregation
before adding to the combined interface (otherwise have to untangle).• As an example, may wish to aggregate a number of the 1GB
interfaces to provide higher bandwidth to MQ client applications• Note that currently cannot aggregate the interfaces used for
integrated HA/DR
VLAN
• The appliance supports native VLAN tagging (‘trunked’)• May be combined with link aggregation to give fault
tolerant, highly available, and securely separated interfaces for use by MQ traffic– See Redbook for good write up of this type of configuration
• Again, cannot currently be used on the HA/DR interfaces (though of course external ‘port tagging’ can be used).
High Availability and Disaster Recovery(Session HHM-3465 for deep dive)
HA Terminology - Notes
• MQ Appliance HA feature is akin to an HA product – Such as Veritas, PowerHA, etc– It is not an implementation of Multi-Instance Queue Managers
• HA Group– A configuration of MQ Appliances that monitor each other
• Try and ensure that each HA queue manager runs on one appliance but can fail over to the other if necessary
– An Appliance can be in at most one HA Group– An HA Group consists of exactly two Appliances– Not all queue managers must be members of an HA Group
• HA Queue Manager– A queue manager that is under the control of the HA Group and which has its data replicated
between the appliances• Preferred Location
– Appliance where the HA implementation will run the queue manager, all else being equal– Initially the appliance on which the HA Queue Manager is created
Setting up HA
4. Then create an HA queue manager:crtmqm -sx HAQM1
Implementing HA is a simple with the MQ Appliance!
2. On Appliance #1 issue the following command:prepareha -s <some random text> -a <address of appliance2>
1. Connect two appliances together
That’s it!
Note that there is no need to run strmqm. Queue managers will start and keep running unless explicitly ended with endmqm
3. On Appliance #2 issue the following command:crthagrp -s <the same random text> -a <address of appliance1>
Setting up Disaster Recovery
DR has different goals (asynchronous, manual) so slightly different externals but similar process
2. On ‘live’ appliance, convert queue manager to Disaster Recovery ‘source’:crtdrprimary –m <name> -r <standby> -i <ip address> -p <port>
1. Connect two appliances together (only one, 10GB, connection needed)
Synchronization begins immediately (‘status’ command shows progress)
In event that need to start the standby instance, ‘makedrprimary’ takes ownership of queue manager and then business as usual
3. On ‘standby’ appliance simply paste the text provided by the abovecrtdrsecondary <some provided parameters>
HA Queue managers in MQ Console
30
HA GroupAppliance #1
HA GroupAppliance #2
HA Queue managers in MQ Console: After Failover
31
• Appliance #1 is now in Standby
• All HA queue managers are now running on Appliance #2
• The console shows the High Availability alert, and a menu to allow you to see the status and to suspend or resume the appliance in the HA group.
Designing a group
Notes: Designing a group
• This image is straight from the Infocenter and gives a good overview of the possible combinations of Queue managers in an HA Group.
• As long as IP Addresses etc. for the three HA interfaces are correctly pre-configured, defining a group is as simple as executing the ‘crthagrp/prepareha’ commands.– Appliances can only be in exactly one group of exactly two appliances
• Queue managers may be added to the group at crtmqm time, or after creation using the ‘sethagrp’ command– Queue managers can be active on either appliance (both appliances can
simultaneously be running different active queue managers). Up to 16 active/passive instances per appliance are permitted.
• Unlimited (other than by storage capacity etc.) non-HA queue managers may also be present on either appliance.– This might be desirable for example if you have applications/queue managers with
different QOS agreements, or Test and production environments on the same system.
Monitoring
Monitoring MQ
Typically, third party (or other IBM product) tooling will already support MQ appliance queue managers without changes.
This image shows Tivoli Remote Agent displaying queue and channel information from an appliance queue manager.
Various third party vendors have already explicitly confirmed support (check with vendor for specific product information).
But we also have some more appliance specific tools…
Resource Monitoring - Notes
36
• The appliance doesn’t feature common OS monitoring tools– No vmstat, iostat, nmon, etc
• Understanding resource usage on the MQ Appliance is a must– Everything is self-contained
• CPU, Memory, Disk, etc• To assist with this, new monitoring capabilities were added
– Along with a new style of event generation– Provides information that would normally be accessed via OS-level monitors– Intent is to provide insight into how appliance resources are being utilized
• MQ Console plugs into these new performance events– New style of event generation allows multiple consumers of the same information– Use is not restricted to the MQ Console
• More on this later• The next few slides explore how to access this data using Chart widgets
Monitoring System Resources: Chart Widgets
• To create, click hotspot
37
Configure the widget
Display appliance resource use Platform-wide or queue manager CPU, Disk, memory, etc
Select: Resource class/type/element Queue manager(s) to monitor
We’ll look at setting up a simple example in the demo
Choose Resource class CPU Data stores API Usage stats
Chart Widget - CPU
• Select queue manager(s) to monitor
38
Result is a configured chart widget Refreshed every 10 seconds Different colors for each Hover over for list view
Select CPU usage by queue manager
Can monitor CPU over time as well 1, 5 and 15 min averages
Select System or User CPU usage
Chart Widget – MQI Statistics
• Can quickly configure chart widgets – Focus on suspect problem areas (queues, channels, etc)
– Statistics gathered “inside” rather than outside MQ
– Overhead minimal compared to value of information
39
Dashboards can be saved/shared Enables creation of “what if” scenarios Also repeatable monitoring profiles
Programming practices often cause poor MQ performance
Chart widget allows gathering of MQI usage Queue-manager wide, or queue-specific Insight into MQI usage can reveal:
Excessive CONN/DISC and OPEN/CLOSE Use of PUT1 vs PUT Persistent vs non-persistent Queue avoidance Queue lock contention
Chart Widget – Logger Statistics
• MQ Appliance makes some of this available via Chart widgets– Log space used, bytes written, write latency, etc
40
MQ has collected logger statistics for many releases But very hard to get at Not documented
Very useful in appliance as disk is finite Knowing when log I/O is a bottleneck vital Also knowing if logs are over/under allocated
Chart WidgetsAnalyzing performance issues
• Queue statistics help provide a clue– Queue lock contention very high
• Worse than 10% can be a problem
• In this example it’s at or near 100%!
41
Logger statistics provide further clue Write latency high
25-30ms Very high for a local disk
Problem conclusion Investigation showed large number of concurrent
putters Putting large messages outside syncpoint Log write latency aggravated problem
New statistics help pinpoint bottlenecks Scenario: Messages backing up on queue
Draining very slowly
• All data used by the chart widgets is also available programmatically or through the CLI
• Sample program amqsrua (source included with client download)– Returns classes of available data– CPU, Disk, MQI Stats, Queue Stats– How does it know what data is available?
• Performance data available via Pub/Sub topics• At startup, appliance queue managers publish a set of metatopics
– Describe performance data that can be subscribed to– Consumer(s) can subscribe to one or more topics– When subscriptions registered, MQ will publish data on requested
topic(s)
Using amqsrua for Appliance Monitoring
• Performance data available via Pub/Sub – Departure from existing MQ stats and monitoring – Performance data offered in a highly dynamic way
• Includes new data unique to the appliance– e.g. Logger write latency, queue lock contention
• Monitor application can subscribe to metatopicsMQSUB("$SYS/MQ/INFO/QMGR/<qmgrname>/Monitor/METADATA")
– Retained publications respond with what information is available
– Subscribing application can then choose to subscribe to specific topics
• MQ Console requires admin authority– So access to chart widgets will likely be restricted– But developers, etc could use amqsruac
• Client-connect• Or modify sample to suit needs
Monitoring Data - Notes
Monitoring applications
Classic use cases for dedicated/specialised exit code:
Which applications make use of which resources (e.g. queues)
What is coming in off this set of channels right now?
How can I keep an audit log of all messages put by a particular application?
Application Activity Trace
• Activity trace produces information about application MQI calls– Provides a detailed view of the parameters used by an application – Also shows the sequence of MQI calls issued by an application– Allows offload of entire message or MQMD
• In the past configured by editing mqat.ini• Appliance introduces a new method of subscribing to activity trace
– Data published to special IBM MQ system topics• Pub/Sub approach means there can be multiple subscribers to the same data
– Basic topic is "$SYS/MQ/INFO/QMGR/<qmgr name>/ActivityTrace"• Subscriber can then add "/ApplName/amqsputc.exe"
• Or "/ChannelName/SYSTEM.DEF.SVRCONN“
• Events in same format as on other platforms
• Very good write-up here:– http://www.ibm.com/developerworks/community/blogs/messaging/entry/Tracing_applications_with_the_MQ_Appliance
45
Monitoring System Health
• Because based on DataPower stack, much of the ‘low level’ system tooling is very similar– Existing skills will transfer well and many education resources available– See system logs in ‘logtemp:’ for general information (not persisted across
restarts) and audit log for major event tracking (persisted).• Logs can be offloaded for external processing via various mechanisms
– Syslog– SMTP – Etc.
• Consider configuring this early for better alert capabilities and ‘root cause’ data• note – no general SNMP support or traps at present
DemoLogin - Appliance Management - Dashboards -Queue Manager widgets - Object widgets - Charting -CPU monitoring.
Monday10:30-11:30 3592 New MQ features
3452 Managing applications
12:00-13:00 2835 MQ on z/OS and Distributed
15:00-16:00 3470 Latest MQ z/OS features2833 Where is my message?3544 MQ Light in an MQ infrastructure
16:30-17:30 3573 Hybrid cloud messaging2941 MQ Advanced
Tuesday08:30-09:30 3540 The MQ Light API
12:00-13:00 3456 The IBM MQ Appliance
13:15-14:15 3499 Introducing Message Hub3458 MQ Appliance administration
14:30-15:30 6432 MQ updates and futures (InnerCircle)2849 Messaging feedback roundtable
16:00-17:00 3544 MQ Light in an MQ infrastructure3513 MQ hands on lab
Wednesday08:30-09:30 3602 Managing your MQ environment
12:00-13:00 3613 Designing MQ self service6408 Hybrid messaging roadmap (InnerCircle)
13:15-14:00 3416 HA and DR with MQ3433 Why secure your messaging?
15:45-16:30 3429 Securing MQ2847 Meet the messaging experts
16:00-17:00 3508 MQ Light hands on lab
16:45-17:30 2275 Migrating to the IBM MQ Appliance
Thursday08:30-09:15 3420 MQ Clustering
2931 Business agility with self service MQ
09:30-10:15 3479 MQ z/OS clusters and shared queue3450 Optimising MQ applications2849 Messaging feedback roundtable
10:30-11:15 3465 MQ Appliance high availability3481 MQ z/OS messaging connectivity
11:30-12:15 3474 Active-active messaging3537 Monitoring and managing MQ3425 MQ publish/subscribe
Find us at the EXPO:Hybrid Integration peds 65-68
Check out the Hybrid Messaging sub topic under theHybrid Integration topic for further customer and business
partner sessions
Hybrid Messaging from the IBM experts at InterConnect 2016 Sunday
14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)
Notices and Disclaimers
49
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
Notices and Disclaimers Con’t.
50
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Thank YouYour Feedback is Important!
Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your
smartphone, laptop or conference kiosk.