Embed Size (px)
Citation preview
HHM-3613 Deploying MQ resources in a self-service or as-a-service infrastructure David WareIBM MQ Architect
Please Note:
2
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.
• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
Agenda
• Business drivers
• Self service of MQ resources
• Running MQ as-a-service
Why?
Enterprises need increased agility with improved administration efficiency
“Applications deployed in minutes or hours, not months”
“Same size middleware team, more and more applications”
“The system must be able to respond to change”
• Business pressures on the infrastructure teams to do more with less
• Low MQ skills with application owners
• MQ must fit into a cloud deployment
How?
• MQ infrastructure teams must increase the speed of delivery to
their customers
• MQ complexities need hiding from the application teams
• Confidence in the MQ infrastructure must be maintained
• But how do you get that from MQ?
– Self service
• Providing a portal for self-service provisioning of MQ resources
– As a service
• Adopt a service style messaging architecture
http://ibm.biz/mqaas_red
Benefits
• Running a self service/as a service infrastructure will provide many benefits
– Increased agility
• Empowers lines of businesses
– Improved usability
• Concentrate on the meaning to the users, not MQ
– Improved efficiency
• Streamline the development, test and production roll outs
– Improved consistency
• Provides a framework for enforcing company standards and designs
A real example
The MQ Light Service in IBM Bluemix
• IBM runs a fully managed messaging
service for users in Bluemix
• Providing applications access to
asynchronous messaging using the
MQ Light API
• This is underpinned by a highly available,
fully automated MQ environment…
Select yourself a service
Create an instance of the service
Within seconds the service is there
But behind the scenes
Bluemix Network Bridge
MQ Light Operations Tools
MQ Light Service Runtime
BlueMix Fabric
MQ Light Service Broker
(Provisioning Interface)CloudController DEAs
Chef server
MQ Light Lookup App
(Directory Service)
Agent Pairs[Multi-instance MQ Queue Managers]
(User apps)
Operations and Monitoring
Browser
LogstashServers
[Log Collation]
Conductor[OperationsFunctions]
ComponentStore
Build server
RetrieveConnectonDetails
Connections from Applications
DevOps
Single Instance 2 Instances
https://mqlightprod-lookup.ng.bluemix.nethttps://mqlight-broker.ng.bluemix.net
Highly Available GPFS Storage Cluster [QM Data] ZooKeeper Cluster [Service State]
Grafana / Seyren[Statistics and
Monitoring]
Omnibus Gateway[Alarm Relay]
KeyBox[Audited SSH
Access]
Opera
tions
Porta
lQuorum NSD
VM
Templates Omnibus Dashboard[Alarm Dashboard]
VMRAID NSD RAID NSD VM VM VM VM VM
VM
BMBM
VM VM VM VM VM VMVM
Authenticated HTTP Proxy [Blink]Port Forwarder
[BoschCli]
VM VM VM VM
MQ Light Delivery pipeline
Firewall rule allowing access for
Omnibus dashboard port 10002
Key Authenticated
SSH Proxy [BoschCli]
Which automatically ties directly into our operations
Self service
Self service
• Who is this ‘self’?
– Typically the application owning teams
– But could be the administration team
• Scope of self service?
– Development environments
– Test systems
– Production environments
• Extent of what to self serve?
– Simple provisioning of queues?
– Provisioning of whole queue managers and architectures?
Self service tooling
• From one point of view MQ’s “self service portal” is
simply its traditional administrative tooling
– I.e. MQ Explorer, runmqsc
– This provides remote access to MQ configuration, with per-
user/group authorisation for visibility and modification.
• Some expose all this to their development users
• However…
– You can’t expect your users to know all there is to know about
MQ configuration
– You probably only want to expose a subset of capabilities
– MQ is not the centre of the universe! You’ll need to coordinate
MQ resources with the other components in the solution
anyway
Self service tooling
• So from that point of view, MQ doesn’t have self service tooling
• And as no off the shelf solution will 100% match your corporation’s
requirements you’ll need to build/customise your own
– And many have built such tooling over the years
• Options for coordinated self service provisioning
• UrbanCode Deploy
• Pure Application Systems
• Chef, Puppet, …
• Third party tooling
• Hand rolled
• Instead, MQ provides the interfaces to make using these possible…
**** Message ****
length - 724 of 724 bytes
00000000: 080A 4103 0000 0000 5744 5220 0200 0000 '..A.....WDR ....‘
00000010: 8800 0000 6700 0000 514D 4752 315F 3230 'ˆ...g...QMGR1_20‘
00000020: 3135 2D31 302D 3239 5F30 392E 3431 2E31 '15-10-29_09.41.1‘
00000030: 3620 2020 2020 2020 2020 2020 2020 2020 '6 ‘
00000040: 2020 2020 2020 2020 514D 4752 3120 2020 ' QMGR1 ‘
00000050: 2020 2020 2020 2020 2020 2020 2020 2020 ' ‘
00000060: 2020 2020 2020 2020 2020 2020 2020 2020 ' ‘
00000070: 2020 2020 2020 2020 0000 0000 0000 0000 ' ........ ‘
00000080: 58CA 0000 0000 0000 0000 0000 0000 0000 'X...............‘
00000090: 644E 4656 2116 4656 3230 3135 2D31 302D 'dNFV!.FV2015-10-‘
000000A0: 3239 2020 0000 0000 3039 2E34 312E 3233 '29 ....09.41.23 ‘
000000B0: 0100 0000 4D51 4D4D 0000 0000 3038 3030 '....MQMM....0800‘
000000C0: 3030 3034 0000 0000 434C 5553 5445 5231 '0004....CLUSTER1‘
000000D0: 2E51 4D47 5231 2020 2020 2020 0B00 0000 '.QMGR1 ....
000000E0: 0800 0000 0200 0000 2020 2020 2020 2020 '........
000000F0: 2020 2020 2020 2020 2020 2020 2020 2020 '
MQ’s administrative interfaces
• MQ provides comprehensive interfaces for automating administrative changes and monitoring
• Scripting– MQSC
– Pipe commands into runmqsc
– Runs locally or remotely [V8]
• Programmatically– Programmable Command Format (PCF)
– Use messaging to send administrative commands to queue managers
– Consume messages containing monitoring, statistics and event information
IBM UrbanCode Deploy
• UrbanCode orchestrates and automates the
deployment of applications and middleware
configurations into development, test and
production environments.
• Coordinates deployments across multiple
machines.
• MQ plugin for UrbanCode provides automations
for off the peg or customised deployments of
UrbanCode artefacts
• Queue manager creation
• Queues, channels, topics, etc.
• Custom mqsc scripts
• …
See the MQ as-a-service redpaper for details
IBM MQ Chef Cookbook
• Experimental IBM MQ cookbook
• Demonstrates idempotent installation on
MQ installation on Linux
• Includes queue manager creation and
start
• More to be added
• Feedback very welcome
https://github.com/ibm-messaging/mq-chef
Self-service portal considerations
• Limit the MQ options available to
users as much as possible
– This will depend on the expected MQ
skill level of users
– Minimise MQ resources and attributes
exposed to the users
– Ideally abstract the underlying MQ
resources from the users
– We’ll discuss this area more later
• Use a portal to act as an
intermediary
– Do not grant your users MQ
administrator rights, the portal is the MQ
administrator
• Log all changes
– Either using the portal
– Or (and) with MQ configuration events
• Integrate resources direct into the
monitoring system
– When provisioning systems, enable
background monitoring as standard
As a service
As a service MQ
• MQ has near infinite variations in architectures and
configurations
– Excellent for allowing customisation to your companies exact requirements
– Bad to expose to the end users through self-service
• To support self-service, limit those variations and
customisations to a minimal set of patterns
– Once the patterns have been determined, the self-service tooling should
map any requests onto the appropriate pattern
– This reduces the level of end user knowledge required to make changes
– It also simplifies the job of the operations teams in supporting such systems
• You may need to compromise on some of your customisations and
optimisations
MQ as a service best practices
• There is no single MQ pattern but there are
good practices that can make such
deployments more successful
– Naming conventions
– Separate the applications from the queue managers
– Application isolation
– MQ Clustering
– Messaging hubs
Naming conventions
• Pick a naming convention and
enforce it
– Use your self service portal to generate
the names, don’t leave it to the user
– Helps the operations teams to
understand the system even if it has
been created by the application teams
– Be mindful of the MQ 48 character,
special character, restrictions
– Map names of MQ resources into a directory
to record additional information
• For example, for each resource:
– Who is responsible for this resource?
– What applications are dependant of this
resource?
– What role is this resource playing?
– Unique within the application
• E.g. “ORGA.APPZ.REQUEST.Q1”
Detach applications from queue managers
• Binding an application to a specific queue manager can restrict future
changes
– Changes to architecture
– Changes to scale of messaging or application runtimes
– Changes to software versions
– Maintenance windows
• Run applications remote from the queue managers
– Connect as clients to queue managers
• Hide specific queue manager names from the applications
– An application should definitely not need to hard code a queue manager’s name
• Use Client Channel Definition Tables (CCDTs)
– These provide encapsulation of connection information
– Centrally administered and distributed to applications
– Enable security, high availability and workload balancing of clients
QMgr
Application ApplicationApplication
Client channel definition table (aka CCDT)
• Applications simply connect to a “queue
manager” name
• MQ architecture detail hidden from the
application
• CCDT defines which real queue managers
the application will connect to
– Can be a single queue manager
– Or a group, defined in the CCDT
• Selection can be ordered or randomised
and weighted
QM1
QM3
QM2
CCDT
QMGRP1: QM1
QMGRP2: QM2
QMGRP2: QM3
Application 1connect: *QMGRP1
Application 2connect: *QMGRP2
Application 2connect: *QMGRP2
Use ‘*’ names to decouple the application from the real queue manager name
• CCDTs can represent connection details to
multiple queue managers
• CLNTCONN channels are defined to identify the
SVRCONN channels
• Define multiple CLNTCONNs in a central place to
generate the CCDT
– It doesn’t have to be any of the queue managers owning the
SVRCONNs
– Pre-MQ V8: Use a dedicated queue manager for this
purpose
– MQ V8: Use runmqsc –n to remove the need for a queue
manager
• Distribute CCDTs to the application locations
– Should be integrated into the orchestration of
MQ resources and applications.
Client channel definition table (aka CCDT)
Create the QMgrs and define
their SVRCONNs
Centrally define all the
CLNTCONNs that represent
all the QMgrs
Take the CCDT and make
available to the connecting
applications
• A single CCDT for your MQ estate or one per
application?
– A single CCDT can be easier to create but updates can be
expensive
– Separate CCDTs make it easier to update when an
application’s needs change
AppApp
Queue manager tenancy
• Multiple applications need isolation from each other
– Security and impact
• Do you share queue managers or provide dedicated ones for your users?
• Multi tenant
– Lower machine overheads
– More care needed in configuring to achieve isolation
• Isolation of machine resources not possible
– Harder/simpler to monitor
• Depends on your view of more queue managers
– Fine grain security required
• Single tenant
– Simple to configure, maintain and monitor
– Very good isolation
– A proliferation of queue managers
– Harder when integration is required
QMgr
App
QMgr
App
QMgr
App
QMgr
AppApp
App
QMgr
AppApp
App
QMgr
App
Queue manager architectures
• Standardize your queue managers
– Minimise bespoke configurations
– Groups of queue managers have roles, providing
equivalent capabilities
– Consider a layered architecture
• For example:
– Connectors: Queue managers that inbound
applications connect to
– Service providers: Queue managers that
host the resources being serviced
MQ Clustering for integration
• MQ clusters is the right way to connect queue
managers together for as-a-service
• Fits the model of easily adding and removing
queue managers
• Dynamically handles end-to-end routing
• Provides workload balancing for horizontal
scaling and dynamic routing for continuous
availability
• Drastically reduces the amount of
configuration required when new resources
are defined, simplifying any self service
solution
QMgr
QMgr
QMgr
QMgr
QMgr
An MQ Hub topology
• Pulling all this together into the MQ “Hub” topology
– The hub decouples the applications from the
underlying MQ infrastructure
MQ
App
App
App
App
App
App
App
App
Service requestor or
event emitters
Service provider or
long running consumer
An MQ Hub topology
App
App
App
App
App
App
App
App
QM1
QM2
QM2
QM1
Make each queue manager highly
available
Have multiple equivalent queue managers
• Pulling all this together into the MQ “Hub” topology
– High availability of the MQ runtime should be baked into the
hub design
An MQ Hub topology
App
App
App
App
App
App
App
App
QM1
QM2
QM2
QM1
Use CCDTs and design
applications to be able to
connect to one of many
queue managers
Connect instances of your
service applications to
more than one queue
manager
• Pulling all this together into the MQ “Hub” topology
– The applications must be configured to exploit the
availability of the MQ runtime
An MQ Hub topology
App
App
App
App
App
App
App
App
QM1
QM2
QM2
QM1
• Pulling all this together into the MQ “Hub” topology
– Separating out the MQ infrastructure into
front facing and service facing tiers can be
used to improve workload balancing.
QM1
QM2
QM4
QM3
Separate out the requestor/emitter queue managers from the
service provider queue managers
Use an MQ cluster to workload balance and dynamically route
work to the service providers
An MQ Hub topology
App
App
App
App
App
App
App
App
QM1
QM2
QM2
QM1
• Pulling all this together into the MQ “Hub” topology
QM1
QM2
QM4
QM3
QM5
QM5App
App
App App
Applications and MQ
infrastructure can now be
independently scaled
Runtime environments
MQ environments
• As a service architectures apply to every type of runtime
– Physical Hardware
– Virtual machines
– Containers
• Management and provisioning layers provide the real benefits
– PureApplication
– Docker with Kubernetes/Swarm/…
– Many, many others…
Physical hardware
• IBM MQ Appliance
• A dedicated environment to easily and
quickly provision and configure queue
managers
• No external dependencies
– Simplifies provisioning
– Normalises behaviour
App
App
App
App
App
App
App
App
QM1
QM2
QM1
QM2
QM1App
App
App App
Virtual machines
• IBM supports MQ running in virtual machines
– Simply build your VM based on a supported MQ O/S
– Install MQ plus any required software and configuration
– Use and reuse, from development through to production
• Many benefits
– Simple replication of environments
• Using pre-canned, tried and tested, software stacks
– Isolation from host and other VMs
• Reduce risks from multi-tenancy
– Separation from physical servers
• More efficient utilisation of hardware
Virtual machine
Virtual machine
Virtual machine
Virtual machine
Virtual machine
Virtual machine
Operating system
Hypervisor
Virtual machine
Operating
system
Bins / libs
App App
Containers
• Containers provide a similar
environment to a VM but lighter in
weight
– A virtual machine provides an abstraction of the
physical hardware
– A container abstracts the OS level, typically at the
user level
• Linux containers
– Containers all share the same OS kernel
– Images are constructed from layered filesystems
– Containers isolate applications from each other and
the underlying infrastructure
Virtual machines
Operating system
Container
Bins / libs
App App
Container
App App
Bins / libs
Container
App App
Containers
Container
App App
Virtual machine
Operating
system
Bins / libs
App App
MQ in Docker
• Docker automates the deployment of applications inside
software containers
• MQ 8.0.0.4+ supported to run inside a Docker image.
Details: https://ibm.biz/mqdocker
• Brings the benefits of Docker to MQ
– Lightweight containers for running MQ
– Predictable and standardized units for deploying MQ
– Process, resource and dependency isolation
• IBM samples for customizing and building your own Docker images
– Best practice guidance
– Runs an MQ queue manager inside a container, isolated from the rest of your system
42
Persistent storage for containers and VMs
• Where VM or container storage is
ephemeral, alternative storage is
required
– Unless you are willing to create a fresh,
empty, queue manager every time the
container re-starts!
– Our example Docker files show the use
of Docker volumes for hosting the
queue manager data
• Storage options
– Local storage
• Simple but limits containers to a single system
– Networked block storage
• Relies on the container management to
reattach storage as required
– Networked filesystem
• Access across containers enables use of multi
instance queue managers and removes
requirement on container management
• MQ is sensitive to filesystem characteristics
The storage layer is critical to a queue manager’s performance, reliability,
availability and even identity
MQ ecosystem – what should we do next?
Architect
DevelopDeploy
Operate
44
Summary
• Self service will increase your agility and efficiency
• Define your best practices up front
• Design your MQ architecture to be deployed as-a-service
• Apply it to the runtime and orchestration framework of your choice
More sessions related to this topic this week:
Wednesday 2:30pm 6332 MQ Flow Automation & Self-service at Bloomberg
Thursday 8:30am 2931 Business agility through self service messaging
Thursday 11:30am 3537 Monitoring and managing hybrid messaging environments
IBM MQ early access programs
Interested in hearing about the future direction of MQ?
Want to influence the shape of features while they’re still on the drawing board?
Want access to early drivers?
Join any of the IBM MQ early programs
IBM MQ v.Next early program
IBM MQ Appliance early program
IBM MQ on HP Non Stop Server early program
Talk to your IBM contact, alternatively email [email protected] for further details
46
Where do I get more information?
IBM Messaging developerWorks
developer.ibm.com/messaging
IBM Messaging Youtube
https://www.youtube.com/IBMmessagingMedia
Ibm.biz/ibmmessaging
@IBMMessaging
IBM MQ Facebook
Facebook.com/IBM-MQ-8304628654/
Monday
10:30-11:30 3592 New MQ features
3452 Managing applications
12:00-13:00 2835 MQ on z/OS and Distributed
15:00-16:00 3470 Latest MQ z/OS features
2833 Where is my message?
3544 MQ Light in an MQ infrastructure
16:30-17:30 3573 Hybrid cloud messaging
2941 MQ Advanced
Tuesday
08:30-09:30 3540 The MQ Light API
12:00-13:00 3456 The IBM MQ Appliance
13:15-14:15 3499 Introducing Message Hub
3458 MQ Appliance administration
14:30-15:30 6432 MQ updates and futures (InnerCircle)
2849 Messaging feedback roundtable
16:00-17:00 3544 MQ Light in an MQ infrastructure
3513 MQ hands on lab
Wednesday
08:30-09:30 3602 Managing your MQ environment
12:00-13:00 3613 Designing MQ self service
6408 Hybrid messaging roadmap (InnerCircle)
13:15-14:00 3416 HA and DR with MQ
3433 Why secure your messaging?
15:45-16:30 3429 Securing MQ
2847 Meet the messaging experts
16:00-17:00 3508 MQ Light hands on lab
16:45-17:30 2275 Migrating to the IBM MQ Appliance
Thursday
08:30-09:15 3420 MQ Clustering
2931 Business agility with self service MQ
09:30-10:15 3479 MQ z/OS clusters and shared queue
3450 Optimising MQ applications
2849 Messaging feedback roundtable
10:30-11:15 3465 MQ Appliance high availability
3481 MQ z/OS messaging connectivity
11:30-12:15 3474 Active-active messaging
3537 Monitoring and managing MQ
3425 MQ publish/subscribe
Find us at the EXPO:
Hybrid Integration peds 65-68
Check out the Hybrid Messaging sub topic under the
Hybrid Integration topic for further customer and business
partner sessions
Hybrid Messaging from the IBM experts at InterConnect 2016
Sunday
14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)
Notices and Disclaimers
49
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission
from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial
publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED
"AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS
INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and
services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers
have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in
which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and
discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their
specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such
laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
Notices and Disclaimers Con’t.
50
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Thank YouYour Feedback is Important!
Access the InterConnect 2016 Conference Attendee Portal to complete
your session surveys from your smartphone,
laptop or conference kiosk.
