AAI-3218 Production Deployment Best Practices for WebSphere Liberty Profile

© 2015 IBM Corporation

Production Deployment Best Practices for WebSphere Liberty Profile

Erin Schnabel WebSphere Liberty Profile Lead [email protected]

Christopher VignolaWebSphere System Management [email protected]

Topics

• Profile Configuration

• Topologies & Practices

• Application Deployment

• Management Configuration

• Numbers and Sizes

2

Profile Configuration

3

Configuring Data Sources

• Use top-level libraries

• Allows multiple data sources and applications to share the same class loader

<library id="DB2JCC4Lib">

<fileset dir="C:/DB2/java"

includes="db2jcc4.jar db2jcc_license_cisuz.jar"/>

</library>

<dataSource id="db2" jndiName="jdbc/sampleDB">

<jdbcDriver libraryRef="DB2JCC4Lib"/>

<properties.db2.jcc databaseName="SAMPLEDB"

serverName="localhost" portNumber="50000"/>

</dataSource>

<application location="myApp.ear">

<classloader commonLibraryRef="DB2JCC4Lib"/>

</application>

4

Configuring Data Sources: Connections

• Connection pooling and timeouts

• Reduce connectionTimeout to confirm that max pool size is

being exceeded

• Increase maxPoolSize if you see connection timeouts

• If you don’t need two phase commit:

• Use ConnectionPoolDataSource rather than XADataSource

• Data sources can enlist in a global transaction as a one-phase

resource even if they are not XA-capable.

5

Configuring Data Sources: Connection sharing

• Set isolation level property instead of programmatically

• Use isolationLevel property on datasource

• Set in resource reference binding/extension

• Declared isolation level allows for better matching/sharing of

connections

• Use containerAuthData instead of user/password

• Prevents applications using res-auth=Application from

accessing the data source with container credentials

6

Threading and Thread pools

• Auto-tuning thread pool

• Optimizes for executor throughput (measured every 1.5 sec)

• Dynamically adjusts between coreThreads and maxThreads

– Default coreThreads = (2*processorCores)

– Default maxThreads = MAX_INT

• Not usually necessary to tune the thread pool

• Some workloads may warrant increasing coreThreads

• Long-running / Outbound-to-self

• Measure performance before and after adjustment

7

Separate App and Admin HTTPS traffic

• Configure Virtual Hosts to isolate the application from other

(internal/administrative) traffic.

8

<httpEndpoint id=“appEndpoint” … />

<!– Restrict access to default_host: only accessible via default

endpoint -->

<virtualHost id=“default_host”

allowFromEndpointRef=“defaultHttpEndpoint”/>



<virtualHost id=“applicationHost”

allowFromEndpointRef=“appEndpoint”>

<hostAlias>*:${app.http.port}</hostAlias>

<hostAlias>*:${app.https.port}</hostAlias>

</virtualHost>



<pluginConfiguration httpEndpointRef=“appEndpoint” />

server.xml:

<virtual-host name=”applicationHost" />

Ibm-web-bnd.xml

Virtual host

binding required in

the application

SSL Certificates and KeyStores

• Use only officially signed SSL certificates

• Use a separate keystore for inbound vs. outbound

• Encode or encrypt passwords in your configuration

<ssl id="defaultSSLConfig"

keyStoreRef="defaultKeyStore"

trustStoreRef="defaultTrustStore" />

<keyStore id="defaultKeyStore"

location="${server.config.dir}/key.jks"

type="JKS" password="{aes}..." />

<keyStore id="defaultKeyStore"

location="${server.config.dir}/trust.jks"

type="JKS" password="{aes}…" />

9

Security - General

• Harden your environment – extensive resources:

http://www.ibm.com/developerworks/websphere/techjournal/1210_lansche/1210_lansche.html


• Avoid vulnerabilities – keep service current.

• Register for support notifications at ibm.com/support .

10



Topologies& Practices

11

Topologies

• Standalone

• collective Assisted Lifecycle

• Collective

When to Use Which Topology?

• Standalone – small, isolated, or situational deployments

• collective Assisted Lifecycle

• Small/Med deployments

• Existing admin skill

• multi-server QoS

• Collective

• Small/Med/Large deployments

• Lightweight environment

• Multi-server QoS

13

Universal Practices

• Server Package Deployment

• High Availability – Rule of 3

• Automated deployment

• Secure environment

14

QoS Practices (Optional)

• Dynamic Routing

• Auto-scaling

• z/OS integration

15

Gold Standard – HA Collective

16

Collective Controller

Replica Set

CCCC

CC• Include AdminCenter

• Use Dynamic Routing

• Use Auto-scaling

Machine Boundary

AppServerAppServerLiberty

Profile

Per

App

IHSIHS

IHS

Per

Collective

z/OS Integration

• Use read-only mount point for servicing runtime

• Use PROCs for running servers

• Use START/MODIFY/STOP commands for server lifecycle

• Use SAF Registry support

• Exploit platform integration as necessary:

• zosWLM

• zosTransaction

• zosLocalAdapters

17

ApplicationDeployment

18

Build & Deploy

• Use server package for deployment – 3 models

• Developers direct deploy

• Developers hand off app, admin packages/deploys

• Developer direct deploy w/packaging automation to ensure

“approved server config”

>> Establish build/deploy pipeline

• Automate – use script, Chef, Urbancode, etc

19

Example Build/Deploy Pipeline

20

Jenkins

Liberty

Server Package Deployment: Managing Config

21

$WLP_USER_DIR/MyServer/

server.xml

apps/

MyApp.war

deploy (apply target overrides)

$WLP_USER_DIR/MyServer/

server.xml

configDropins/

overrides/

prod.xml

apps/

MyApp.war

Server Package Update

• Update through build process

• Blue/Green deploy

• Dual install locations

• Ripple start (stop old/start new, JVM by JVM)

• Delete old instance at your convenience

22

Upgrade – 1 of 5

23

host1.com

http/s: 9080/9443

status: STARTED

/wlp-blue/usr/servers/prod1

/wlp-blue/bin

/wlp-blue/lib

host2.com

http/s: 9080/9443

status: STARTED


/wlp-blue/bin

/wlp-blue/lib

Upgrade – 2 of 5

24

host1.com

http/s: 9080/9443

status: STARTED


/wlp-blue/bin

/wlp-blue/lib

host2.com

http/s: 9080/9443

status: STARTED


/wlp-blue/bin

/wlp-blue/lib

host1.com

http/s: 9080/9443

status: STOPPED

/wlp-green/usr/servers/prod1

/wlp-green/bin

/wlp-green/lib

host2.com

http/s: 9080/9443

status: STOPPED


/wlp-green/bin

/wlp-green/lib

Upgrade – 3 of 5

25

host1.com

http/s: 9080/9443

status: STOPPED


/wlp-blue/bin

/wlp-blue/lib

host2.com

http/s: 9080/9443

status: STARTED


/wlp-blue/bin

/wlp-blue/lib

host1.com

http/s: 9080/9443

status: STARTED


/wlp-green/bin

/wlp-green/lib

host2.com

http/s: 9080/9443

status: STOPPED


/wlp-green/bin

/wlp-green/lib

Upgrade – 4 of 5

26

host1.com

http/s: 9080/9443

status: STOPPED


/wlp-blue/bin

/wlp-blue/lib

host2.com

http/s: 9080/9443

status: STOPPED


/wlp-blue/bin

/wlp-blue/lib

host1.com

http/s: 9080/9443

status: STARTED


/wlp-green/bin

/wlp-green/lib

host2.com

http/s: 9080/9443

status: STARTED


/wlp-green/bin

/wlp-green/lib

Upgrade – 5 of 5

27

host1.com

http/s: 9080/9443

status: STARTED


/wlp-green/bin

/wlp-green/lib

host2.com

http/s: 9080/9443

status: STARTED


/wlp-green/bin

/wlp-green/lib

ManagementConfiguration

28

High Availability Services

• Collective scope

• Built-in to collective controller replica set

• Host scope (e.g. scalingMember-1.0 feature)

• Uses local port

29

Collective SSL

• Collective root, member root

• First controller establishes “true root”

• Must be copied to subsequent replicas

• Trust between

• Controllers and members

• Members and controllers

• Members on same host

• Certificates

• Signers

• Identity

30

Security

• Use external user registry – e.g.

• Ldap

• SAF

• Multiple registries allowed – e.g. as granular as per cluster.

• Collective Controller Replica set must use same registry.

31

Highly Available Collective Controller

• Three controllers minimum. Odd numbers only.

• Up to 2000 member per controller

Recommended: Max(Members/2000,3)

+1(for controller failure)

+1(for network partition)

• Use configDropIns directory – it’s replicated!

• Configure members with controller failover addresses.

32

CCCC

CC

Dynamic Routing

• Requires IHS (or Apache).

• Use dynamicRouting-1.0 feature in controllers.

• Double-layer IHS to simplify firewall management.

• Terminate SSL at earliest opportunity.

33

CCCC

CC

IHSIHS

IHS

Auto-scaling

• Put scalingController-1.0 feature in at least 3 controllers.

• Set hostSingleton port for vertical scaling.

• Set min instances based on average demand.

34

AppServerAppServer

AppServer

Elastic

Resources

CCCC

CC

Auto-scaling Policy

• Leave headroom on max settings (e.g.

• CPU <= 90%

• Heap <= 90%

• Memory <=90%

• Include scaling policy in server package

• scaling-metadata.xml

35

Use Admin-metadata

• Owner

• Contacts

• Note

• Tags

• admin-metadata.xml – part of server package

• assignable to host, server, application, cluster, runtime

• There is also an API

• Used in AdminCenter to search and set views

36

Numbers and Sizes

37

38

Liberty Profile Collective Size Design Considerations

• Collectives are design for large scale

• What limits the size of a WebSphere Collective?

• Breadth and currency for shared Information across controller JVMs

• Communication and coordination of shared information

• Product features scale differently with large collective size

• Most affected : collectiveController, scalingController

• It is possible to create a large topology collectiveBut will it function to your requirements?

• Collective works well with defaults settings. Some environments may

require tuning.

39

When do you need multiple collectives?

• Isolation

• Development vs testing vs production

• Critical applications

• Backup site

• Lines of business

• Each funding area may have different policies for when to apply

fixes, when to upgrade

• Geography:

• Controllers can span data centers with qualifying* network and config

• Members can span data centers with tuning

• Large collectives require planning to avoid “urban sprawl”

40

How Large a collective can I create?

• No hard limit – trust, but verify

• Lab tested:

• Tested 10,000 members

• 5 controllers

• 50 VMs

• ~200 members per VM

• Controller VMs: 20GB memory+6 CPUs

• Member VMs: 64GB+16 CPUs

41

How Many Host OS Instances per collective

• No Design Limit

• Typical large topology up to hundreds of hosts.

• Practical Limits

• Operations may take longer:

– Configurations, server deployment

• Notifications flowing back to collective controllers

• Load on controllers with concurrent operations

• 50 hosts per collective controller guideline

42

How Many Application Servers per Host OS?

• Keep WAS JVMs completely within physical memory

• Allow for overhead

• Process footprint about 1.25 to 1.5X maximum heap size for 32 bit

heaps

• 1.6x to 1.8x for 64-bit heaps

• If App Server gets swapped out, add more memory or

else move to different host OS.

• Ensure sufficient CPU is allocated

• Especially for hypervisor

• Avoid CPU starvation

43

How Many Applications Per Server

• Balance between resource usage and isolation

• With one application per server

• One bad application does not bring down all other applications

• Easier to tune each application in isolation

• With more than one application per server

• Less resources: cost of application server runtime amortized

across multiple applications

• Smaller topology to manage

• Configure as much isolation as you can afford

• If you have 300 applications clustered on 3 nodes

– Complete isolation 900 JVMs

– Complete sharing 3 JVMs

44

System Management Best Practices

• Set and track your performance goals• Measure performance of your most commonly used operations

• Track changes as you increase collective size

• Track changes over time to identify new issues

• Use Scripting (or your own Java framework)• Automatable, repeatable, testable

• Don’t overload collective controllers• Give enough memory + CPU

• Run AdminCenter in multiple controllers to spread load

• Don’t co-locate with resource intensive processes, e.g., application servers with heavy load

Wrap Up

Covered

• Topologies & Practices

• Profile Configuration

• Application Deployment

• Management Configuration

• Numbers and Sizes

46

Notices and Disclaimers

Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or

transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with

IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been

reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM

shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY,

EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF

THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT

OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the

agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without

notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are

presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual

performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,

programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not

necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither

intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal

counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s

business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or

represent or warrant that its services or products will ensure that the customer is in compliance with any law.

Notices and Disclaimers (con’t)

Information concerning non-IBM products was obtained from the suppliers of those products, their published

announcements or other publicly available sources. IBM has not tested those products in connection with this

publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM

products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to

interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,

INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A

PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any

IBM patents, copyrights, trademarks or other intellectual property right.

• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document

Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,

ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,

PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,

pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,

urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of

International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and

service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on

the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

http://www.ibm.com/legal/copytrade.shtml

Thank YouYour Feedback is

Important!

Access the InterConnect 2015

Conference CONNECT Attendee

Portal to complete your session

surveys from your smartphone,

laptop or conference kiosk.

Software

AAI-3218 Production Deployment Best Practices for WebSphere Liberty Profile