© 2012 IBM Corporation

DevOps for the Mainframe Leveraging Continuous Integration, the cloud, and beyond to deliver z/OS applications

David Myers, Product Manager, Continuous Delivery for Mainframe and Mobile Software Rosalind Radcliffe, Distinguished Engineer, Enterprise Modernization Solution Architect

2

Please note

IBM’s  statements  regarding  its  plans,  direc3ons,  and  intent  are  subject  to  change  or  withdrawal  without  no3ce  at  IBM’s  sole  discre3on.  Informa3on  regarding  poten3al  future  products  is  intended  to  outline  our  general  product  direc3on  and  it  should  not  be  relied  on  in  making  a  purchasing  decision.    The  informa3on  men3oned  regarding  poten3al  future  products  is  not  a  commitment,  promise,  or  legal  obliga3on  to  deliver  any  material,  code  or  func3onality.  Informa3on  about  poten3al  future  products  may  not  be  incorporated  into  any  contract.  The  development,  release,  and  3ming  of  any  future  features  or  func3onality  described  for  our  products  remains  at  our  sole  discre3on.  

Performance  is  based  on  measurements  and  projec3ons  using  standard  IBM  benchmarks  in  a  controlled  environment.    The  actual  throughput  or  performance  that  any  user  will  experience  will  vary  depending  upon  many  factors,  including  considera3ons  such  as  the  amount  of  mul3programming  in  the  user’s  job  stream,  the  I/O  configura3on,  the  storage  configura3on,  and  the  workload  processed.    Therefore,  no  assurance  can  be  given  that  an  individual  user  will  achieve  results  similar  to  those  stated  here.  

3

3

Social, Local, Mobile Smart Infrastructure Analytics

ERP  Legacy  DB  

CRM   HR  

Manage workloads and maintain security

Rapid innovation in the cloud

User experience and mobile management

New era systems integrate existing operational systems with rapid delivery of new client-facing apps

4

Evolving customer and market expectations

Capabilities and User Experience Today Emerging

Primary Workload Types Systems of Record Transactional

Systems of Engagement (+ Record) Big Data, Analytics, Mobile/Social Channels

Time to Value Planned Opportunistic

Delivery Model Planned Incremental (DevOps)

Development and Operations Team Sizes 100s and Costly 10s with built-in DevOps automation

Release Frequency Months to Years Hours to Days, based on business opportunity

Integration Frequency Weeks Continuous

Service Sourcing Develop Consume and Assemble (Public and Private)

Operational Model Systems Management Built in to application, Recovery Oriented Computing, Continuous Availability

Infrastructure Deployment Days Minutes

Risk Profile Big-Bang (High Risk) Incremental

5

DevOps is…

A set of principles and values that facilitate collaboration across disciplines to… 1. Enable rapid evolution of deployed business services 2. Reduce risk, decrease cost, and improve quality across the portfolio

DevOps Principles §  Collaborate across disciplines §  Develop and test against a production-like system §  Deploy frequently using repeatable and reliable

processes §  Continuously monitor and validate operational quality

characteristics §  Amplify feedback loops

People    

Process  

Tools  

6

Solution: A Continuous Delivery Pipeline

Development Testing Staging Production Business Owners Customers

•  Validate  on  more  produc3on-­‐like  condi3ons  earlier  

•  Automate  hand-­‐offs/promo3ons  to  increase  velocity  through  the  different  stages  

•  Standardiza3on  on  processes  and  assets  between  Dev  and  Ops  

•  Automated  monitoring  and  dashboarding  of  quality  and  performance  against  service  level  agreements  at  mul3ple  stages  

Ensure applications are production-ready throughout the lifecycle and can be released at any time while minimizing rollback due to quality issues

7

Mainframe Delivery Pains…

§  Multiple teams working across restricted dev and test capacity lead to conflict, delays, or bad test results in shared environments

§  Complex and manual

management and configuration tasks result in errors and delays

§  Too much bad code going into test and production causes crit sits and emergency fixes

§  Bottlenecks due to inefficient

communications between disparate platforms and teams (Dev/Test - System Programmers; mobile – distributed-mainframe)

8

Go  on  Offense  

Play  Defense  

…solutions from IBM

§ Provide  cheap,  isolated,  development  and  test  environments  for  project  teams  

– Ra3onal  Development  and  Test  Environment  – Ra3onal  Test  Virtualiza3on  Server  – SmartCloud  Provisioning    – Cloud  Ready  for  Linux  on  System  z    

§ Automate  consistent  build,  configure,  and  deploy  processes  across  all  stages    

– Ra3onal  Team  Concert  – SmartCloud  Con3nuous  Delivery  – SmartCloud  Orchestrator  

§ Enforce  base  quality  standards  automa3cally  prior  to  promo3on  

– Ra3onal  Test  Workbench  – Ra3onal  Quality  Manager  – SmartCloud  Applica3on  Monitoring  – Omegamon    

§  Improve  communica3on  and  collabora3on  with  cross-­‐plaVorm  release  planning  

–  IBM  Collabora3ve  Lifecycle  Management  – Smart  Cloud  Control  Desk  

9

Test  Test  

Dev  Dev  

1  

DevOps for the common mainframe delivery cycle

Dev   Test   Staging   Prod  

Development" Test" System Programmer"

3  

2   2  

4  

10

10 10

Test LPAR

z/OS

…

Typical z/OS Testing Architecture Organized by project team, vertically scaled, sharing resources, limited automation

Project Team [April Maintain]

Project Team [Prototype SOA]

Project Team [June New Func]

Project Team [Dec Sys Upgrade]

Test  Data  

App

App

App

Problems Encountered

1. Shared resources combined with overlapping schedules can elicit conflicts, impede innovation and slow code delivery

2. Coordination of environmental changes and releases cause bottlenecks, delays and additional overhead

3. Shared test data is difficult to manage and can lead to over testing or incorrect test results

11

The ultimate in modern application development for System z

§  Liberate  developers  to  rapidly  prototype  new  applica3ons  §  Develop  and  test  System  z  applica3ons  anywhere,  any3me!  §  Free  up  mainframe  development  MIPS  for  produc3on  capacity  §  Eliminate  costly  delays  by  reducing  dependencies  on  opera3ons  staff    Note:  This  Program  is  licensed  only  for  development  and  test  of  applica3ons  that  run  on  IBM  z/OS.  The  Program  may  not  be  used  to  run  produc3on  workloads  of  any  kind,  nor  more  robust  

development  workloads  including  without  limita3on  produc3on  module  builds,  pre-­‐produc3on  tes3ng,  stress  tes3ng,  or  performance  tes3ng.      

RDz & ISPF user

ISPF user RDz user

RDz user RDz user

   

COBOL,  PL/I,  C++,  Java,  EGL,  Batch,    Assembler,  Debug  Tool  

           x86  PC  running  Linux  

RDz user

IMS

z/OS

WAS

DB2

MQ CICS

Rational Development and Test Environment for System z

12

Simulate subsystem dependencies –  Test impact of latency –  Test application response to unresponsive

services Create a stable test environment –  Simulate “public” services –  Reduce capacity requirements on

infrastructure –  Improve security by reducing access from

external services

Improve  tes+ng  with  Green  Hat  service  simula+on  

Decouple subsystem teams using Green Hat

Databases Mainframe applications

Third-party Services

Ra3onal  Test  Virtualiza3on  Server  

Application changes being tested

virtualized services

13

Testing with dependency virtualization

Phase 1 Phase 2 Phase 3

App  

Test  Case  

Virtual  Services  

3rd Party Call

IMS Data Access

CICS Commarea Call

App  

Test  Case  

Virtual  Services  

3rd Party Call

IMS Data Access

CICS Commarea Call

App  

Test  Case  

Virtual  Services  

3rd Party Call

IMS Data Access

CICS Commarea Call

Controlled large system testing by isolating components under test •  Easier problem determination •  Lower test environment capacity requirements •  Improved component quality

14

Provision new dev / test systems in minutes •  Fast  VM  provisioning  for  near-­‐instant  deployment  of  100s  of  virtual  machines  and  scale  based  on  business  needs  

•  Rich  set  of  web  interfaces  into  the  cloud  that  can  be  driven  by  a  user  or  scripts  for  complete  automa3on  

•  Reliable,  non-­‐stop  cloud  capable  of  automa3cally  tolera3ng  and  recovering  from  soeware  and  hardware  failures  

•  Near-­‐zero  down=me  due  to  faults,  hypervisor/management  soeware  upgrades  or  addi3on/removal  of  hardware  

Requested  VMs  will  be  up    and  running  under  a    

minute  using  standard  HW  

Func3on  Test   Performance  Test  

15

§  Fast, dependable, automatic feedback speeds time to market §  Lower cost of application testing using off-mainframe z/OS test environment §  Enables confidence by automatically tracking and promoting code health

Continuous Integration for z

Developer Frequent, rapid feedback earlier to improve quality

   

Rapid  Feedback  

Code  Check  In  +  Build  +  Deploy  +  Run  

Automated  Tests    

Change    Request  

Automated, continuous testing with isolated development and test environment

Reusable integration tests simplify testing across layers

Promote  To  QA  

Higher quality going into QA

Reduced delivery time, end-to-end visibility of test activities, safer and faster upgrades (V2V)  

Fail fast and drive better quality downstream

16

System z

Continuous Integration for System z Scenario

16

SCM/CI Server

(Rational Team Concert)

Test Environment (RD&T or LPAR)

Test Execution Manager (Ra3onal  Quality  

Manager) Test

Automation (Rational Test Workbench) 1.  Check-in code

2.  Build code and zUnit tests

3.  Deploy build results and test data to Test Environment

4.  Execute zUnit Tests

<Deploys App+zUnit Tests> <D

rives>

5.  Kick-off Automated Test Plan 6.  Run automated

interface tests against Test Environment

7.  Mark execution records Pass/Fail in Test Execution Manager

<Starts Tests>

8.  Report test results in dashboard/build results/defect records in CI server.

<Results>

17

§ Each tool came with its own •  UI - Web and desktop

presentations of views and tasks

•  Logic – Workflow, process, search, query, scale, security and collaboration

•  Storage – Availability, traceability

•  Privacy, backup/archive § Resulting in...

•  Brittle integrations •  Silos everywhere •  High cost to maintain and

administer •  Proprietary API's

Tool E SCM

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

UI

LOGIC

DB

Tool B Release Planning

Tool D Incident

Management

Tool C Reporting

Tool A Dev work Items

Separate tools makes collaboration inefficient or difficult...

18

Tie development release to operational release

Ra3onal  Team  Concert  Release  Plan  and  work  items  

SmartCloud  Control  Desk    

19

19 19

Delivery Organized for Flexibility and Speed Organized by application team, horizontally sliced, dedicated resources, highly automated

Problems Encountered 1.  Shared resources

combined with overlapping schedules can elicit conflicts, impede innovation and slow code delivery

2.  Coordination of environmental changes and releases cause bottlenecks, delays and additional overhead

3.  Shared test data is difficult to manage and can lead to over testing or incorrect test results

4.  Provisioning, managing, and synchronizing project test environments including data

QA/Integra3

on  Test  LPA

R  

z/OS LPAR

…

Project Team [April Maintain]

Project Team [Prototype SOA]

Project Team [June New Func]

Project Team [Dec Sys Upgrade]

Integr.  Test  Data  

LPAR

Data  

RD&T

Data  

RD&T

Data  

LPAR

Data  

20

Deploy application and config changes to test environments in minutes

IBM Internal and Business Partner Use Only

IBM SmartCloud Provisioning

Opera+ons  team  establishes  deployment  pa;erns  for  use  by  development  and  test  teams  

Development  team  makes  applica+on  code  and  configura+on  changes  

Deliver Environments 3x

Faster!!

IBM Workload Deployer

Test against production-like environments with Applications & middleware AUTOMATICALLY installed and properly configured

IBM SmartCloud Continuous

Delivery

21

What are we working on in the lab…

22

Go  on  Offense  

Play  Defense  

…solutions from IBM

§ Provide  cheap,  isolated,  development  and  test  environments  for  project  teams  

– Ra3onal  Development  and  Test  Environment  – Ra3onal  Test  Virtualiza3on  Server  – SmartCloud  Provisioning    – Cloud  Ready  for  Linux  on  System  z    

§ Automate  consistent  build,  provision,  configure,  and  deploy  processes  across  all  stages    

– Ra3onal  Team  Concert  – SmartCloud  Con3nuous  Delivery  – SmartCloud  Orchestrator*  

§ Enforce  base  quality  standards  automa3cally  prior  to  promo3on  

– Ra3onal  Test  Workbench  – Ra3onal  Quality  Manager  – SmartCloud  Applica3on  Monitoring  – Omegamon    

§  Improve  communica3on  and  collabora3on  with  cross-­‐plaVorm  release  planning  

–  IBM  Collabora3ve  Lifecycle  Management  – Smart  Cloud  Control  Desk  

Cloud-­‐style  deploy  of  zLinux  and  z/OS  dev  and  test  paMerns  

on  RD&T  or  zHardware  

23

Infrastructure  as  Code  

Source  Control    Management  

   Source  Ar3facts  

.cbl  

.asm  

.cpy  

CLIST  .rexx  

Provisioning using Cloud

#!/usr/bin/env ruby

class DevopsDeployer def initialize(build_url, build_id) @log = Logger.new(LOG_FILE) @log.level = LOG_LEVEL @iaas_gateway = IaasGateway.new(HsltProvider.new(),LOG_FILE, LOG_LEVEL) @server_instance = nil rtc_build_system_provider = RtcBuildSystemProvider.new(RTC_REPOSITORY_URL, RTC_USER_ID, RTC_PASSWORD_FILE) @build = rtc_build_system_provider.resolve_build(build_url, ENV['buildResultUUID'], build_id) @build_system_gateway = BuildSystemGateway.new(rtc_build_system_provider, LOG_FILE, LOG_LEVEL) end def add_build_stamp template_file = WEB_APP_ROOT + "/app/templates/pages/page.html" @log.info "Adding build ID stamp #{@build.id} to \#{template_file}" # Read in the file's contents as a string, replace # the build_id, then overwrite the original contents # of the file text = File.read(template_file) new_text = text.gsub(/\{\{ build_id \}\}/,"<a href=\"#{@build.uri}\">#{@build.id}</a>") File.open(template_file, "w") { |file| file.puts new_text } end

# ...

24

Delivery Pipeline Fundamentals

Deploy  

Environment  Running  System  

Source  Control    Management  

   Source  Ar3facts  

.cbl  

.asm  

.cpy  

CLIST  .rexx  

Deployable  Ar3facts  

Build,  Package,  &  Unit  Test  Applica+on  Binaries  &  PlaGorm  Configura+on  

Library  

25

Consolidate, standardize, simplify deployment

COBOL  

Java  

Copybooks  

WSDL  

DB2  DDL  IMS  DBD/PSB  

HTML  

MQ  Queue  Def  

DB2  Plans  

Procs  

JCL  

Deploy  

Environment  Running  System  Deployable  Ar3facts  

Build,  Package,  &  Unit  Test  Applica+on  Binaries  &  PlaGorm  Configura+on  

Library  

Enterprise  Applica3ons  have  thousands  of  disparate  parts  •  Currently  maintained  in  separate  systems,  if  maintained  at  all  •  Limited  linkage  between  systems  for  applica3on  and  configura3on  dependencies  •  Missing  assets  and  informa3on  is  rampant    DevOps  force  linkages,  automa3on,  and  standardized  packaging…  

Source  Management  

26

Create, secure, and deploy test data

2TB

-Subset

Production or Production Clone

-Compare -Refresh

Create “right-size” production-like environments

for application testing De-identify sensitive information with realistic but fictional data for testing & development purposes

De-identify sensitive information with realistic but fictional data for testing & development purposes

z/OS  

UT  RD&T  

Automated test data deployment for each

build and test

1 GB

z/OS  

UT  LPAR  

1 GB Masked Test Data

Automated Build & Deploy Test Data

Automated Testing

Optim RTC

Test Data

27

Standardize z/OS region configurations to ease deployment

•  Standard topologies exist today (production LPARs)

•  Standardized regions which can be repeatedly deployed are rare

•  Standardized/Automated deployment of COMPLETE system is spotty

To adopt the cloud for testing… Standardize and automate provisioning of everything …

Standard  Middleware  Palern  

CICS  5.1   IMS  11.1  

DB2  10.1  Z/OS  1.12  

Configura3on  

Data  

Applica3on  

28

RTC

/CIz

Ava

ilabl

e C

apab

ilitie

s

Standardize z/OS region configurations to ease deployment

Standard  Middleware  Palern  

CICS  5.1   IMS  11.1  

DB2  10.1  Z/OS  1.12  

Configura3on  

Data  

Applica3on  

Pattern/Platform

Application load modules

Test Data

JCL and REXX

29

Evolving to deploying regions … cloud-style

“Cloud” RD&T Or LPAR

RD&T Or LPAR

RD&T Or LPAR

RD&T Or LPAR

SmartCloud  Orchestrator  +  

<SCD  for  Mainframe  Project>  

Region/Middleware  Palerns  

CICS+DB2  

CICS+  IMS  DB  

Batch  

30

Acknowledgements and Disclaimers:

     ©  Copyright  IBM  Corpora6on  2013.  All  rights  reserved.  

§  U.S.  Government  Users  Restricted  Rights  -­‐  Use,  duplica6on  or  disclosure  restricted  by  GSA  ADP  Schedule  Contract  with  IBM  Corp.  §  Please  update  paragraph  below  for  the  par+cular  product  or  family  brand  trademarks  you  men+on  such  as  WebSphere,  DB2,  Maximo,  

Clearcase,  Lotus,  etc  

 IBM,  the  IBM  logo,  ibm.com,  [IBM  Brand,  if  trademarked],  and  [IBM  Product,  if  trademarked]  are  trademarks  or  registered  trademarks  of  Interna3onal  Business  Machines  Corpora3on  in  the  United  States,  other  countries,  or  both.  If  these  and  other  IBM  trademarked  terms  are  marked  on  their  first  occurrence  in  this  informa3on  with  a  trademark  symbol  (®  or  ™),  these  symbols  indicate  U.S.  registered  or  common  law  trademarks  owned  by  IBM  at  the  3me  this  informa3on  was  published.  Such  trademarks  may  also  be  registered  or  common  law  trademarks  in  other  countries.  A  current  list  of  IBM  trademarks  is  available  on  the  Web  at  “Copyright  and  trademark  informa3on”  at  www.ibm.com/legal/copytrade.shtml  

 If  you  have  men3oned  trademarks  that  are  not  from  IBM,    please  update  and  add  the  following  lines:    

 [Insert  any  special  3rd  party  trademark  names/alribu3ons  here]      Other  company,  product,  or  service  names  may  be  trademarks  or  service  marks  of  others.  

   Availability.    References  in  this  presenta3on  to  IBM  products,  programs,  or  services  do  not  imply  that  they  will  be  available  in  all  countries  in  which  IBM  operates.    

                   The  workshops,  sessions  and  materials  have  been  prepared  by  IBM  or  the  session  speakers  and  reflect  their  own  views.    They  are  provided  for  

informa3onal  purposes  only,  and  are  neither  intended  to,  nor  shall  have  the  effect  of  being,  legal  or  other  guidance  or  advice  to  any  par3cipant.    While  efforts  were  made  to  verify  the  completeness  and  accuracy  of  the  informa3on  contained  in  this  presenta3on,  it  is  provided  AS-­‐IS  without  warranty  of  any  kind,  express  or  implied.  IBM  shall  not  be  responsible  for  any  damages  arising  out  of  the  use  of,  or  otherwise  related  to,  this  presenta3on  or  any  other  materials.  Nothing  contained  in  this  presenta3on  is  intended  to,  nor  shall  have  the  effect  of,  crea3ng  any  warran3es  or  representa3ons  from  IBM  or  its  suppliers  or  licensors,  or  altering  the  terms  and  condi3ons  of  the  applicable  license  agreement  governing  the  use  of  IBM  soeware.  

           All  customer  examples  described  are  presented  as  illustra3ons  of  how  those  customers  have  used  IBM  products  and  the  results  they  may  have  achieved.    Actual  environmental  costs  and  performance  characteris3cs  may  vary  by  customer.    Nothing  contained  in  these  materials  is  intended  to,  nor  shall  have  the  effect  of,  sta3ng  or  implying  that  any  ac3vi3es  undertaken  by  you  will  result  in  any  specific  sales,  revenue  growth  or  other  results.  

31

© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

www.ibm.com/software/rational