Building a z/OSMF Workflow-Driven InstallEd Jaffe

Phoenix Software International

November 2019

Session BM

Why Consider z/OSMF for Installs?• There are many highly-rated z/OS product installation strategies.

• Despite that, there are problems:• Every vendor’s product installer is unique. IBM has ServerPac (for z/OS release-

boundary installs) and CBPDO for other situations. Each ISV has its own installer –sometimes multiple installers across their product line. Some are better than others, but the real issue here is a lack of standardization.

• 3270s give an unflattering impression of z/OS to the younger generation. Despite their simplicity, they are often perceived much like a rotary phone. Sitting down at such a terminal is like going into a “time warp” back before you were born!

• Traditional z/OS installs often consist of dozens of manual steps. Contrast this with a Windows-based install of something like “Microsoft Office” where you answer a few simple questions, watch it for a minute or so, and VOILA it’s installed.

• z/OSMF Software Management and Workflows provide a path/opportunity to eventually help mitigate these issues.

2

z/OSMF Software Management and Workflows

• Software Management first appeared in z/OS 1.13 as Software Deployment.

• It was renamed to Software Management in z/OS 2.1 (with Deployment now a sub-function of Software Management), and has been steadily improved with an eye on replacing ServerPac and CBPDO.• It implements the notion of a Software Instance (SWI) which is a collection of files

comprising a software product. The product does not need to be SMP/E maintained.• An SWI may be exported to a Portable Software Instance (PSWI) which can be

transmitted across networks. The SWI is reconstituted by deploying the PSWI.• Early this year IBM delivered functionality to optionally launch an embedded

workflow after an SWI is deployed.

• Workflows first appeared in 2013 with z/OS 2.1.• Initially supported instructions and batch job submission only.• When immediate execution steps, feedback, and other features were added via

continuous delivery late in the z/OS 2.2 timeframe, it started looking interesting.• Recent enhancements include array variables. 3

Where Do Workflows Fit Into the Strategy?

4

Provider Web SiteAdd PSWI

Deploy PSWI

Post-Deploy Workflow

_step1_

_stepn_

Provider Web SiteAdd PSWI

Deploy PSWI

Post-Deploy Workflow

_step1_

_stepn_

Installation Workflow Instead of

Existing Installation

Manual

_step1_

_stepn_

Available Today Our Hopeful Vision for the Future

Provider Web Site

Workflow Components• Prologue

• Parent steps and leaf steps

• Content for each step• Name, Title & Description• Instructions (shown at “Perform” time)

• Metadata for each step• Dependencies (prerequisites and conditions)• Weight and Various Flags• Variable names, descriptions, help pop-ups

• Program templates (support variable substitution)• For JCL you have a batch job• For TSO-REXX-JCL, TSO-REXX, and TSO-UNIX-REXX you have a REXX exec• For shell-JCL and TSO-UNIX-shell you have a shell script

5

Convert from existing product documentation

Existing Product Documentation• Our existing product documentation lives in Microsoft Word documents.

• From that base we create and distribute Adobe PDF, Raw HTML, and Eclipse Plug-Ins intended for use with IBM Knowledge Center for z/OS (KC4z).• KC4z has been around over four years (since z/OS 2.2).• If you haven’t deployed it yet, shame on you!• If you’re an ISV and you don’t provide plug-ins for it, shame on you!

• We convert the Word documents to these other formats using helpful software (MadCap Flare) and self-authored scripts.

• Ideally, the Installation Workflow should eventually be able to completely replace the existing installation documentation. Therefore, it should look every bit as good as the other formats. It should support colors, sizeable fonts, embedded graphics, internal links, external links, and everything else that makes modern documentation usable.

6

7

WORD

8

PDF

9

HTML

10

KC4z

z/OSMF Workflow Result• Unlike the other formats, which

are documentation only, the workflow is a tabbed display.

• Some tabs like Details, Dependencies, Status and Input Variables contain metadata that might be of interest to the sysprog user.

• The Notes tab lets you author and keep your own notes associated with this step.

• Perform actually “runs” the step.

• Feedback lets you answer survey questions to provide useful feedback about your experience.

11

Our Content Transformation Approach

• It is possible to save a Word document as a Word XML Document.

• That’s the format we probably would have used if we weren’t already using MadCap Flare to help us create the HTML and Eclipse versions of our product documentation. No reason to believe it wouldn’t work just as well for you.

• Instead, we had MadCap Flare convert the native Word document into XHTML documents – one for each chapter in the book.• What is XHTML?

• From Wikipedia: XHTML documents are well-formed and may therefore be parsed using standard XML parsers, unlike HTML, which requires a lenient HTML-specific parser.

• We then ran a self-authored script (a Windows BAT file) against the XHTML files to transform them into XML files that are compatible with z/OSMF workflows.• That script is here: ftp://www.phoenixsoftware.com/pub/demo/workflow.bat

12

Our Content Transformation Approach

• Workflows do not support style sheets (CSS). That is a MAJOR restriction for anyone attempting to directly author content using an XML editor such as the z/OSMF Workflow Editor. However, since our approach is to script a conversion from Word, it’s really not terrible at all.

• The script uses the sed utility (originally from UNIX) on Windows to do the necessary transformations. Here’s what one of the commands looks like to convert our “Railroad” style (used to document command syntax) into HTML:sed -i -E "s/<pre Railroad><code>/<pre style=\"padding:0;\"><code

style=\"font-family: 'Courier New', Courier, monospace;font-

size:100%%;border:none;background-color:#ffffff;\">/g" ..\*Chapter*.xml

• At first we ran into an unacceptable number of restrictions. Generally, only simple HTML is supported by the Workflow and the Workflow Editor. We found this extremely disappointing because the published result looked like we were in an “HTML for Dummies” class learning the basics. Yours will too…

13

Our Content Transformation Approach

• Then we discovered the <![CDATA[some stuff]]> XML tag is accepted!• CDATA stands for Character Data and it means that the data in between the innermost

brackets includes data that could be interpreted as XML markup, but should not be.

• It looks a bit like an XML comment, but isn’t. It’s actually part of the document.

• It’s not perfect because there is no way to escape the CDEnd sequence ]]>. Just don’t use that anywhere and you should be AOK.

• By wrapping all of our content in the CDATA tag, we were able to convince Workflow and Workflow Editor to ignore (and pass along) HTML tags they previously rejected as not supported.

• Of course, the browsers understand those tags and that’s what matters most.

• It would be best if Workflow and Workflow Editor were enhanced to natively understand everything, but what’s there now is still very usable, especially for a scripted conversion situation.

14

Our Content Transformation Approach• We struggled with images for a while.

• When you “create” a workflow, you are actually copying it from whichever source location you specify into some undisclosed location deep inside z/OSMF.

• All file references are relative to that undisclosed location rather than the original source location. Explicit file references suffer “out of zone” security errors.

• Therefore, the only workable approach for images is to embed them as base-64 text within the workflow. Again, the trusty CDATA tag allows this magic to occur.

• This restriction is not all bad. There are advantages to having a single, self-contained workflow file that isn’t dependent on anything else. But it can become quite large if you have many images. It’s also cumbersome to edit the XML.

• Anyone brave enough to include video might ultimately choose to host it on the Internet instead of embedding it.• The downside to Internet-based ancillary information is that access might not be allowed in

all situations or at all customer sites.

15

Our Content Transformation Approach

16

Our Metadata Creation Approach• The Workflow Editor does a pretty good job with most of the metadata

• Beware! It has been shown to strip the CDATA tags around non-complex content if it thinks it understands everything therein.

• This does not alter what the end user sees, but can cause confusion for anyone attempting to later inspect/edit the content XML directly. You might need to replace it with newly-converted text.

• Other than within a chapter, we generally implement dependencies on prior chapters (parent steps) rather than on any singular leaf steps within them.

• Within a chapter, we generally make each leaf step dependent on the prior one unless we want to allow parallel steps.

• We generally try to set the weight of a step that actually performs an action (submits a job, runs a script) ten times higher than one that is instructions only.

• We use simple, hand-written HTML for our variable descriptions and help/information pop-ups. You really don’t need anything fancy there.

17

Value Choice Variable• Creates a drop-down list of choices

18

Value Choice Variable and Help Pop-Up• This choice populates the workflow instance variable downloadMethod with the

chosen string in the drop-down list. It can be used for step conditions, in templates (JCL, REXX execs, shell scripts), etc.

19

• Clicking on the blue information bubble produces this pop-up

String Variable• Creates an ordinary entry field

20

Boolean (YES or NO) Choice Variable• Creates a check-box (checked = TRUE)

21

String and Boolean VariablesHelp Pop-ups

22

Defining a Conditional Step• You need conditions in addition to prerequisites

• In this case variable installPath controls the state

23

Types of Steps• A File or Inline Template lets you use an Apache

Velocity Template alone or to run a batch job, REXX exec, or z/OS UNIX shell script.• A File Template is external to the workflow.• https://velocity.apache.org/engine/2.0/user-guide.html

• A REST API call allows you to use services that are defined and available in your instance of z/OSMF or outside servers whose URLs you know.

• Instructions Only is, as it’s name suggests, a step that educates you about something or asks you to perform some action manually.• When you first convert a book into a workflow, all steps

will be Instructions Only.

• A Calling Step is how you launch one workflow as a subroutine to another. I have not yet tried this.

24

Template Steps Run Batch, REXX or Shell• Not specified uses template functions only

• JCL is a batch job

• TSO-REXX-JCL is a batch wrapper around a TSO/E REXX exec

• shell-JCL is a batch wrapper around a z/OS UNIX shell script

• TSO-REXX is an immediate execution of a REXX exec under TSO/E

• TSO-UNIX-REXX is an immediate execution of a REXX exec running in the OMVS z/OS UNIX environment

• TSO-UNIX-shell is an immediate execution of a shell script running in the OMVS z/OS UNIX environment

• You can have inline or file templates

• The templates can have embedded workflow variables.

• You can save the results as a data set or z/OS UNIX file

25

Some Tips and Techniques for Scripts• Scripts complete normally only when they produce the message defined as

“Message for Program Success.”• Scripts complete with failure if they produce any of the messages defined as

“Message for Program Failure.”• You can’t do ANYTHING in your script after you produce one of these messages.

Workflow pulls the “rug” out from under you. So make sure message issuance is the very last thing the script does. (We put a “dummy” EXIT instruction after.)

• Scripts that produce none of the defined messages will time out and fail that way.• Workflow variables can be prompted for and used in the same step.• If you wish to use ISPF services, be sure to allocate ISPLOG to DUMMY or any

action you take will cause prompting for log data set disposition at ISPF session termination causing your script to hang and time out.• Thanks to Marvin Knight of IBM for helping me with this!

• If you have z/OS background, stick to REXX. It’s easier to use syscalls() to get z/OS UNIX functionality in a REXX exec than it is to go the other way around.

26

REST API Steps Can Do Cool Stuff Too• GET, PUT, POST and

DELETE methods supported.

• The workflow editor makes required some fields that are documented as optional.

• Your workflow can “map” the response and set workflow variables from the results.

• So far, no PSWI management capabilities

27

Launching Workflows After Deployment• You can define and embed workflows within an SWI.

• The installer gets to choose which workflows to run after deployment.

• Software Management makes certain special variables available to those workflows.

28

izud-varsversion = version

izud-created = yyyy-mm-ddThh:mm:ssZ

izud-createdby = user-id

izud-system = system-nickname

izud-fmids = ["fmid-name"]

izud-globalzone = csi-data-set-name

izud-zones = [{

"izud-name":"zone-name",

"izud-type":"zone-type",

"izud-related":"related-zone-name",

"izud-csi":"csi-data-set-name"

}]

izud-datasets = [{

"izud-blksize":"block-size",

"izud-dddefs":[{

"izud-dddef":"dddef-name",

"izud-zone":"zone-name",

"izud-path":"unix-directory"

}],

"izud-dscategories":["data-set-category"],

"izud-dsname":"dataSetName",

"izud-dstype":"dataSetType",

"izud-extents":"allocated-extents",

"izud-extendedformat":true | false,

"izud-lrecl":"logical-record-length",

"izud-mountpoint":"UNIX-path",

"izud-recfm":"record-format",

"izud-storclas":"storage-class",

"izud-tracks":"allocated-tracks",

"izud-unixdirs":["unix-directory"],

"izud-used":"used-tracks-percent",

"izud-volumes":["volser"]

}]

izud-products = [{

"izud-eosdate":"yyyy-mm-ddThh:mm:ssZ" | "NotAnnounced" | Unknown",

"izud-features":[{

"izud-featname":"feature-name",

"izud-featid":"feature-id",

"izud-fmids":["fmid-name"]

}],

"izud-gadate":"yyyy-mm-ddThh:mm:ssZ",

"izud-prodname":"product-name",

"izud-prodid":"product-id",

"izud-prodinfoversion":"yyyy-mm-ddThh:mm:ssZ",

"izud-prodsups":[{

"izud-prodid":"product-id",

"izud-release":"vv.rr.mm"

}],

"izud-release":"product-level",

"izud-srels":["srel"],

"izud-url":"product-url",

"izud-vendor":"vendor-name",

}]

General Variables

Data Set Variables Product Variables

Using Array Variables In A Template• Templates can use workflow array variables

• In this simplistic case, we create a PDS member that contains the data set name and volume serial for every data set in the software instance.

• See SYS1.SAMPLIB(IZUDWFVR) for more examples of variable use.

29

<template>

<inlineTemplate substitution="true">

List of Data Sets Used:

##

## Loop For All Data Sets

##

#foreach($item in ${instance-izud-datasets})

DSNAME=${item.izud-dsname}) VOLUME=${item.izud-volumes[0]}

#end

</inlineTemplate>

<saveAsDataset substitution="true">

${instance-izud-createdby}.PARMLIB(DSNLIST)</saveAsDataset>

</template>

Workflow Steps Can Be Automated• Automated steps are our best hope for a

“Windows-style” install with minimal intervention.

• You check the “Auto-Enable” box on General tab.• When the user clicks “Perform” from the Steps

page, they get a confirmation dialog.• You must prompt (or provide discovered defaults)

for every required variable needed for the steps you wish to automate together as a group.

• Additional reasons automation will stop:• Processing reaches an automated step for which one

or more required variables are not satisfied.• Processing reaches an automated step that is

somehow not eligible for automatic processing.• For example, if the step is Unassigned.

• Processing is stopped through a user request.

• Press <Refresh> to see the results!30

General Rules of Thumb• Organize your actionable steps as leaf steps under at least one parent level.

• IMHO you should put your primary step content under the <description> tag. That’s the content users see when they select a step from Steps page. The text under the <instructions> tag isn’t shown until after Perform is selected and is better suited to helping explain what to expect from the workflow than it is to explaining how to perform an activity on your z/OS system.

• Use existing high-quality content wherever possible.

• Use high-quality graphics to explain things. A picture is worth 1000 words!

• Keep abreast of APAR activity in this space.• Development is ongoing and usually satisfied through SPEs.

• Documentation is not as thorough as one might hope. Be prepared to experiment. Use the Velocity Template doc provided by Adobe!

• Collaborate with colleagues – even if they work at other companies. Share your experiences and make the “installer” ecosystem better for everyone…

31

Workflow in Action (Video)

• In March 2019 I gave a webinar demonstrating a draft version of our workflow in action with exhaustive explanations of each step – plus history, many asides, and other commentary to bring attendees up to speed with this emerging technology.

• The link below is to a stripped-down version of that webinar (with no audio) demonstrating just the z/OSMF Software Management and Workflow steps without any pauses or commentary. It is approximately ten minutes long and should give you a good feel for the types of interactions one can expect when using z/OSMF Workflows and Software Management for software installation.

32

https://youtu.be/ne8LntRPMIQ

Internet Workflow Resources

• My Webinar Using the Draft Workflow• https://www.phoenixsoftware.com/Webinars or directly to

https://register.gotowebinar.com/register/8842965600164253453

• https://github.com/IBM/IBM-Z-zOS/tree/master/zOS-Workflow

• https://github.com/openmainframeproject/zorow

• ftp://www.phoenixsoftware.com/pub/demo/workflow.bat

33

Please submit your session feedback!

• Do it online at http://conferences.gse.org.uk/2019/feedback/BM

• This session is BM