Better Mainframe Automation using Zowe CLI and Python

{Core} You will learn how to embed The Open Mainframe Project’s Zowe project Command Line Interface (CLI) into Python programs and automate the mainframe.

Automation is a helpful tool for every developer or system administrator on any platform. Mainframe is not an exception. There are plenty of reasons why to automate. I will not list them all here but rather share a tweet with one of them and more in the replies:

Why automate?

If you want to automate the mainframe, you need a tool that will help you to communicate with z/OS and many subsystems such as CICS or Db2 and vendor tools. Ideally, it should be one tool than a myriad of different tools. And there is a tool like that — Zowe CLI. It provides a command-line interface that you can use on any platform and control the mainframe. At its core it allows you to work with data sets, jobs, console, issue TSO commands, or interact with z/OSMF. It is extensible by plug-ins so you can easily extend it to work with your subsystem or z/OS tool from the same command-line interface. For the full list, see Zowe Conformance Program — Open Mainframe Project.

If you want to start with Zowe CLI, there is a good resource on Medium about it:

Getting Started with Zowe CLI

Shell Scripts

Issuing standalone CLI commands from your terminal is a good way how to learn Zowe CLI and do simple actions. But you will need to do more. One of the approaches is to develop shell scripts to put more commands together, for example using Bash. There is a simple example that will delete all data sets with the same HLQ:

Delete all data sets with prefix MY.PROJECT.DS*

Bash is a good starting option but when it comes to more complex scripts, popular programming languages such as Python, JavaScript, or TypeScript provide a better option and more tooling.

The advantage of shell scripts is the ease of embedding CLI commands into the script. You will learn how to do it easily in Python too and moreover, you will see how easily you can process the results of the CLI command. The Zowe CLI can contain a lot of data that are returned by it. If you want to have your automation more capable, you will need to work with them.

We have picked Python because it is a popular programming language and it is very popular in the area of automation. This is a popular book that will teach you how to Automate the Boring Stuff with Python.

You will learn to issue a simple command first and then we will develop a Python program that will help us to find the root cause of a failed job.

Calling CLI Commands from Python

There are multiple ways a Python program can call other processes and interact with them. The recommended is to use the subprocess.run(args) function. It has multiple arguments and we will see what makes the best sense for calling Zowe CLI. We would like to pass the same arguments that we use with Zowe CLI in our terminal, have a simple way how to access the data that are returned by it, and get an exception with details about a failure when the command fails.

Basic example:

List data sets with the same prefix

The subprocess.run() function will run the following command:

zowe zos-files list data-set "ZOWEUSER.*"

Option shell=True allows you to enter the same command as you would do in your terminal. capture_output=True causes the standard and error output of the command to be stored in the Python variable rather to be displayed to the user. That allows you to process them. check=True will raise a CalledProcessError exception when the command fails. It is important for complex scripts to react to exceptions and have access to the details of the failure rather than blindly continue with other commands.

In this case, the output is just a list of lines with data set names. We can split it using this expression completed_process.stdout.splitlines() and then process them in a for-loop.

With more complex data, it might be difficult to parse them by ad-hoc Python code but there is a better way. It is --response-format-json or just --rfj option of the Zowe CLI. It returns structure data in a JSON format that we can easily parse in Python into Python data types.

Command zowe zos-jobs list jobs as it returns a table in text form:

JOB10931 CC 0000 ZOWEUSERA OUTPUT
JOB10930 CC 0000 ZOWEUSERA OUTPUT
TSU08423         ZOWEUSER  ACTIVE

It is hard to parse since you do not know where the columns are and how they are named. If there will be a new column added, your code that expects just the fourth column might fail. But when you use the --rfjoption:zowe zos-jobs list jobs --rfj . You will get a JSON data that look like this:

A part of the job list output in JSON format

You can easily JSON using json.loads(s) and then work with the data in Python. In the following example, we display the completion code of completed jobs:

List completion codes of completed jobs

Creating a Wrapper Function

It might be annoying to repeat the same boilerplate to run a command over and over. You can easily make a mistake and it will prevent you from adding a common functionality to all the commands later.

We will create a new Python function called zowe(args) that you can call in your code. It will call the Zowe CLI command and return the parsed JSON data. It can look like that:

zowe() function with an example

You can call Zowe CLI just by calling the function zowe(args).

Handling and Reporting Errors

When a command fails, it raises a CalledProcessError exception. That is not the best experience. CalledProcessError contains a lot of details but they are not printed when the program fails, so we will implement a better exception ZoweCallError that will be printed better:

Longer Example

We will conclude with a longer example of the code that will find failing jobs with a provided job name prefix and owner and print their SYSOUT.

This can be achieved using zowe zos-jobs list jobs, zowe zos-jobs list spool-files-by-jobid, and zowe zos-jobs view spool-file-by-id commands combined together:

Summary

Python provides a good way how to write more complex automation over to Bash or similar shell scripts. Zowe CLI can be called from Python quite easily using subprocess.run(args) function but you can achieve a better developer experience by creating a wrapper than will reduce calling Zowe CLI into a single function call — zowe(args).

You can find the full source code of the zowe() function and the examples in GitHub: https://github.com/zowe/zowe-cli-sample-scripts/tree/master/python/pyzowe.

I wish you a lot of automated tasks!

If you enjoyed this blog check out more Zowe blogs here. Or, ask a question and join the conversation on the Open Mainframe Project Slack Channel #Zowe-dev, #Zowe-user or #Zowe-onboarding. If this is your first time using the OMP slack channel register here.

Photo by Possessed Photography on Unsplash