IBM Watson Studio on IBM Cloud Pak for Data

© 2021 by The Enterprise Strategy Group, Inc. All Rights Reserved.

Operationalizing AI/ML and Automating Lifecycle Management for ModelOps

By Tony Palmer, Senior Validation Analyst August 2021 This ESG Technical Validation was commissioned by IBM and is distributed under license from ESG.

Enterprise Strategy Group | Getting to the bigger truth.™

Technical Validation

IBM Watson Studio on IBM Cloud Pak for Data

Technical Validation: IBM Watson Studio on IBM Cloud Pak for Data 2


Contents

Introduction ............................................................................................................................................................................ 3

Background ......................................................................................................................................................................... 3

Timeframe of Deploying a Model ....................................................................................................................................... 3

Federated Learning ............................................................................................................................................................. 5

ESG Technical Validation ......................................................................................................................................................... 6

AutoAI ................................................................................................................................................................................. 7

ESG Testing ...................................................................................................................................................................... 7

Trustworthy AI .................................................................................................................................................................. 10

ESG Testing .................................................................................................................................................................... 10

Decision Optimization and Pipeline Builder ...................................................................................................................... 12

ESG Testing .................................................................................................................................................................... 12

The Bigger Truth .................................................................................................................................................................... 15

ESG Technical Validations

The goal of ESG Technical Validations is to educate IT professionals about information technology solutions for companies of all types and sizes. ESG Technical Validations are not meant to replace the evaluation process that should be conducted before making purchasing decisions, but rather to provide insight into these emerging technologies. Our objectives are to explore some of the more valuable features and functions of IT solutions, show how they can be used to solve real customer problems, and identify any areas needing improvement. The ESG Validation Team’s expert third-party perspective is based on our own hands-on testing as well as on interviews with customers who use these products in production environments.



Introduction

ESG recently completed testing of IBM Watson Studio on IBM Cloud Pak for Data. Watson Studio is designed to accelerate

the value organizations can extract from AI while increasing trust by operationalizing model development, validation,

deployment, and mitigation of data drift and bias. Watson Studio is built to handle virtually any AI model including machine

learning, deep learning, and decision optimization across any cloud, which can help to unify talent and processes by

bringing together IBM and open-source tools and ecosystems to address the diverse needs of data scientists, developers,

analysts, and subject matter experts.

Background

ESG recently completed a research survey on AI initiatives and the underlying modern infrastructure stack.1 ESG asked

respondents a series of questions associated with the common stages of the AI lifecycle. The results supported the ongoing

narrative that deploying AI into production, including pre-deployment tasks such as conducting A/B testing, are proving to

be top challenges that are preventing organizations from seeing value from AI investments. In fact, ESG research data from

2019 highlighted that 72% of organizations with AI initiatives had yet to operationalize AI.2 More recent ESG research

indicates that operationalizing AI has not gotten easier over the last two years. In fact, 55% of organizations cite

deployment, including A/B testing, as one of the AI lifecycle phases that generates the most headaches for their

organization.3

Timeframe of Deploying a Model

Due to the challenges that AI deployments can create, the time it takes to get a gold model into production is being viewed

as an opportunistic area of improvement. As shown in Figure 1, ESG research uncovered that an eye-opening 1% of

organizations go from a trained model to production in under 10 days. 96% of respondents indicated that their

organization typically takes between 11 to 30 days to go from a trained model to deploying into production. With the

speed at which data changes in a modern, dynamic business, it is increasingly being viewed as unacceptable to take nearly

a month to operationalize AI. For agile businesses that require real-time insight, that time delay could directly cost

business, whether by forcing them to rely on stale data and therefore not deliver expected value to customers or pushing

them to miss out on the opportunity to capitalize on a real-time trend.

Figure 1. Time to Deploy a Model into Production

Source: Enterprise Strategy Group

1 Source: ESG Master Survey Results, Supporting AI/ML Initiatives with a Modern Infrastructure Stack, May 2021. All ESG research references and charts in this technical validation have been taken from this master survey results set, unless otherwise indicated. 2 Source: ESG Master Survey Results, Artificial Intelligence and Machine Learning: Gauging the Value of Infrastructure, January 2021. 3 Source: ESG Brief, Operationalizing AI: Time, Infrastructure Considerations, and Data Drift, June 2021.

1%

51%45%

3%

1 to 10 days 11 to 20 days 21 to 30 days More than 30 days

How long does it typically take your organization to go from trained model to

being deployed into production? (Percent of respondents, N=146)

Estimated mean=20 days

https://research.esg-global.com/reportaction/SupportingAIMLInitiativeswithModernInfrastructure/Toc

https://research.esg-global.com/reportaction/aimlmsr/Toc

https://research.esg-global.com/reportaction/Opportunities-to-Improve-Operationalizing-AI-Brief/Toc



IBM Watson Studio on IBM Cloud Pak for Data

IBM Watson Studio on IBM Cloud Pak for Data is designed to accelerate organizations’ journeys to AI: simple, accessible

data collection and organization to support a trusted foundation for analytics; scalable analysis with AI everywhere; and

transparent operationalization of AI through AI-powered business processes and apps. Watson Studio can help an

organization build ModelOps (Model Operations), defined as a principled approach to operationalizing a model in apps.

ModelOps is a key technology imperative for organizations seeking to operationalize AI though processes and apps.

As part of IBM’s prescriptive approach to AI, IBM Watson Studio is designed to help organizations build and scale trusted AI

on any cloud by surfacing hidden patterns and intelligence or facilitating prediction and optimization as shown in Figure 2.

Organizations can choose any combination of cloud providers to deploy Watson Studio, including IBM Cloud, Amazon Web

Services, Microsoft Azure, and Google Cloud.

Figure 2. IBM Watson Studio—Build and Scale Trusted AI Across any Cloud


IBM Watson Studio provides tools for data scientists, application developers, and subject matter experts to collaboratively

and easily work with data to build and train models at scale. It is designed to provide the flexibility to build models where

the data resides and deploy applications anywhere in a hybrid environment so that organizations can operationalize AI

faster.

Trustworthy AI is aimed at delivering business outcomes that can be trusted, at scale. Considerations include:

• Trust in data: Quality of data, fairness in training data, lineage and provenance, and statistical significance while

ensuring data privacy.

• Trust in models: Quality of model performance, ability to explain model behavior, validation of model performance

before production, and continuous monitoring for fairness and drift.

• Trust in process: Ability to track a model’s lifecycle, reproducibility, compliance, and audit preparedness.



IBM Cloud Pak for Data is an open, cloud-native information architecture for AI. Designed as an integrated, fully governed

platform, organizations can keep data secure at its source and add preferred data and analytics microservices as needed.

Available as a service to build and scale trusted AI, Watson Studio on IBM Cloud Pak for Data helps unify processes, talent,

and tools by bringing together open-source notebooks, visual modeling with IBM SPSS Modeler, and prescriptive analytics

with IBM Decision Optimization.

Federated Learning

Across the globe there are many different laws and regulations that govern the use and handling of personally identifiable

information (PII). The legal landscape around PII is constantly evolving—for example, the introduction of the General Data

Protection Regulation (GDPR) in Europe in 2018, the Personal Data Protection law enacted in Serbia in 2019, the Lei Geral

de Proteção de Dados Pessoais (LGPD) in Brazil as of 2020, and the California Consumer Privacy Act (CCPA), which began

being enforced in 2020. Add to these the facts that enterprise data is fragmented, performance-critical AI models need to

leverage data across multiple clouds, moving data across multiple clouds is often prohibited by regulations, and, even if

there is no compliance concern, moving data is often not practical because it can be costly, risky, or slow.

The following table lays out some important federated learning use cases.

Table 1. Top Federated Learning Use Cases

Use Case Name Patient Analytics

(Healthcare) Fraud & Financial Crimes

(Banking) Predictive Maintenance & Quality (Manufacturing)

Motivation to adopt federated learning

Cannot freely share or pool patient data due to privacy

policies

Need more complex analysis data sets like medical images or data from medical sensors

Data is also a valuable

proprietary resource for pharma/healthcare

organizations

Strict data privacy rules, both for regulatory and competitive

reasons

Traditionally, banks use rule-based and manual efforts to

detect fraud and risk, which is prone to human error

Risky small and micro enterprise

loans are an important rising indicator of bank success,

without credit risk identification

Largest amount of data from sensors/IoT devices on

individual machines

Data cannot be gathered fast enough in one place to

do analysis

Unscheduled machine outages are a top challenge that can derail the business


IBM Federated Learning is designed to deliver exceptional training performance without moving data across disparate

locations and is used in multiple industries. Healthcare organizations can keep patient data secure and private, financial

organizations can save time by minimizing manual and rule-based human identification of fraud, and manufacturers can

achieve high asset utilization and savings in operational costs, as models trained on many data sets are more robust in

predicting failures.

In practice, a data scientist can configure training that accesses training data across multiple sources located in multiple

public and private clouds, and an analytics manager can monitor the training, all aggregated through IBM Cloud Pak for

Data.



Figure 3. IBM Federated Learning—How it Works


ESG Technical Validation

ESG performed evaluation and testing of IBM Watson Studio on IBM Cloud Pak for Data. Testing was designed to

demonstrate how IBM’s Data and AI portfolio can help organizations accelerate their data science journeys, providing tools

to simplify data collection, organization, and analysis, with a goal of operationalizing AI lifecycle management for

trustworthy AI.

Figure 4. IBM Watson Studio on IBM Cloud Pak for Data


As shown in Figure 4, users are provided with guidance and automation to enable them to prepare data, find insights, build

models, and add services with a few clicks. We looked at how an organization would build, deploy, test, optimize, and re-



deploy a model used for credit risk management. The goal of the model is to determine creditworthiness accurately and

fairly.

IBM Watson Studio can automate nearly every aspect of an AI/ML project, from cluster creation, to connection of data

sources, through modeling, deployment, and ongoing optimization, which is designed to enable organizations to make sure

the model continues to provide the best possible fairness and quality, while minimizing drift.

AutoAI

AutoAI in IBM Watson Studio is designed to enable data scientists to build multiple models in just a few clicks, treating data

selection and prep, algorithm selection, hyper parameter optimization, data transformation sequencing, and model

building as a single, coherent optimization problem.

ESG looked at how streamlining the AI lifecycle with AutoAI can help speed time to AI/ML value. AutoAI in Watson Studio is

designed to help organizations deliver trustworthy AI outcomes faster by reducing manual work. Data scientists frequently

collaborate with business stakeholders for problem definition and domain understanding, data engineers for access to

needed data sets, and application developers for model deployment to obtain and deploy the best models for the situation

at hand.

ESG Testing

ESG walked through a scenario where a data scientist needs to execute accurate risk prediction for her organization and so

needs to deploy an AI model that can identify high-risk potential customers so those customers can be passed to a human

for review. First, we reviewed and selected the experiment settings in Figure 5.

Figure 5. AutoAI Experiment Settings in Watson Studio


We selected binary classification as the prediction type for this experiment since we have only two categories—no risk and

risk. Other prediction types available include multiclass classification for multiple, distinct categories, and regression—

where the prediction column contains a large number of values and time series forecasting—to predict future values

against structured, sequential data. Experiment settings let a data scientist select the most important parameters and

metrics for the model. In addition to prediction type, the settings also include positive class—the value to measure

performance by, the metric to optimize for, how AutoAI should optimize algorithm selection, which algorithms to include,

and how many pipelines to generate for each algorithm.



It’s important to acknowledge that many seasoned data scientists, statisticians, and engineers are averse to anything

“auto,” especially when it comes to explainability of the model itself. It feels easier to explain something that you’ve

created yourself from scratch. IBM’s internal experience with this phenomenon, in combination with consultation with

their customers, underscored how important it was to address this issue in AutoAI.

After running the experiment. ESG looked at the AutoAI relationship map in Figure 6. The relationship map ranks the tested

models and shows clearly why the top pipeline was selected, identifying the algorithm selected (extreme gradient booster),

showing the enhancements that were used, and letting the data scientist drill down into the underlying details, like

features, feature importance, model evaluation, and others.

Figure 6. AutoAI Relationship Map in Watson Studio


By itself, the relationship map might still not be sufficient for professionals who are uncomfortable with “auto” anything, so

IBM provides a notebook with all of the generated Python code. When ESG clicked Create, IBM Watson Studio created a

complete, documented notebook in less than a minute in Figure 7. Many users then further modify the notebook to reach

desired accuracy or use this notebook as a starting point for other projects. In this manner even some of the most

sophisticated data scientists can save time and speed time to complete projects while running multiple data science

experiments. Beginners can evaluate how models are built and use the knowledge to get up to speed much faster,

increasing their contributions to data science projects. Data science and AI leaders can help solve some of the most

pressing AI talent issues by making AutoAI and its artifacts available broadly within the unified, governed platform.



Figure 7. AutoAI-generated Notebook in Watson Studio


The table of contents is made up of clickable links to enable navigation to any area of interest instantly with details to use

as shown in Figure 8.

Figure 8. Scikit-learn Definition Available for AutoAI-generated Notebook in Watson Studio


Everything that goes into creation of the model is documented in the notebook, from package selection and compatibility,

to metadata, and reading training data, through preprocessing and hyper parameters. Creating a web service deployment

was fast and easy as well. It can be accomplished in Python by clicking a link in the notebook or directly in the UI. In less

than two minutes, the model was deployed and online in IBM’s cloud, with prewritten code in cURL, Java, JavaScript,

Python, and Scala that can be used to make requests to the model.



Trustworthy AI

Watson Studio helps organizations trust their AI models in key dimensions including explainability, fairness, robustness,

transparency, and privacy. Maintaining model fairness is part of corporate social responsibility (CSR). Further, lack of model

monitoring and management can diminish the returns on AI investments and derail AI projects altogether.

ESG Testing

Once the model is created, the insights dashboard gives users a visualization of the quality, fairness, and data drift of the

model. In the example shown in Figure 9, we can see the production model on the left, the preproduction in the middle,

and a challenger model on the right. In our example, the fairness of the production model is at 50%, well below our goal of

98%.

Figure 9. Model Insights Dashboard in Watson Studio


To investigate the issue, we looked at an evaluation of the model to determine what factor(s) were responsible. Watson

Studio showed that, out of the monitored groups, the female group was receiving favorable outcomes 35% of the time,

Why This Matters

When you consider that just one percent of organizations surveyed by ESG said they could go from a trained model to production deployment in 10 days or less, it’s easy to see that streamlining the AI lifecycle would be a significant benefit to most organizations. It’s time-consuming to write code then wait for others to get access to data sets and create models. More time and effort are consumed trying to maintain model accuracy with cumbersome manual versioning and no visibility into previous models. More importantly AutoAI in Watson Studio helps experienced data scientists and beginners with varying skills and backgrounds speed time to build models, share model insights, and continuously improve results.

ESG was able to create a trustworthy, fully optimized model for risk prediction with complete explainability from initial setup of the experiment, through testing and verification, to production deployment in less than ten minutes.

ESG testing validated that automation of the AI lifecycle can significantly shorten organizations’ time to AI/ML value, enabling them to deliver trustworthy outcomes faster with marked simplification of manual work and no compromise of explainability.



while the male group was receiving favorable outcomes 65% of the time. To further investigate, we looked at a prediction

for an individual transaction in Figure 10. This chart shows each feature in the model with an assigned percentage of

relative weight indicating the strength of the influence of that feature.

Figure 10. Explainability in Watson Studio


For this transaction, each feature of the model has been assigned a percentage of relative weight that indicates the

strength of influence of the feature on the model’s predicted outcome.

Positive weight indicates influence toward the predicted outcome, negative indicates influence toward a different

outcome.

Similarly, Watson Studio helps an organization track drops in accuracy as shown in Figure 11. The model drift monitor can

identify the percentage drop in model accuracy and data consistency within a data range specified by a user.

Figure 11. Model Drift Monitor in Watson Studio




Decision Optimization and Pipeline Builder

To support the needs of organizations making critical business decisions involving thousands of decision variables and

millions of alternatives, IBM Decision Optimization helps drive business results by enabling data science teams to solve

complex problems using a combination of optimization technology and other data science techniques like machine

learning within the unified IBM Watson Studio environment. Watson Studio on IBM Cloud Pak for Data empowers an

organization to run virtually any AI model, including machine learning, deep learning, decision optimization and others.

Once organizations have a model, they need efficient, scalable ways to deploy it through business processes and apps.

ModelOps is an automated approach to model pipeline building that helps synchronize the handoff between data science,

AI, and DevOps teams for AI-powered application development.

ESG Testing

ESG walked through a scenario where a hospital was planning a massive construction project and needed to optimize

resource assignments and timing of activities, examining how IBM Watson Studio can be used to accelerate and optimize

the customer experience of making business decisions. Once the cluster was created and data sources were connected, we

looked at how a model would be built using the modeling assistant. As seen in Figure 12, the modeling assistant lets data

scientists choose a decision category, select and prepare the data, and build the model.

Figure 12. Modeling Assistant with Decision Optimization in Watson Studio


Watson Studio also provides visualization to show how various resources can be scheduled as shown in Figure 13.



Figure 13. Visualization with Decision Optimization in Watson Studio


Next, we looked at how a data scientist would use Pipeline Automation to quickly build and deploy an optimized challenger

model using modular components (Figure 14). A challenger model is essentially a new version of the model created to

address fairness and data drift issues. The grey area is called the canvas and the objects on the canvas are called nodes.

Users can drag and drop nodes that correspond to tasks a user would need to execute to build a model.

Figure 14. Pipeline Builder in Watson Studio




Once a model is built and deployed, this tool can be used to address issues with fairness, quality, and drift within the

production model. A user would trigger this process to create an experiment that builds new, optimized models that can

be easily deployed into production. This process can also be triggered from the command line tools, so users can easily

integrate this into their continuous integration and continuous delivery (CI/CD) systems to automate the re-optimization of

the model proactively, not just when an issue is detected. In the visual interface, all underlying data, logs, and code are

available, and data scientists are not restricted in any way, with full functionality also provided through command line and

API for maximized flexibility. In other words, the pipeline builder can replace dozens to hundreds of lines of custom code.

Why This Matters

According to ESG research, improving the customer experience (34%), improving operational efficiency (33%), and reducing risk around business decisions and strategy (33%) are among the most important objectives respondents expected to accomplish from their investments in AI/ML. ESG asked about data drift in the same survey, and while 35% said they retrain existing models, 57% said they build entirely new models. Clearly, AI and ML are becoming more strategic to businesses, the efficiency and quality of the insights they obtain are integral components, and there is a need to improve the iterative process of AI deployment.

IBM Watson Studio enables organizations to quickly identify model quality, fairness, and drift issues; find the root cause; and quickly build and deploy challenger models utilizing modular components using an approach of combining trustworthy AI, decision optimization, and pipeline management on a unified platform.

ESG testing revealed an optimized user experience with the right tools for all roles involved with decision optimization—data scientist, app developers, and IT admins. Organizations can deploy applications anywhere in a hybrid environment to operationalize analytics faster and solve complex business problems at scale. ESG found that Watson Studio provides an end-to-end environment that helps organizations apply learning from production and quickly iterate while ensuring visibility across data science, application development, and business teams.



The Bigger Truth

ESG research found that 99% of organizations using AI/ML in production environments take more than 10 days to go from

fully trained machine learning models to production. When you understand that organizations need to demonstrate

responsible and explainable AI, it’s easy to see that such long timelines can incur significant liabilities due to model drift,

bias, and risk that is not addressed quickly and definitively.

IBM Watson Studio on IBM Cloud Pak for Data is designed to accelerate organizations’ journeys to AI leveraging simple,

accessible data collection, data organization to support a trusted foundation for analytics, scalable analytics with AI

everywhere, and infusion—in other words, transparent operationalization of AI with AI-powered business processes and

apps. Organizations can choose any combination of cloud providers to deploy Watson Studio to help accelerate time to

value with their cloud and AI investments.

ESG testing validated that IBM Watson Studio provides an end-to-end environment that helps organizations apply learning

from production models and iterate quickly and easily while ensuring visibility across data science, application

development, and business teams. Using IBM Watson Studio, ESG was able to create a trustworthy, fully optimized model

for risk prediction with complete explainability. The entire process, from initial setup of the experiment, through testing

and verification, to production deployment took less than ten minutes. Considering that 99% of organizations surveyed by

ESG reported that it took them more than 10 days just to go from trained models to deployment, this is quite an

impressive result.

With IBM Watson Studio, organizations can deploy AI-powered applications anywhere in a hybrid environment to

operationalize analytics faster and solve complex business problems at scale. If your organization is looking for an end-to-

end platform that enables applied learning from production to quickly iterate optimized models while ensuring visibility

across data science, application development, and business teams, ESG can confidently recommend serious consideration

of IBM Watson Studio on IBM Cloud Pak for Data.


All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The

Enterprise Strategy Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject

to change from time to time. This publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this

publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express

consent of The Enterprise Strategy Group, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable,

criminal prosecution. Should you have any questions, please contact ESG Client Relations at 508.482.0188.

www.esg-global.com [email protected] 508.482.0188

Enterprise Strategy Group is an IT analyst, research, validation, and strategy firm that provides market intelligence and actionable insight to the global IT community.


Documents

IBM Watson Studio on IBM Cloud Pak for Data