Predicting Failures at SAP

A talk by Kim Herzig

ReproductionMining Metrics to Predict Component Failures

Zeller et al.

Structure of this Talk

1. Building failure predictor

2. Working at SAP and initial problems

3. Current status

4. Future work

Hypotheses

Hypothesis

Increase in complexity metrics of an entity E correlates with the number of post-release defects of E

There is a common subset of metrics for which H1 applies in all projects

There is a combination of metrics which significantly predicts the post-release defects of new entities within a

project

Predictors obtained using H3 from one project also predict failure-prone entities in other projectssimilar

H1

H2

H3

H4

RoadmapStep 1: Map bug report to code changes

get reported bugs(post release)

get version datafor reported bugs

get source code

source of graphics: Mining Metrics to Predict Component Failures, Zeller et al.

BugDatabase

VersionDatabase

SourceCode

RoadmapStep 2: Map failures to defects in entities

number of failures found

source of graphics: Mining Metrics to Predict Component Failures, Zeller et al.

BugDatabase

VersionDatabase

SourceCode

Entity Entity Entity

RoadmapStep 3: Predict failure for new entities

How to build this?

source of graphics: Mining Metrics to Predict Component Failures, Zeller et al.

Entity Entity Entity

Entity Predictor FailureProbability

BugDatabase

VersionDatabase

SourceCode

Building Failure Predictor

compute complexity metrics for each entity

e: m(e)m(e)

metrics data

Entity

EntityEntity

Building Failure Predictor

find metrics m(e) correlating to

quality(e): failure predictor

metrics data

Failure Predictor

Entity

EntityEntity

To Be Reproduced !For more details read paper

SAP vs. Microsoft• Other programming language (ABAP)‣ no metric tools!‣ ABAP highly database oriented

• Customers involved in processes‣ not all processes are in house (communication)‣ bug database more like ticket system‣ many different releases with different features‣ Service packs contain new feature

• New terrain for SAP‣ pure future work‣ many current projects but lack of experience

Goals for SAP

• Global Quality Governance:

‣ Help to improve SAP quality strategy

‣ Make development processes more efficient

‣ Integrate into product standard framework

• Development Teams:

‣ Which module to test most?

‣ Identifying critical components!

How does SAP work?online!

R/3

• Application Server

‣ programs written in ABAP/4

‣ programs executed on server

• Development Environment

‣ completely online (no data local available)

‣ integrated version repository

Initial Problems

Limited Access

• Working on Maintenance Servers

‣ Code frequently shipped

‣ Limited data access

‣ Limited development permission

Bug Data

• Ticket System

• separate system (no integration)

• used by clients and internally

• difficult data access

• Version Repository

• integrated in development system

• no direct connection to ticket system

ABAP/4

• Source code stored online (database)‣ how to get code? see later!

• Mixture between functional and object-oriented programming‣ What metrics to be used?

• Integration of SQL‣ Additional metrics?

• Gimmicks like ‘implicit function calls’‣ Hard to detect!

*

* Allgemeiner Berichtsaufbereitungsprozessor

What can be reused?

Basic Framework

• Maps package name to code objects

• Gets source code (incomplete)

• Computes McCabe complexity

• Gets code deltas from repository (untested)

Structure of this Talk

1. Building failure predictor

2. Working at SAP and initial problems

3. Current status

4. Future work

Initial Summary

we have we need

bug data source code

version data metrics data

Requirements

• SAP wants to reuse code‣ programming ABAP/4

• Keep adoption effort as low as possible‣ programming JAVA

• Use as many code information as possible

• Compute metrics efficiently

Let’s get started!

identify code objects

Getting Source

≥ 20 TABLES

initial package

compute dependencies

dependent packages

Source Code

Meta Data

Connectto R/3

The Hybrid

Start GUI

RFC call collect sourcecode

compute basicmetrics

CodeObjects,Metrics/Meta- Data

persistentDB

store results

compute metametrics

SAP Reuse

collect sourcecode

compute basicmetrics

ABAP/4 GUI

ABAP/4 tables

compute meta metrics(to be implemented)

Compute Basic Metrics

Definition: can be computed by simply scanning code

• While assembling code objects

• Granularity: Function/Method - Level

• Collect meta data for later processing(parameter types, x calls y, etc.)

Definition: can not be computed by simply scanning code

• Uses meta data collected by basic metrics

• Computed on database (MSSQL)

• Never computed at SAP

Compute Meta-Metrics

The Metrics

The Metrics- Module related metrics- Counting number of classes, functions, etc.- Detect database connectivity- Look at include statements

- Function related metrics- McCabe, FanIn, FanOut etc- Usage of global variables

- Class related metrics- Inheritance, Coupling

The Metrics

~50% of metrics implemented

metrics data available byend of February

Work to Come

• Measure entity quality‣ Assumption: every delta is bug fix‣ filtering bug data (reopened bugs, delta size)

• Build the failure predictor‣ which metrics correlate to quality

• Statistic analysis‣ try to merge with JAVA approach (Tilman)

• Reproduction on other systems‣ Let’s get excited!

Thank you!Questions?

Questions?Thank you!