Elicitation and Utilization of Application-level Utility Functions

Paul de Grandis and Giuseppe ValettoDrexel University - Dept. of Computer Science

Outline

Motivation Goal Approach Engineering considerations Experiments Conclusions

Motivation

Autonomic provisions are often costly and complex

Many times, those provisions target only specific classes of problems expected or likely conditions

It’s not only a matter of requirements … … it’s also hard to develop complete system models

that enable general-purpose adaptations

The challenge

Unexpected conditions are often the most critical Typically occur only on the field By definition, they have not been accounted for by

designers, programmer, testers etc. Potentially devastating for the system

One could argue that autonomic provisions fulfill their promise when they can handle the unexpected

Designing for the unexpected

Our goal

Enable non-analytic self-assessment of SW applications

Self-assessment: map the state space of the application to some abstraction of value Recognize where the system on in that map

Non-analytic: do not require a complete model of the system in order to enable self-assessment

Two-steps strategy

Self-assessment

Self-adaptation

State space

“bad” state“good” states

Approach: application-based utility

Utility of an application

What is the app. value for, e.g., its client? A single scalar: Uapp [0,1]

Utility function: Uapp = f(V[s0,…,sn]) s0,…,sn are state variables

Network measures System measures Application specific measures User-observable measures

Our vision

Elicitation: construct Uapp for a given application Often, for a given task or feature

Embedding: insert runtime code to monitor state variables and compute Uapp

Utilization: continuously evaluate Uapp

We are interested in a method that can handle legacy applications as well as new ones

Our process

1. Identify utility from the client perspective

2. Instrument application and environment

3. Run experiments and observe V[s0, …,sn]

4. Synthesize Uapp(V[])

5. Substitute Uapp in the instrumented code

1. Manual, expert knowledge

2. Automated

3. Automated

4. Off-line, statistical correlation

5. Automated

Observation and synthesis We operate at multiple

distinct levels Typically, application-

vs. environment-level

Collect lots of lower-level monitoring data from the running environment …

… and relate it statistically to some externally visible manifestation of application goals

Environment

Application

Statistical correlation

Example – FTP service

Simple protocol-based application Utility is straightforward:

For an FTP client, utility = client throughput

Relevant environment (socket-level) parameters: Latency Bandwidth Loss rate …

Eliciting application utility

Environment level

Application level

U

t

Embedding application utility

Environment level

Application level

U

t

Engineering approach

Elicitation and embedding framework

Aupy: “Application Utility in Python” Framework leveraging meta-programming

capabilities of Python to instrument applications

Two flavors: Python annotations (legacy code-friendly) Python contexts (granular and flexible)

Open-sourced at: http://www.aupy.org

Aupy annotations

Added to method definitions @monitor(foo) bar triggers evaluation of utility function

“foo” when “bar” is executed

u.f.method declarations: @pre-utility foo @post-utility foo @utility foo @concurrent-utility foo

foo() can contain monitoring code (elicitation) or code computing Uapp (embedding)

Aupy contexts

Contexts are Python scoping constructs “with” keyword establishes a custom scope

Aupy uses them to associate utility functions to arbitrary blocks of code Finer-grained control

You can pass to a context utility function, arguments, configuration data, callback information …

Benefits of contexts

Contexts (and hence u.f.) can be composed

Multi-step u.f. Utility of a u.f. Support of selection logic among multiple

alternative u.f.

Experiments

FTP experiment set-up

In-the-lab data collection 125 test runs using dummynet to throttle network create a uniform map of environmental conditions

Two download scenarios: Bulk-ftp: transfer one large file Small-ftp: transfer multiple small files

Off-line data analysis and u.f. synthesis

FTP results

X=inter-arrival time Y= socket throughput Very different u.f. for the 2

scenarios We plugged in elicited u.f.

kept monitoring client throughput online and compared

€

Ubulk

= a1X1+ a2X2+ a4X4+ a5X5+ b1Y1+ b2Y2+ b5Y5

Usmall

= a4X4+ a5X5+ b1Y1+ b2Y2+ b4Y4+ b5Y5

Bulk-FTP Small-FTP

a1 -175150.7079 a1 0

a2 -145163.8736 a2 0

a3 0 a3 0

a4 -579.7485 a4 1028.3839

a5 2.0136 a5 -157.4976

b1 0.9241 b1 0.5368

b2 -0.2341 b2 -0.42717

b3 0 b3 0

b4 0 b4 612.90247

b5 -0.17755 b5 15.18621

FTP-bulk results

Estimated Actual

Another experiment – log explosion

Real-world case study Logging component of a Python-powered

popular WWW site gone awry Explosive growth of log would block the WWW

site in a few seconds Even prevent remote login from sysadmins

Could we apply non-analytic self-assessment to this problem?

Log explosion experiment

Phase 1: catching log explosions Utility of WWW site drops almost immediately to 0 upon

log explosion need to quickly capture acceleration peaks in disk writes

per process before disk resource is exhausted

Phase 2: preventing log explosions: hypothesis: utility is Uapp ≈ f(1-pn

boom)

pboom : probability that a log explosion is going to happen “some time soon”

Log explosion results

We were able to catch log explosions Captured log explosion dynamics for disk writes Write speed raises 2 orders of magnitude each second Uapp looks like a step function

Collapse occurs in a few seconds But we could detect the front and turn off logging in time

The prevention problem is more complex Would require very frequent sampling of disk write speed That interferes too much with normal operation We cannot really “see” preliminary signs and estimate “pboom”

Conclusions

Lessons learned

We have understood the elicitation process Repeatable Requires only a modicum of domain

knowledge

We have acquired know-how on the mechanisms for elicitation and embedding Can be made minimally intrusive Can be applied to legacy systems Layered approach is generally valid

Lessons learned

We have insight on the task-based nature of application utility

Need an array of u.f. that can be selected opportunistically On a per-feature basis Also based on different tasks

Idea applies especially well to: Protocol-based applications Infrastructure-level services

Research directions

Towards in-the-field data collection Deploy Aupy-instrumented application

Multiple features are exercised Assign clusters in the monitored space to

exercised features Elicit all feature-specific u.f. from the clustered

areas

Research directions Towards automated utility definition

Use cooperative negotiation and dependency discovery to figure which application characteristics to consider as utility

Leverage utility map to guide adaptations Reinforcement learning to compute trajectories

back from bad utility to good utility states Actionable trajectories: can be traversed by

tuning some state variables Utility delta as RL “reward”

Questions?