Presentation on IPAW‘08: Henning Bergmeyer

Folie 1A Python Provenance Library and Requirements for a Provenance Visualization Panel > Henning Bergmeyer > Poster Presentation IPAW_08 > 18/06/2008

"A Python Library for Provenance Recording and Querying“

"Requirements for a Provenance Visualization Panel“

Presentation on IPAW‘08: Henning Bergmeyer

A Python Provenance Library and Requirements for a Provenance Visualization Panel > Henning Bergmeyer > Poster Presentation IPAW_08 > 18/06/2008

Folie 2

Overview

Brief Overview: Provenance System

A Python Library for Provenance Recording and Querying

Usage Examples

Initializing, Recording, Querying, Extending

Architecture

Requirements for a Provenance Visualization Panel

User Groups and Intentions

Graphical Representation and Exploration

Requirements


Folie 3

Main Model Concepts of the Provenance System„Grid Provenance“, PReServ 1.0 (University of Southampton)

Interactions between actors

Relationships (1 subject, 1..n objects, 1 relation type)

are dependencies between interactions (e.g. cause-and-effect)

describe internal, otherwise hidden functionality of actors

Actor States

are assertions about internal states of actors

Interaction Records

complete documentation of an interaction through assertions of all influencing incidents and dependencies

Tracers

unique markers that serve to identify individual workflow executions

distributed along message paths


Folie 4

Main System Concepts of the P-System

Distribution

Several connected P-Stores

differentiation of asserter views


"A Python Library for Provenance Recording and Querying“(Roland Gude, Carsten Bochner)


Folie 6

A Python Library for Provenance Recording and Querying

Open Source: http://sourceforge.net/projects/provenance-csl/

Purpose

easy Provenance recording and querying for Python applications or applications with interface to Python

independent of Java on the client side

Examples for

Initialization

Recording

Querying

Extending own types


Folie 7

Code Examples: Initialization

from provenance.api import *

looks like bad coding style at first

but automatic lazy-loading of required modules prevents severe performance losses

cl = client.Client(“http://localhost:8080”, asserter=“me”)

That‘s it!

A trace file can be specified to log communication with P-Store


Folie 8

Code Example: Recording

subj = utils.createSubjectId(1, “dataAccessor”, "parametername")

objlist = [utils.createObjectId(

utils.createInteractionKey("http://sink", http://source"),

pAssID, 'anything', 'dataAccessor', 'parameter', 'isSender')]

keys,response = self.cl.record([

[utils.createActorState(a_content_0, doc_style),

utils.createRelationship(subj, rel_type, objlist),

utils.createInteraction(m_content_0, doc_style),

utils.createInteraction(xml_content_0)

] ], "isSender", sink, source)

res = interfaces.IRecordAck(response)


Folie 9

Code Example: Querying

queryString = "for $n in $ps:pstruct return $n"

response = self.cl.query(queryString)

result = interfaces.IQueryAck(response)

Afterwards „result“ contains an XML structure containing all „pstructs“ available in that store.


Folie 10

Architecture

SOAP interface translated from WSDL by ZSI

pyProtocols

Python lacks of OO-concept "Interfaces"

pyProtocols allows protocol definitions and automatic adaption

used to make SOAP interface transparent to user

Lazy-loading

PEAK framework


Folie 11

Code Example: Extending Types

class IAddress(IZSITypeCode): """ interface for string typecodes """ def getAsString(self): """ returns a String with the Value of the Stringlike. """IString = protocols.protocolForType(basestring,[])

class AddressAdapter(object): protocols.advise(instancesProvide=[IAddress], asAdapterForProtocols=[IString]) def __init__(self, string): self._delegate = serverAPI.Address(string.__str__()) def getAsString(self): return self._delegate.__str__() def toTypeCode(self): return self._delegate


Requirements for a Provenance Visualization Panel

(Markus Kunde, Henning Bergmeyer)


Folie 13

Motivation

Determine requirements for a Provenance visualization panel

Requirement to document Provenance in our projects (e.g. AeroGrid)

No specification for concrete use of the documented provenance, yet

=> Tool at least for general browsing of low-level documentation is needed

Raw provenance data in XML is hard to browse

Verification of records

Experimental browsing to determine better query and interpretation methods

Panel provided by project „Grid Provenance“ not suitable


Folie 14

Approach

Identify User GroupsUser interests (What do they want to explore?)User intentions (Why do they want to explore that?)

Analyse the Provenance data structureElementsPropertiesConnectionsScale

Determine visualization and analysis methodsWhat information to be shown,Where to show itWhen, for how long, static or animatedClear and consistent semantics for visual elementsDetermine exploration strategy


Folie 15

Identifying User Groups

Interest / Scope

What documentation is asked for?

What documentation is a user allowed to see?

Abstraction high-level border, range of access

Intention

Why is that documentation asked for?

Abstraction low-level border, type and level of detail of required documentation


Folie 16

Identified User Groups

General UserScientist, Engineer, Portal UserInterest: own work, own results,origin of used dataIntentions: reliability and authenticityof results, reproducibility

DesignerSoftware Engineer, Workflow DeveloperProject related, all origins, monitored system, partner-made componentsworkflow behavior, service interaction, product evolution

ManagerWorkflow Provider, Provenance Analyst, User Supportall assigned user and system Provenancecorrectness of services, interpretation support, quality of the P-system

AdministratorDeveloper / Admin of Provenance Systemall P-data available in connected P-storesbuilding the P-system and maintaining its function


Folie 17

User Analysis Intentions

Process: Evaluation of the approach of a workflowActors, Interactions, Sequence of Process steps

Results: Quality of intermediate and end results of processesDependencies of inputs and outcome

Relationship: Analysis of the evolution of dataRelationships of interactions or actors

Time Line Finding performance bottlenecks, improving workflowsEvolution of results, actor behavior

Participation Trust to resultParticipating actors

Comparison Validate correctness of processes and results, by comparing documented executions with referencestructures, like processes, views on interactions, results

Interpretation Custom visualization requirements, deriving knowledgefrom Provenance dataCustom, probably all aspects => Exploration required


Folie 18

Exploration

Difficulty in a large scale graphic exploration system:

Where to start?

1. Begin with on overview

2. Select processes, interaction channels or actors

3. Fade out the rest and choose specific detail visualizations.

4. Read application specific content


Folie 19

Elements


Folie 20

Actor / Asserter Views


Folie 21

Focus on Interaction

Process Map (inspired by tube map)ProcessesParticipating ActorsBottlenecks

Interaction Stretch

Individual Interactions

Relationships and order


Folie 22

Combined Flow-ChartTypical Data Flow GraphShows directions of message flowsNo notion of time => Requires previous selection of recorded process.

System / Process Context


Folie 23

Process Aerial

Find individual executions of selected processes

Find anomalies

Show only interesting actor states and relationships

Scrolling up and down along time axis


Folie 24

Visualisation Methods


Folie 25

Graphical and Exploration Requirements

distinct, consistent representations of documentation elements to allow intuitive interpretation

extensible support of different layout methods

adjustment of alignment helps to interpret

switching of scope and detail

proxy displays for large data sets

e.g. navigation maps

mixing and migrating of layouts (animated)


Folie 26

Architectural Requirements

support of VO managementstore accessactor/asserter views

caching and merging of query resultsextensible architecture

layout methodselement representationsexploration methods"content" support

GUI abstractionWeb PortalsDesktop Applications

Documents

Presentation on IPAW‘08: Henning Bergmeyer