Evaluating Dynamic Services in Bioinformatics

Evaluating Dynamic Services in Bioinformatics

Maíra R. Rodrigues

Michael Luck

University of Southampton, UK

Tenth International Workshop CIA 2006, Edinburgh

CIA 2006ECS - University of Southampton2

Outline

Bioinformatics Agents and Bioinformatics Model for Cooperative Interactions: Overview Requirements for Service Evaluation Evaluation Method Example Scenario Conclusion Future Work


Bioinformatics

Bioinformatics Application of computer technology to manage

and analyse biological data Bioinformatics Services

Heterogeneous Locally and remotely used Continuous update

Management and analysis of biological data and tools Suitability of an agent-based approach


Bioinformatics

Interrelated data Cooperative applications

Participants request and provide services to each other

Services free of charge Non-economic exchange of

different types of tools and data

Interactions are based on reciprocal relations


Agents and Bioinformatics

The agent-based approach: Agents provide and request bioinformatics

services Existence of alternative providers Services are provided with different levels of

quality (heterogeneity) Therefore..

Agents need to select service providers


Agents for Interaction

Agent-based applications in bioinformatics: Concerned with high-level management tasks

Our concern: Model non-economic cooperative interactions Evaluation method for bioinformatics services

to determine an agent’s satisfaction Guide agent’s decisions over service

providers


Model for Cooperative Interactions

A1 A2

service

Model non-economic cooperative interactions based on exchange values (Piaget 1973)

• satisfaction

• debt

• effort

• credit


Model for Cooperative Interactions

A1 A2• satisfaction

• debt

• effort

• credit

Model non-economic cooperative interactions based on exchange values (Piaget 1973)

A1 A2• credit

• satisfaction

• debt

• effort


Model for Cooperative Interactions Exchange values result from the agent’s

evaluation of the service

Partner Selection(future interactions)

Exchange ValuesService

Evaluation


Model for Cooperative Interactions Exchange values result from the agent’s

evaluation of the service Exchange values (Rodrigues, Luck 2005, 2006)

Current work focus on service evaluation

Partner Selection(future interactions)

Exchange ValuesService

Evaluation


Bio-Services are dynamic: Constant updates Regular behaviour, but Sensitive to different parameter configuration

Evaluation requires Repeated evaluation Attach context information Evaluation of different aspects of the service

Service Evaluation


Evaluation method should address: Generality: apply to different types of bio-

services and aspects of these services Continuity: repeat evaluation every time a

service is received Consistency: compare evaluations made at

different points in time Discriminated information: allow flexible

decision-making by using evaluation of individual aspects or a global evaluation

Service Evaluation


Alternative Approaches

Quantitative approaches Scoring or utility functions

• Objective values• Precision, consistency, combination is

straightforward Qualitative approaches:

Classification rules (e.g., poor, good, excellent)

• Subjective values


Evaluation Method

Choose evaluation attributes for service examples: performance, quality, reliability,

etc. For each attribute, associate result

measures Pieces of information derived from service

result that can determine the service utility in relation to an attribute (observed value).

Static or dynamic measures (e.g., quality of interface and response time)


Evaluation Method

General evaluation function for evaluation attributes (utility): For a set of attributes A = {a1,..,ai}

Ui = bc

result measure for ai

evaluation strictness

0

1

c

Ui


Evaluation Process

Before evaluation: Identify evaluation attributes for services and result

measures for each attribute

Repeat evaluation process every time a service is received Input is the service result and configuration used For each evaluation attribute ai

• Compute result measures

• Calculate evaluation Ui

• Store evaluation Output is a set of evaluations (evaluation tuple)


Evaluating Bio-Services

Proteomics research Protein identification services

• Input: file (list of unknown peptides)• Process: database + matching algorithm• Output: list of proteins, peptides per protein

Services: OMSSA, MASCOT, Tandem Local and Remote Heterogeneous results for same input data Sensitive to different input configurations Evaluation can be used as criterion for future

selection


Evaluation attributes: Sensitivity

• Capacity of matching related proteins Accuracy

• Capacity of identifying true matches Performance

• Time taken from input submission until result is received



Result measures (rm): Sensitivity

• Number of proteins• Peptide ratio - peptides per protein

• Influence of input size• Increasing utility

input_size

peptide_ratio x protein_number


rm =


Accuracy • Number of false positives

• Decreasing utility

rm = false_positives

Performance: • Response time

• Influence of input size• Decreasing utility

response_time

input_size


rm =


Evaluation functions:

Ui = 0.5rm

Sensitivity (U1):

• U1 increases with peptide_ratio and protein_number

Accuracy (U2):

• U2 decreases with false_positives

Performance (U3):

• U3 decreases with response_time




Practical evaluation: Same input spectra Two different configurations (C1 and C2)

Evaluation of sensitivity

Evaluation reflects different results for C1 and C2


Evaluation of performance

Again, evaluation reflects different results for C1 and C2



Conclusions

Present an evaluation method to be used by agents requesting dynamic services in bioinformatics

Discussion of issues for efficient evaluation of these services, including Adoption of a repeated evaluation process Absolute evaluations Generation of individual and compatible evaluations Single evaluation must be calculated during selection


Conclusions

Show the application of the evaluation method for protein identification services

Importance of dynamic (repeated) evaluation is shown through empirical results

Provide more accurate information for agents that need to select services with dynamic characteristics


Future Work

Develop selection strategies that use and combine service evaluations Combination through objective and

subjective values Probabilistic analysis of past evaluations Consider similarity between different service

configurations Validate evaluation results with those of

bioinformaticians


Thank you


References

J. Piaget. Sociological Studies. Routlege, London, 1973.

M. R. Rodrigues and M. Luck. Analysing partner selection through exchange values. In Jaime Sichman and Luis Antunes, editors, Multi-Agent-Based Simulation VI, volume 3891 of Lecture Notes in Artificial Intelligence, pages 24-40, Berlin Heidelberg, 2006a. Springer-Verlag.

M. R. Rodrigues and M. Luck. Cooperative interactions: An exchange values model. In Coordination, Organization, Institutions and Norms in Agent Systems (COIN), ECAI Conference, Riva del Garda, Italy, August 2006b.

Documents

Evaluating Dynamic Services in Bioinformatics