School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING Evaluation Kleanthous Styliani

School of somethingFACULTY OF OTHER

School of ComputingFACULTY OF ENGINEERING

Evaluation

Kleanthous Styliani

www.comp.leeds.ac.uk/stellak

Overview

Common Evaluation Approaches

Before Planning an Evaluation

Evaluation for this PhD


Evaluation approaches for Intelligent Systems• Formative & Summative Evaluation

• Layered Evaluation (Specific to Adaptive Systems)

• Simulations

• Control Groups


Simulations: preferred when you need large amounts of data or data is too expensive to collect or when people have to be involved and there is no available sample. (e.g. P2P Communities, Social Networks)

Control Groups: allow to different samples to use a system with and without the intelligent functionality and measure which group did best or according to what they are evaluating. (Comtella)

(negative: the non-intelligent version cannot be optimal in any way if the system built with intelligent functionality at the beginning )


Formative: the system should be evaluated for its usability and effectiveness in the early stages.

Summative: the effectiveness of the system is determined in real environments after development or completion of a major stage.

Layered approach:

• Layer 1 – Interaction Assessment Evaluation

e.g. Are the user’s characteristics being successfully detected by the system and maintained in the user model?

• Layer 2 – Adaptation Decision Making Evaluation

e.g. Are the adaptation decisions valid and meaningful for selected assessment results?



Before planning the evaluation:

• What are you evaluating?

• What are your research questions/hypothesis?

• How will you test the above with the evaluation?

• Plan the evaluation

What are you evaluating?

• Extraction of community model

• Pattern detections

• Evolution algorithms

• Advantages of interventions


What are your research questions?• Formative Evaluation

• Do the CM algorithms work as are intended to work?

• Do the pattern detection algorithms extract the correct patterns?

• Do the evolution algorithms pick the changes happen through time?


• Summative Evaluation

Structured specifically for this framework with specific questions to be answered

• Suitability of interventions:

• How do members evaluate the interventions?

• Benefits for users:

• Do the users find the interventions helpful in

• Identifying people relevant to them?

• Identifying resources relevant to them?

• Become aware of who is working on what?

• Identify potential collaborators?


Benefits for the community:

Newcomers

How quickly are they integrated in the VC?

Oldtimers

How active they are?

Transactive Memory

Do they know who to ask or where to find resources for topic A?

Do they know what other members know in the VC?

To whom is your knowledge important?


•Benefits for the community:

• Shared Mental Models

• What others are doing in this VC?

• What is the purpose of this VC?

• Cognitive Centrality

• Who shares the most valuable resources in this community?

• Have the centrality shifted more effectively between members?

• Any peripheral members became cognitively central?


Where to evaluate?

• BSCW Data of Semantic Web VC (Formative)

• AWESOME Simulated Data (Formative) & (prove generality of approach)

• Active VC to evaluate the whole framework & focusing on the intelligent interventions (Summative)

Qualitative (questionnaires)

Quantitative (statistics)


Why am I telling you these things?

BSCW VC for our group

I need your contribution to complete the major part of evaluation – Summative Evaluation

•Share some resources with the others

•Download resources that might interest you

•Try to follow the guidelines given


Thank you!


Mark, M. & Greer, J. (1993). Evaluation Methodologies for Intelligent Tutoring Systems. Journal of Artificial Intelligence and Education, 4 (2/3), pp. 129-153

Karagiannidis, C. and D.G. Sampson. Layered Evaluation of Adaptive Applications and Services. in International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems. 2000 Springer-Verlag.

Millan, E. and J. Perez-De-La-Cruz, A Bayesian Diagnostic Algorithm for Student Modeling and its Evaluation. User Modeling and User-Adapted Interaction, 2002. 12(2): p. 281-330.

Shlomo, B., et al. Evaluating User Model Effectiveness by Simulation. in Workshop on Personalized Access on Cultural Heritage at 11th international conference on user modeling. 2007.

Cheng R., Vassileva J. (2006) Design and Evaluation of an Adaptive Incentive Mechanism for Sustained Educational Online Communities. User Modelling and User-Adapted Interaction, 16 (2/3), 321-348. (special issue on User Modelling Supporting Collaboration and Online Communities).


Documents

School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING Evaluation Kleanthous Styliani