Introduction Background The solution Conclusions

Programming Learning: a Hierarchical Model BasedDiagnosis Approach

Wellington R. Pinheiro, Leliane N. de Barros, Karina V. Delgado

Department of Computer ScienceInstitute of Mathematics and Statistics - University of Sao Paulo

{wrp,leliane,kvd}@ime.usp.br

November 5th, 2008

ICBL- November 5th, 2008 Slide: 1/22

Introduction Background The solution Conclusions

Contents

1 Introduction

2 Background

3 The solution

4 Conclusions

ICBL- November 5th, 2008 Slide: 2/22

Introduction Background The solution Conclusions

General Concepts

Intelligent Tutoring System for Programming

Intelligent Tutoring System [Wenger, 1987] (ITS) is an e-learningtool that uses Artificial Intelligence techniques.

An ITS for programming should be able to:

propose programming problems based on the student knowledgelevel;

detect possible faults in the student program solution (i.e., todiagnose the student’s program);

communicate the detected faults in such a way that the studentcan learn with this experience;

construct the student model (representation of the student’sknowledge) based on the results of the tutor diagnosis.

ICBL- November 5th, 2008 Slide: 3/22

Introduction Background The solution Conclusions

General Concepts

Intelligent Tutoring System for Programming

Intelligent Tutoring System [Wenger, 1987] (ITS) is an e-learningtool that uses Artificial Intelligence techniques.

An ITS for programming should be able to:

propose programming problems based on the student knowledgelevel;

detect possible faults in the student program solution (i.e., todiagnose the student’s program);

communicate the detected faults in such a way that the studentcan learn with this experience;

construct the student model (representation of the student’sknowledge) based on the results of the tutor diagnosis.

ICBL- November 5th, 2008 Slide: 3/22

Introduction Background The solution Conclusions

General Concepts

Diagnosis of the student’s program

ProPAT [Delgado, 2005]

Features:

Able to detect logical faults in students’ programs.Based on previous work that uses model based diagnosis toperform program debugging (model based software debugging).Only needs the program, the test cases and some informationabout the programming language to detect the faultyinstructions.

Limitation:

The faults are communicated to the student in terms of programlines, which gives little or no information to the student to findout his mistakes or misconceptions.

ICBL- November 5th, 2008 Slide: 4/22

Introduction Background The solution Conclusions

General Concepts

Model Based Diagnosis

Model based diagnosis (MBD) [de Kleer and Williams, 1987][Reiter, 1987]

An Artificial Intelligence technique to find faulty components inphysical devices.

Reasons over models that describe the correct behavior of the system.

Involves three sub-tasks [Benjamins, 1993]: symptom detection,hypothesis generation and hypothesis discrimination.

Model based software debugging (MBSD) [Mayer et al., 2002]

An application of MBD to program debugging.

Basic idea: step by step program simulation that points out theinstructions that justifies the observable discrepancies (i.e., thedifferences between program outputs and expected outputs -symptoms).

Can be used by an ITS to try to understand the difference betweenthe student’s intentions and his program.

ICBL- November 5th, 2008 Slide: 5/22

Introduction Background The solution Conclusions

General Concepts

MBD versus MBSD

Comparison between model based diagnosis and model based softwaredebugging

Model based diagnosis Model based software debugging

ICBL- November 5th, 2008 Slide: 6/22

Introduction Background The solution Conclusions

General Concepts

Model based software debugging

ICBL- November 5th, 2008 Slide: 7/22

Introduction Background The solution Conclusions

General Concepts

MBSD for programming learning: limitations

Pointing out the possible faulty lines of the program can not beinformative enough for a novice to correct his program.

Even for a regular size program, there can be many hypothesesof failure to be communicated (and discriminated) to thestudent, which can make him tired and confused.

A novice student does not have enough knowledge to makepredictions about the program behavior.

ICBL- November 5th, 2008 Slide: 8/22

Introduction Background The solution Conclusions

Goals

Proposal

To extend the MBSD technique with the use of Hierarchical ModelBased Diagnosis (hierarchical diagnosis), so that:

Faults in the student’s program can be found at different levelsof the program abstraction.

Functions, procedures and programming (elementary) patterns,are viewed as abstract components.

The communication with the student can be made in terms ofthese abstract components, i.e., through a high-level languageinstead of program lines.

In the average case, the tutor would communicate a smaller setof hypotheses of failure to the student.

ICBL- November 5th, 2008 Slide: 9/22

Introduction Background The solution Conclusions

HMBD

Hierarchical Model Based Diagnosis (HMBD)

A technique used to increase performance during the diagnosis.

The system is modeled at different levels of abstraction.

Abstractions are used to represent components (structuralabstraction) or the system behavior (behavioral abstraction).

Uses traditional MBD techniques to find the solution.

ICBL- November 5th, 2008 Slide: 10/22

Introduction Background The solution Conclusions

HMBD

Models and abstractions

Example:

ICBL- November 5th, 2008 Slide: 11/22

Introduction Background The solution Conclusions

HMBD

Abstraction tree

Subsystems are supposed to be independent

ICBL- November 5th, 2008 Slide: 12/22

Introduction Background The solution Conclusions

HMBD

Abstract component

Models of an abstract component

An abstract component can be described by:internal structural model, external structural model and behavioral model.

Behavior of an abstract component

An abstract componentAC1 and the aggre-gated components C1,C2 and C3

The behavior of AC1 isassumed as correct

So, the behavior of C1,C2 and C3 must be as-sumed as correct

This behavior can be formally described by the following axiom:

ok(CA) → ok(C1) ∧ ok(C2) ∧ · · · ∧ ok(Cm) (1)

ICBL- November 5th, 2008 Slide: 13/22

Introduction Background The solution Conclusions

HMBD

Top-down Hierarchical Diagnosis

An algorithm to perform hierarchical diagnosis can be described as[Mozetic, 1991]:

1 Verify the observations that can be used at each level ofabstraction.

2 For each level i , from l to 0, where l is the highest level thatdoes not hide all the connections with value, find thehypothesis set with an MBD algorithm. For each abstractcomponent CA not involved in any hypothesis set at level i ,apply the Axiom 1 for the level i − 1.

3 Return the hypothesis set for level 0.

ICBL- November 5th, 2008 Slide: 14/22

Introduction Background The solution Conclusions

HMBD

HMBD example

ICBL- November 5th, 2008 Slide: 15/22

Introduction Background The solution Conclusions

HMBD

HMBD example

ICBL- November 5th, 2008 Slide: 15/22

Introduction Background The solution Conclusions

HMBD

HMBD example

ICBL- November 5th, 2008 Slide: 15/22

Introduction Background The solution Conclusions

HMBD

HMBD example

ICBL- November 5th, 2008 Slide: 15/22

Introduction Background The solution Conclusions

HMBD

HMBD example

ICBL- November 5th, 2008 Slide: 15/22

Introduction Background The solution Conclusions

HMBD

HMBD example

ICBL- November 5th, 2008 Slide: 15/22

Introduction Background The solution Conclusions

HMBD

HMBD example

ICBL- November 5th, 2008 Slide: 15/22

Introduction Background The solution Conclusions

HMBD

HMBD example

ICBL- November 5th, 2008 Slide: 15/22

Introduction Background The solution Conclusions

MBSD+HMBD

Basic idea of MBSD + HMBD

Give the opportunity for the student to understand and correctthe faults in the program at different levels of abstraction.

After the hypothesis set for a specific level has been found, thestudent can choose between:

change his program with an expectation to correct the faults;discriminate the hypotheses at the current level of abstraction(by supplying new observations about the expected behavior ofthe program);try to correct the problem at a more detailed level, by replacingan abstract component by its aggregated components(refinement).

ICBL- November 5th, 2008 Slide: 16/22

Introduction Background The solution Conclusions

MBSD+HMBD

Program example

Programming Problem: Given a set v of integer numbers, calculate the difference

between the sum of even numbers and odd numbers.

1 public int evenMinusOdd(int v[]) {2 int n = v.length;

* 3 int counter = 1;

4 int evenSum = 0;

5 int oddSum = 0;

6

* 7 while (counter < n) {* 8 if (v[counter] % 2 == 0) {* 9 evenSum = evenSum + v[counter];

* 10 } else {* 11 oddSum = oddSum + v[counter];

* 12 }* 13 counter = counter + 1;

* 14 }15 return evenSum - oddSum;

16 }

(*) Lines used to compose a Counting Loop elementary pattern.

ICBL- November 5th, 2008 Slide: 17/22

Introduction Background The solution Conclusions

MBSD+HMBD

Program model

Assignment

counter = 1

Assignment

Assignment

Assignment

n

evenSum

oddSum

n = v.length

evenSum = 0

oddSum = 0

counter

evenSun

oddSum Expr

evenSum - oddSumresult

Assignment

oddSum = 0

Assignment

Assignment

Assignment

Count ingLoop

n

evenSum

oddSum

counter

evenSun

oddSum

n = v.length

evenSum = 0

oddSum = 0

Expr

evenSum - oddSum result

AssignmentoddSum = 0

Level 0

Level 1

Expr counter < n

LOOP

LOOPBODY

counter

v.length

const1

0

const2

0

const3

1

v.length

const1

0

const2

0

ICBL- November 5th, 2008 Slide: 18/22

Introduction Background The solution Conclusions

MBSD+HMBD

Characteristics of an algorithm for MBSD + HMBD

We consider the following items as important characteristics of ouralgorithm to MBSD + HMBD for programming learning:

The student can discriminate the hypotheses of failure atdifferent levels of abstraction.

Abstract components can be refined according to the decisionof the student.

At any level of abstraction, if the student corrects the faults inhis program, the diagnosis process is finished (even if thecurrent level has abstract components).

All observations can be used during the diagnosis process (i.e.,there is no connection with observed value hidden in anabstract component, during the whole diagnosis process).

ICBL- November 5th, 2008 Slide: 19/22

Introduction Background The solution Conclusions

Conclusions and future work

In this presentation, we showed:

A technique of model based diagnosis to locate logical faults inprograms (MBSD) and its limitations when applied to programminglearning.

An extension of the hierarchical diagnosis to detect faults in programsrepresented with different levels of abstraction, considering abstractcomponents, such as functions, procedures and elementary patterns.

A suitable approach to debug the students’ program and communicatethe faults in a high-level language (considering abstractions), that webelieve to be more appropriate to promote learning.

We have planned to:

Finish the implementation for MBSD + HMBD;

Do an experimental evaluation with students.

ICBL- November 5th, 2008 Slide: 20/22

Introduction Background The solution Conclusions

References

Benjamins, R. (1993).Problem Solving Methods for Diagnosis.PhD thesis, University of Amsterdam.

Chittaro, L. and Ranon, R. (2004).Hierarchical model-based diagnosis based on structural abstraction.Artif. Intell., 155(1-2):147–182.

de Kleer, J. and Williams, B. C. (1987).Diagnosing multiple faults.Artif. Intell., 32(1):97–130.

Delgado, K. V. (2005).Diagnostico baseado em modelos num sistema tutor inteligente para programacaoocom padroes pedagogicos.Dissertacao de mestrado, Instituto de Matematica e Estatıstica.

Johnson, W. L. and Soloway, E. (1984).Proust: Knowledge-based program understanding.In ICSE ’84: Proc. of the 7th international conference on Software engineering,pages 369–380, Piscataway, NJ, USA. IEEE Press.

ICBL- November 5th, 2008 Slide: 21/22

Introduction Background The solution Conclusions

References

Mayer, W., Stumptner, M., Wieland, D., and Wotawa, F. (2002).Observations and results gained from the jade project.In Proc. of the 13th International Workshop on Principles of Diagnosis,Semmering, Austria.

Mozetic, I. (1991).Hierarchical model-based diagnosis.Int. J. Man-Mach. Stud., 35(3):329–362.

Reiter, R. (1987).A theory of diagnosis from first principles.Artif. Intell., 32(1):57–95.

Wenger, E. (1987).Artificial intelligence and tutoring systems: Computational and cognitive

approaches to the communication of knowledge.Morgan Kaufmann Press.

ICBL- November 5th, 2008 Slide: 22/22