Upload
igualproject
View
1.689
Download
2
Tags:
Embed Size (px)
Citation preview
A Tool to Visualize Verbal Protocols A Tool to Visualize Verbal Protocols to Enhance Strategic and to Enhance Strategic and
Metacognitive Abilities in Basic Metacognitive Abilities in Basic Programming Programming
Dr. Carlos Argelio Arévalo MercadoDr. Carlos Argelio Arévalo Mercado
M. en C. Lizeth Itziguery Solano RomoM. en C. Lizeth Itziguery Solano Romo
TAMK Opening IGUAL Project meeting. TAMK Opening IGUAL Project meeting. February 14 – 18, 2011February 14 – 18, 2011
UAA IGUAL Proposed Project StaffUAA IGUAL Proposed Project Staff
Academic Staff Dr. Carlos Argelio Arévalo Mercado (Local Coordinator) M.C. Lizeth Itziguery Solano Romo (Student coordinator and research
support) Dra. Lizbeth Muñoz Andrade (Experiment design and execution) Dra. Laura A. Garza González (Financial support) M.C. Jorge E. Macías Luévano (Infrastructure support)
Student Staff Francisco Javier García Pilar (programmer) Bertha Alicia Cobos (protocol recording) Student 3 (programmer) Student 4 (protocol recording)
2
Aguascalientes.Aguascalientes.
3
Population 1.065.416
Autonomous University of AguascalientesAutonomous University of Aguascalientes
History and Background Different names:
1887 Agriculture School 1871 to 1875 Scientific and Literature Institute 1985 State Science Institute 1906 State High School Before 1973 Autonomous Science and Technology Institute
Since June 19, 1973 Autonomous University of Aguascalientes 36 years since foundation 9 University presidents 54 Under graduate programs, Graduate programs 38, 7 technical
programs, 2 high school programs
4
University PopulationUniversity Population
Professors: 1,985 Students: 14, 310
5
Section Total
Graduate 235
Under graduate 12,456
High School 1,619
Internal UAA Programming StatisticsInternal UAA Programming Statistics
6
Cognitive aspects of programming (Literature Review)Cognitive aspects of programming (Literature Review)
7
Cognitive Aspect Description References Implications
Mental Models
A mental model is an internal representation of a complex system or task, whose construction enables the learner to understand and predict that system's behavior.In the context of programming, this aspect is sometimes called "The Notional Machine"
(George 2000); (Bayman 1983); Johnson-Laird (1983); Winslow (1996); Du Boulay (1989); . (Fixx, Wiedenbeck, y Scholtz 1993; Hegarty 1993; Ma et al. 2007; Ramalingam 2004)
Valid mental models have to be taught explicitly.
Problem occurs when novice creates an invalid mental model.
Schemas and Strategies
A strategy (or plan, or cliché) is a predefined solution within the programmers knowledge structure, used when a recurring problem pattern is recognized.Writing a program is not a straightforward process. It is incremental and hypothesis driven.A key part of a program (called focal point,) is used by an expert, and the solution is built around that point.
(Fixx, Wiedenbeck, y Scholtz 1993); Brooks (1990); Rist (1995, 514); Soloway y Ehrlich (1984); Davies (1993)
Rist…
Strategic knowledge of program construction is hard to teach, and not sufficiently emphasized in traditional courses.Not enough instruction is given to novices on how to "put the pieces together“.Text books often only show the final state of a problem/program, but not the process of construction.
Metacognition
Metacognition is defined as "awareness about how a person learns or solves a problem". Flavell (1979)Through Metacognition, an individual can define the nature of a problem, select a useful mental representation, select an effective strategy, monitor the progress of problem resolution and adjust strategies during the task.
Flavell (1979); Gourgey (1998); Williams y Upchurch (2001) ; Roth (2004),
Metacognitive aspects of program design are included within strategic knowledge.
These Metacognitive knowledge must also be taught explicitly
Instructional aspects related to programmingInstructional aspects related to programming
8
Instructional Aspect
Description References Implications
Learning Style According to a previously selected model, an individual can have different preferences or characteristics that facilitate his/her learning process
Kolb (1984)Felder (1996)(Briggs y Myers, 1995)
There are no statistically significant evidence about some particular learning style being a predictor of success in the first programming course.
Previous Experience
A previous programming experience can occur in the context of high school education
(Fauxx 2006); (Hagan y Markham 2000); (Holden y Weeden 2005).
There is a positive effect on performance if the student has had a previous programming experience.
This positive effect tends to disappear following undergraduate semesters.
Self-Efficacy How well one can execute courses of actionrequired to deal with prospective situations
(Bandura 1982); (Brosnan 1998); (Byrne 2001); (Ramalingam 2004); (Heggestad 2005)
It has been identified as a positive influence on first programming course performance.
Some have stated that self-efficacy its just a residual variable of previous course performances.
Math ProficiencyThe abstract and logical reasoning ability required to solve mathematical problems, is similar to the ability required to program computers
(Byrne 2001); (Hu 2006); (White 2003); (Jenkins 2002)
Has a strong correlation with positive performance.
But ethical and practical curricular issues arise.
9
Components of an information system design theory Components of an information system design theory (ISDT) (Adapted from Walls, et. al, 1992)(ISDT) (Adapted from Walls, et. al, 1992)
1. Meta requirements Describes the class of goals to wich the theory applies2. Meta-design Describes a class of artifacts hypothesized to meet the meta-requirements3. Kernel theories Theories from natural or social sciences governing design requirements4. Testable design products hypothesis Used to test whether the meta-design satisfies the meta-requirements
1. Design method A description of procedure(s) for artifact construction.2. Kernel theories Theories from natural or social sciences goverment design process itself.
3. Testable design process hypotheses
Design Product
Used to verify wether the design method results in an artifact wich is consistent with the meta-design
Design Process
Components of an information system design theory (ISDT)
Doctoral Research General ObjectiveDoctoral Research General Objective
Apply the ISDT framework to produce a design theory that outlines a kind of software artifact whose properties include: visualizations of valid mental programming models, strategic problem solving knowledge and Metacognitive reasoning of programming experts on the act of writing programs, to foster meaningful learning and better knowledge transfer to novice programming students.
10
Specific ObjectivesSpecific Objectives
Propose an Information Systems Design Theory for a kind of software artifacts that promotes the transfer of valid programming mental models, problem solving strategies and Metacognitive abilities to novice students.
Design and build de artifact.
Empirically test the effectiveness of the artifact.
11
ISDT Design Product Kernel TheoriesISDT Design Product Kernel Theories
12
Kernel Theories
Description Evidence / Principles References
Dual Coding Theory
When information is registered both verbally and visually, it's said that be "Dually Coded". Information is better retrieved if it is dually stored, because if one part is lost (visual or verbal) the other is available and can be retrieved.
Empirical data suggests that students using study materials designed to promote dual coding, seem to have better performance
(Kuo y Hooper 2004; Meyer y Sims 1994; Sadoski y Paivio 2004)
Verbal Protocols
Verbal protocols where developed to study the short termmemory process of people (to what things the pay attention and in what order) in the process of solving a problem.
They document the mental behavior of a person, by asking him or her to "Think aloud" while doing a certain task
Verbal data are a valid source of comprehension and study of human cognitive processes.A phrase is the representation of an idea and can be used to identify different kinds of information y different types of cognitive processes.Used in the context of basic programming to analyze common solution patterns.Also used in usability studies.
Ericsson y Simon (1980), Nielsen, Clemmensen, y Carsten (2002), (Détienne 1995; Burkhardt, Détienne, y Wiedenbeck 1997; Rist 1996; Rist 1989);
Worked examples and
completion problem
effect
Worked examples can facilitate the construction of schemas and strategy transfer more efficiently than solving the problems themselves. This prediction is explainedin terms of "cognitive load reduction".It has the limitation that worked examples do not force low motivated students to study them. Worked examples can be complemented with the "completion problem effect" pedagogical strategy.
Empirical studies have demonstrated the effectiveness of worked examples in fieldssuch as algebra, and heuristic knowledge domains.Important: they should be used only during the first stages of learning, to avoidthe so called "expertise reversal effect" and they should not have to be used excessively to avoid the transfer of too much stereotyped solution patterns
(J. Sweller, van Merrienboer, y Paas 1998; John Sweller 1994); (Gerjets, Scheiter, y Catrambone 2004; Renkl, Hilbert, y Schworm 2009; Renkl, Atkinson, y Große 2004; M. Ward y J. Sweller 1990; J. Sweller y G. Cooper 1985)
Verbal Protocols editing and recording processVerbal Protocols editing and recording process
13
Select and Categorize a problem
(instructor)
Record the problem solving protocol for the
selected problem(Programmer &
Interviewer)
Transcribe and Edit the protocol in the systemAs a Worked Example
(Instructor)
Add Visual Aids(Animated Code)
(Instructor)
Add additional notes to the protocol(i.e Focal Point, general strategy)
(Instructor)
Visualize and Study theProblem Solving Protocol
(Student)
Development of the ArtifactDevelopment of the Artifact
14
Evolution of the user interfaceEvolution of the user interface
15
FlashFlash
AccessAccess
Web (PHP)Web (PHP)
User interface for verbal protocol visualization User interface for verbal protocol visualization
16
Protocol Verbalization
Dual Coding
Programming Problem
Instructor aditional Notes
Protocol Navigation
Modified user interfaceModified user interface
17
Problem search by keywords
The interface was subject to several tests to refine its usability. For example a feature was added to allow a student to “jump” directly to a specific step of the protocol
18
Instructor annotations(i.e. focal point)
Animated code
Go to a specific protocol step
Navegación
Verbalizations
Modified user interfaceModified user interface
Measurement instrumentMeasurement instrument
19
To measure the tool’s capability to transfer strategic knowledge to novice undergraduate programmers, we designed a standard test, consisting of three basic programming problems, to assess the student’s skills
Cuasi-experimental conditionsCuasi-experimental conditions
To test the effectiveness of the tool, a semi-experimental setting was designed, using two groups of programming students: one from 2nd semester Computer Science students, and another from 2nd semester Electronic Engineering students, both
from Autonomous University of Aguascalientes (UAA), México. Selection of participants was not random. Complete groups were invited,
given that the experiment was conducted during a period of normal classes. The Electronic Engineering group (n=20) was selected as the control group
(TRAD). The computer science group (n=18) was to serve as the experimental group (EXP). 55% of the (TRAD) group had previous programming experience from high school, while 41% of the (EXP) group had previous programming experience.
Two days before the application of the test, four exercises were given to both groups for them to study and practice. This exercises were the same ones loaded in the Protocol Visualizer Tool.
20
Cuasi-experimental conditionsCuasi-experimental conditions
Controlled conditions for both groups were: Explanatory lecture sessions had a one hour duration for both groups. The given time to answer the instrument was limited to an hour for both
groups. A “motivation” factor was introduced for both groups in form of “extra
points”. Characteristics of the lab computers used from both groups were the
same Previous programming experience was similar in both groups (55% for
TRAD and 41% for EXP). At the time of the experiment, the instructional content given (during
normal classes) to both groups was the same. Both groups where studying basic data structures in C Language.
21
Cuasi-experimental conditionsCuasi-experimental conditions
Uncontrolled conditions were: Teachers from both groups were different, but they came from the same
programming academy, taught the same content, and had the same experience in teaching.
It was not possible to record the individual answering time of the test participants.
Selection of participants was not random. In both cases, complete groups where invited to participate in the study.
22
ResultsResults
23
Statistical dataStatistical data
24
Correlation AnalysisCorrelation Analysis
25
Limitations of the studyLimitations of the study
Given the uncontrolled conditions reported, the positive results obtained by the EXP group using the Protocol Visualizer Tool, cannot be generalized to be valid for all the population of first year undergraduate programming students.
An uncontrolled variable was teaching style: that is, even though both teachers were part of the same academy and had similar background and experience, the two groups having different teachers could have had an unmeasured effect on the final results.
Also, duration of the actual study was limited to three days, given the limited availability of the students who voluntarily participated.
26
Current state: Interface RedesignCurrent state: Interface Redesign
27
Current state: Interface RedesignCurrent state: Interface Redesign
28
Current state: Interface RedesignCurrent state: Interface Redesign
29
Current state: Interface RedesignCurrent state: Interface Redesign
30
Current stateCurrent state
Protocols being recorded by social service students
Problem selection Quality control Editing Transcription / System mounting
31
Future activitiesFuture activities
Future studies need to be longitudinal in nature, so that the effect of a longer exposure of the students to the tool can be measured.
A randomly selected sample of participants is desirable, but this kind of scenario is not always possible (or practical) given the nature of every day lectures.
It is also possible to load the tool with protocols that involve problems related to other programming languages or paradigms such as Java, C#.
Also, the graphical user interface can still be improved through usability test, in order to use with other Web-enabled platforms (such as mobile devices).
Extend the functionality of the tool, by adding pedagogical features such as completion problems, in selected segments of the protocols.
Combine the protocol visualizer with other learning technologies were possible.
32