Introduction to the Cognitive Tutor Authoring Tools (CTAT) and Example-Tracing Tutors Bruce McLaren Systems Scientist, Co-Manager of the CTAT Project Team

Introduction to the Cognitive Tutor Authoring Tools (CTAT) and Example-Tracing Tutors

Bruce McLarenSystems Scientist, Co-Manager of the CTAT Project Team

Human-Computer Interaction InstituteCarnegie Mellon University

PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006

Kinds of Computer Tutors

Intelligent tutoring systems

e.g., Sherlock Model-tracing tutors

e.g., Andes Cognitive tutors

e.g., Algebra

Tutoring systems

CAI e.g., Microsoft’s PersonalTutor

Constraint-based tutors

e.g., SQL Tutor


Cognitive Tutors: Real-world success of Intelligent Tutoring Systems technology

• Cognitive Tutor Courses– Computer-based one-to-one tutoring

• Based on computational models of student thinking• Based on the concept of Model Tracing

• Used by many students in many schools– Algebra Cognitive Tutor: In over 2,000 schools in the USA,

300,000 students per year– Geometry Cognitive Tutor: In 350 schools– Most widely used intelligent tutoring system

• University created a company, Carnegie Learning, to disseminate

Algebra Cognitive Tutor Sample

Use graphs, graphics calculator

Analyze real world problem scenarios

Use table, spreadsheet

Use equations, symbolic calculator

Tutor learns about each student

Tutor follows along, provides context-sensitive Instruction


ACT-R: A Cognitive Theory of Learning and Performance

• Big theory … key tenets:– Learning by doing, not by listening or watching

– Production rules represent performance knowledge:

These units are: Instruction implications:• modular • context specific

isolate skills, concepts, strategiesaddress "when" as well as "how"

Anderson, J.R., & Lebiere, C. (1998). The Atomic Components of Thought. Erlbaum.


• Cognitive Model: A system that can solve problems in the various ways students can

Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d

Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a

Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d

Cognitive Tutor Technology:Use ACT-R theory to individualize instruction


3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology:Use ACT-R theory to individualize instruction• Cognitive Model: A system that can solve problems in

the various ways students can

If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction


3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology:Use ACT-R theory to individualize instruction• Cognitive Model: A system that can solve problems in

the various ways students can

If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Hint message: “Distribute a across the parentheses.”

Bug message: “You need tomultiply c by a also.”

• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

Known? = 85% chance Known? = 45%


CTAT motivation: Make tutor development easier and faster!• Development costs of instructional technology are, in

general, quite high– E.g., ~300 dev hours per hour of instruction for Computer Aided

Instruction (Murray, 1999)

• Cognitive Tutors: – Large student learning gains as a result of detailed cognitive

modeling– ~200 dev hours per hour of instruction (Koedinger et al, 1997)– Requires PhD level cog scientists and AI programmers

• Solution: Easy to use Cognitive Tutor Authoring Tools (CTAT)

Murray, T. (1999). Authoring Intelligent Tutoring Systems: An Analysis of the state of the art. The International Journal of Artificial Intelligence in Education, 10, 98-129.

Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big

city. The International Journal of Artificial Intelligence in Education, 8, 30-43.


How to reduce the Authoring Cost?

• Less programming, more automation– Drag & drop interface construction– Demonstration-based programming

• Human-Computer Interaction methods– User studies, summer schools, informal & formal

comparison studies

• Exploit tools already in use– Component-based architecture & standard inter-

process communication protocols


Authoring with CTAT• Cognitive Tutors

– Difficult to build; for programmers– General for a class of problems

• Example-Tracing Tutors– Much easier to build; for non-programmers– Limited to a single problem instance

• New: “Mass Production” feature greatly facilitates authoring of multiple problem instances (in Excel)

• Typical Approach– Build Example-Tracing Tutor first

• Cognitive Task Analysis• Rapid prototypes - perhaps sufficient for the problem

– Develop Cognitive Tutor from Example-Tracing Tutors• Examples guide planning• Serve as semi-automated test cases


Where does CTAT fit into the Realm of Authoring Tools?

Murray survey (1999) discusses 7 categories of authoring tools for intelligent tutoring systems (ITS), including:

• Device Simulation - RIDES (Munro et al 1997)• Tutoring Strategies - REDEEM (Ainsworth et al 2003;

Major et al 1997)• Domain Expert System - Demonstr8 (Blessing 1997)

Cognitive Tutor Authoring Tools:• “Domain Expert System” category of authoring systems• No commercially successful tools for authoring model-

tracing, Cognitive Tutors• First to provide functionality to simplify & speed cog.

tutor development

Behavior Recorder

Data Shop

CTAT’s Modular Architecture

Student Interface(or external problem-solving environment)

Cool Modes

CyclePad

Java Swing

Flash

Learner Management System

LMS

Tutor Shop

Cognitive Model Development Tools

TDK (Lisp-based)

Jess tools (Java-based)

Editor

Eclipse

Tutor Engine

Tertle (Lisp-based) + Model Tracer

Jess (Java-based) + Model Tracer

Example Tracer (Java-based)

Example Tracer (Flash-based)

GUI Builder

IntelliJ

Dreamweaver

Code Warrior

Netbeans

Flash MX 2004


CTAT’s Track Record• CTAT-based tutors have been used in

experiments in Geometry, Chemistry, Chinese, and French

• Over 200 users

• Three releases in past year (1.4, 1.5, 1.6)

• 13 papers published during 2005-2006

• CTAT Web site– 36,000 unique visitors in 2005– 8,800 so far in 2006

• CTAT Downloads– 840 in 2005 (27% CMU & Pitt)– 952 so far in 2006 (4.6% CMU & Pitt)


http://ctat.pact.cs.cmu.edu


PSLC Example-Tracing Tutors built with CTAT

Chinese - tone study

Chinese - listen and readFrench culture

FrenchChemistry (stoichiometry)

Geometry (self-assessment) Geometry (on-line testing)


PSLC Case study - Chemistry (Stoichiometry)

• Goals:

o Test hypotheses about personalization and worked examples

o Exercise and test the Cognitive Tutor Authoring Tools (CTAT) in a live, web-based experiment

• Results:

o Students learned: Significant difference between pre and post

o No sig differences due to personalization or worked examples

McLaren, B. M. et al. (2006). Studying the Effects of Personalized Language and Worked Examples in the Context of a Web-Based Intelligent Tutor. In the Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, June 26-30, 2006.


PSLC Case study - Chemistry (Stoichiometry)

Behavior Recorder

Flash authoringenvironment with

tutor interface shown

Web browser withExample-Tracing tutor

embedded in HTML


CTAT Features used in Case Study • Web delivery

– Use of Macromedia Flash MX 2004 to build Student Interface

– Tutor Shop handles problem sequencing on the web

• Template-based tutor development– Use of Excel lowers skill threshold even more– More efficient authoring– Easier maintenance

• Support for experiments– Pre/post tests adminstered and graded automatically– CTAT-built tutors log all student-tutor interactions– PSLC Data Shop provides storage and data analysis

facilities


Development time comparison• Past estimates of ITS development

200:1 for robust beta versions used in real classrooms

• Example-Tracing Tutor Development25:1 for initial alpha versions: (Koedinger et al, 2004)

# Of PseudoTutors

DesignTime

Dev.Time

InstructionalTime

Design/Devto Instr.

Economics 11 3600 2190 180 32.2

Math Assistments 20 810 1170 98 20.2

LSAT 3 240 3000 180 18.0

Language Learning 8 210 575 50 15.7

Totals 4860 6935 508 23.2

• If creating a robust beta doubles dev time, we would have ~50:1 ratio -- a reduction of 4 times!

Koedinger, K., Aleven, V., Heffernan, N., McLaren, B. M., and Hockenberry, M (2004). Opening the Door to Non-Programmers: Authoring Intelligent Tutor Behavior by Demonstration;. In the Proceedings of the Seventh International Conference on Intelligent Tutoring Systems (ITS-2004).


CTAT Example-Tracing Tutor Web-Delivery Options

• Java– WebStart– Utility may depend on the situation– We are still gaining experience with this; can be a bit

tricky

• Flash (from Macromedia)– With plug-in, compatible with virtually all browsers– Easy to deploy


Flash or Java for Authoring and Delivery?

Cost Web Delivery

Look-and-Feel of Student Interface

Maturity Availability ofDevelopers

Flash - License must be purchased from Macromedia(Academic: $100, $150, for authoring only)

+ More compatible across platforms & browsers

+ Emphasis on presentation; Easy to include movies, graphics, other visuals (Currently 6 widgets)

-First released as part of CTAT in Feb 2005; Much newer than Java

- Less established programming environment, so less programmers available

Java + NetBeans and Java are free downloads

- Less compatible; Use of apps like WebStart for web delivery

- Visual elements not as crisp; more difficult to include other media (Currently 15+ widgets)

+ Available for over 2 1/2 years

+ More established programming language, so more programmers available


Movie Showing How an Example-Tracing Tutor is built


Hands-On Assignment: Build your own fraction addition Example-Tracing Tutor• To get started:

– Start Flash MX2004 (From Start menu: All Programs:Programming Tools:MX2004)

– Start the Behavior Recorder (Double click the “Behavior Recorder for Flash” icon on the desktop)

– In Flash open the partially built fraction-addition GUI (W:/Cognitive Tutor Authoring Tools/Projects/Examples/Exercise1-ExampleTracingTutor/fractionAddition.fla)

• To finish the GUI: Follow the hand-out “Building a student interface for fraction addition” (Because you have a partially built user interface, you can just review the first three pages; begin work in the middle of page 4)

• To build the Example-Tracing Tutor: Follow the hand-out: “Building a student interface for fraction addition”


THE END (Of “Introduction to CTAT and Example-Tracing Tutors”)

Documents

Introduction to the Cognitive Tutor Authoring Tools (CTAT) and Example-Tracing Tutors Bruce McLaren Systems Scientist, Co-Manager of the CTAT Project Team