1

Using concept map approaches to communicate and present knowledge

University of Oulu, FinlandEDTECH A41857 (1 credit) – Challenges,

Problems, & Future of EdTech

Wednesday March 30, 2005

Dr. Roy ClarianaPenn State University

email: RClariana@psu.eduhome: www.personal.psu.edu/rbc4

"First we build the tools, then they build us!" -- Marshall McLuhan

2

goals

Your take aways:• Some experiences with collaborative concept

mapping, mindmapping

• Some understanding of how/why it works

• Some examples that you could implement on Monday morning in your classroom or in you research

Your Digital Portfolio for future reference and for sharing

3

1 credit option

Digital portfolio – Formulate as a group a  digital portfolio of mindmapping, which you may utilize in the future in your studies or work. • For teachers, provide specific examples for

using mind mapping in your classroom

• For researchers, provide specific examples for using mind mapping in your research

4

2 credit option?

Digital Portfolio plus a White paper – a 5-10 page (double-spaced)

persuasive review of some aspect of mind mapping, i.e., scripting MM in CSCL, MM as an artifact, etc. [Based on your intuition, describe how a MM can work, this is your first iteration of a “solution”. The White papers is a “soft sell” for your “solution” that describes the problem (90% of the document) and then states clearly how your solution solves the problem (10%). Avoid straw man arguments.]

5

If you are interested…

Manuscript for presentation – I hope that we can publish this experience, i.e., based on several projects we will do, together we formulate questions, collect and analyze data, write… (this will likely go beyond the workshop time frame and is also more open-ended)

For example:How does interaction develop/evolve in online collaborative mind mapping?What scripts can support online collaborative mind mapping?

6

Agenda for today

Welcome and introductions all around Q&A Brief overview of concept maps Intro to Cmap tools software Brainstorm activity (group roles) Set up Project 1 (see handout) Set up Project 2 (see handout) Does anyone have any student essays that we

can use in Project 3 on Monday?Click here for projects handout

7

some terminology Concept map – diagrams indicating interrelationships

among concepts and representing conceptual frameworks within a specific domain of knowledge (vanBoxtel)

Concept map – a visual set of nodes and arcs (a network representation) that embodies the relationships among the set of concepts. Also called knowledge maps, mindmaps, semantic maps (Turns, et al.).

Nodes – terms/complexes/concepts (usually nouns, things, examples, ideas, categories, people, locations…)

Links (arcs) – lines connecting nodes, usually labeled with a relationship term (usually verbs)

Propositions – node-link-node combinations, also called “soup” (ketti) by IHMC

Turns, Atman, & Adams, 2000

Vygotsky

contrast

Some foundation stuff

8

Mindmaps vs. concept maps

Bahr (2004) using concept maps to teach English to German students

9

Mindmap of “group” knowledge (Anni, Anna, Paula, Esa, ja Herkko), source is

the second floor hallway

muistahuumori !

konkretisoi !

opettajanoma tarina

elävöittää

kytkeoppilaanarkeen !

liikuta oppilas ylös penkistä

kikkoja

istumajärjestys !

huiputa !

perustele !

haasta,kysele !

yllätä !

oppettajanvaikutus- mahdollisuudet

pelkkäkalvoshow

samatyötapa

liian pitkään

vältä ! työtavat

vaihto kyllin usein

demot,konkreettisetesimerkit

tekeminen vs. pelkkäkuunteleminen

nopeat

oppilaiden erot

lisätehtäviä

hitaat

tukiopetus. apu

huomaaerot

luokkakohtaiseterot

ikäluokkavaikuttaa

näennäinenkeskittyminen ?

hiljainenluokka

vilkas luokka

erityisenpaljonkikkoja

eipalautettaopettajalle

10

Mindmaps vs. concept maps

My question is, do concept maps or do mindmaps fit better with the Finnish language?

11

Tools to support mapping

Yellow stickies!! Pencil and paper may be best for your classroom

Software – PowerPoint is pretty good Inspiration is good but expensive CMAP tool is free, but your tech person

will have to agree to support it At least 22 other tools are available,

some free some not

12

Other concept map automatic scoring approaches

CMap tools (IHMC) that we will use today C-TOOLS – Luckie (PI), University of Michigan NSF

grant available: http://ctools.msu.edu/ctools/index.html TPL-KATS – University of Central Florida (e.g., Hoeft,

Jentsch, Harper, Evans, Bowers, & Salas, 1990). TPL-KATS: concept map: a computerized knowledge assessment tool. Computers in Human Behavior, 19 (6), 653-657.

SEMNET – http://www.semanticresearch.com/about/ CMAT – Arneson & Lagowski, University of Texas,

http://chemed.cm.utexas.edu Plus 22 other non-scoring map tools, Inspiration,

Kidspiration

13

Some previous uses of mapping

Usually involve individuals working alone, and involve text in some way

Some collaborative strategies have been used

Lets look at a few…

14

Using a student mindmap to “capture” a text (note taking)

Textbook

Text text text text text text text text text text text

text

texttext

Mindmap notes

student

text

memo

Examples?

15

Using a student mindmap to “capture” research on a topic

textText text

text text tex Text text text text

textttext

texttext

Mindmap notes

student

text

memo

textText text

text text tex Text text text text

textt

www

video

Examples?

video

16

Then using the mindmap to write an essay

essay

Text text text text text text text text text text text

text

texttext

Mindmap notes

student

text

memo

Examples?

17

Using a researcher drawn mindmap to “capture” an interview transcript

Interview 1

Text text text text text text text text text text text

text

texttext

The capability and experience of the person coding the text is critical…

Interview 1

coder

text

memo

attributetheory noteissue

18

Using a group drawn mindmap to “capture” an interview

text

texttext

text

The capability and experience of the person coding the text is critical…

Interview 1

interviewer

Qs

19

Example of dyad collaboration (not online)

text

texttext

text

text

texttext

text

Yergin

Mindmap artefact Verbal discussion (taped)

Analyze the discussion

Blah blah blah blah Blah blah

Hannah

Blah blah blah blah Blah blah

Observations:On taskAbstract talk3-propositions/minQuestionAnswerCriticizeConflictElaborationCo-construction

van Boxtel, van der Linden, Roelofs, & Erkens (2002)

Problem: Sometimes unscientific notions are ingrained

Inferred:Active use of prior knowledgeAcknowledged problemsLook for meaningful relationsNegotiation

Shared objects play an important role in negotiation and co-construction

The incredible value of talk!

Note the attentional effects of the artifact

20

Chiu et al. example of an online collaboration

p.22, Chiu, Huang, & Chang (2000)

text

texttext

text

text

texttext

text

Hannah(lead)

Jari

Yergin

H: WE should …J: Did you see…Y: Yeah, but …Etc.Etc.

Mindmap artefact

Online chat

Analyzed the chat textAnd the mindmap

creates

Mindmap session lasted 80 minutes. 3 x 12 online groups, communicate by chat, 745 messages were exchanged (avg. of 62 per group).

Only the lead could alter the mindmap

The ‘other 2 members used chat to “advise”

Researchers

21

Project 1 and 2 We will experiment with two online collaboration

approaches Project 1 is a synchronous concept map

collaboration using Cmap tools software Project 2 is an asynchronous concept map

collaboration using PowerPoint software and email But next, we will try brainstorming with Cmap tools to

become familiar with the tools and process before setting up Project 1

Click here for projects handout

22

First Mind map CSCL roles… Starter: You work as a discussion moderator. Your assignment is to engage your

group members to the discussion by asking questions and commenting. And if the wrapper makes small summaries during discussion you can utilize his or her work to raise new questions. Active participation in the discussions is essential.

Wrapper: Your assignment is to sum up the discussion. If you think it is easier you can summarize frequently and weave ideas together. For example, if five participants of your group are having a discussion about collaborative and co-operative learning you can summarize their main points during the discussion. An alternative way is to sum up the discussions in the end of article-videoclip task (and the last course assignment). Please overview your group's discussions and make a brief summary of the main topics. Active participation in the discussions is essential.

Group member: Your assignment is to participate actively into discussions by asking questions making comments and stating arguments. You are expected to be a critical inquirer.

Evaluator (an optional role): You are required to evaluate your group's work during the course. Please focus on the group interaction and group dynamics, for example how the starters, wrappers and group members performed during the discussions and last course assignment. The tutors inform you when to perform evaluations. Notice that you are also a deputy starter and a deputy wrapper if the originally named persons are not available. If you are called to work as a starter or wrapper please see the instructions given above. The role of evaluators are used only if you have not had a role of starter or wrapper during this course.

Mindmap activity…

23

Cluster analysis

Brainstorming(corpus list)

Sorting(move like terms closer)

Merging & Pruning(combine like terms,

delete or move unlike terms,synthesize terms)

enter

Naming Clusters(name the categories/themes)

Sorting Clusters(move like clusters closer)

Naming broad themes(name the cluster of clusters)

and if necessary

E-document (to save/print)

Build consensus!

24

Brainstorm, then make the map

Open IHMC Cmap tools Fill in personal information on first use (I’ll tell

you what to type in here) Click Other Places Open brainstorm file Click collaborate icon

if necessary Type in your first name Collaborate

25

Now go back andadd Small Group RolesGroup Task RolesInitiator-contributor. Proposes new ideas or approaches to group problem solving; may

suggest a different approach to procedure or organizing the problem-solving taskInformation seeker. Asks for clarification of suggestions; also asks for facts or other

information that may help the group deal with the issues at handOpinion seeker. Asks for clarification of the values and opinions expressed by other

group membersInformation giver. Provides facts, examples, statistics, and other evidence that pertains

to the problem the group is attempting to solveOpinion giver. Offers beliefs or opinions about the ideas under discussionElaborator. Provides examples based on his or her experience or the experience of

others that help to show how an idea or suggestion would work if the group accepted a particular course of action

Coordinator. Tries to clarify and note relationships among the ideas and suggestions that have been provided by others

Etc..

Mindmap activity…

26

Project 1 – Cmap tools synchronous collaboration

(see the Project handout)

Set day and time to join online …….

27

Project 1

IHMC Public Cmaps conv v2 on Jan 22 2004

28

Oulu EDTECH Public

Project 1

29

Project 2 – Overview of “Pass the soup”

Email to

Email to

Email to

Email to

PowerPoint file

(see the Project handout)

30

Project 2 – “Pass the soup” PowerPoint file

Slide 1 – mindmap is developed bit-by-bit here by the group by adding only 3 to 5 elements and then emailing it to the next person on the list

Slide 2 – numbered list of names of group members with email address, other instructions

Slide 3, 4, etc. – comments about changes that you want to make, suggestions, etc.

1. Bob – bob@oulu.fi2. Mary – mary@oulu.fi3. Tiina – tinna@oulu.fi4. Etc.

Instructions: Add 3 or 4components, pass to thenext person…

B: I decided to add blah and blah because I am interested in artifacts

M: I deleted Bob’s blah because it is stupid, and then added blah

T: I linked blah and blahetc…

31

How to use ALA-Reader

Monday, April 4, 2005

32

Agenda for today Debrief “pass the soup” activity, and come up

with a better Finnish name for it Q&A Brief overview of my concept map

assessment research ALA-Reader demo (English language

essays) Set up Project 3 for Finnish (see handout) How can we find Finnish essays for use in

Project 3?

33

Final map for Project 2: Team 1Why don’t we read from computer screens concept map?

Poor screen resolution (96 dpi)

paper more portable

paper has weight,texture, and feel

easy to underline andwrite notes on paper

familiarity

with paper,easier to multi-task

several paper pagessimultanosly viewed and

comparedworking options/possiblities

and requirements

appearance

easier to makegood-looking

slides and copiesabout drafts with

computer

paper has better contrast

comp screen is smaller than paper

computer to store, paper to read

computers require

computers require constant updatesmanual dexterity

of child and adultfeelings/perceptions

computer to communicate, paper

to study

computers skills

Computer screens

Group: Tanja, Henna & Roy

Click her toSee progression

Of this map

34

Final map for Project 2: Team 2

Why don’t we read text fromcomputers?

screensize

Paper /Hard copy

e-book

resolution

luminosity

able to write notes

possibility to store

underline

own commentsreliable

archive

paper easy toread for eyes

doesn’t need any hardware

never seen one

easy to use etext feelsephemeral

book feelscomfortable

can make papercopies my own

connections

technical problems

differentversions

reliable but heavy to travel with

electronic documents

light to carry/ travel

personalpreference

amount of text

1 page or less

screen

2-10pages

print topaper

copyright

e-text easy to copy/paste

eyes gets mixed up

ergonomy

multimedia

decision toprint

headache

shoulder problems

Group: Maria, Paivi & Roy

Click her toSee progression

of this map

35

Debriefing

What happened? What worked? What did not work? What would you do differently next time?

If you like, write this up as a team for your final paper.

36

My research interests

Mind map assessment – automatic scoring software tool called ALA-Mapper http://www.personal.psu.edu/rbc4/ala.htm

Essay assessment – automatic scoring software tool called ALA-Reader http://www.personal.psu.edu/rbc4/score.htm

for Latent Semantic Analysis (LSA) see: http://www.personal.psu.edu/rbc4/frame.htm

prototypes

37

Novak

Novak says “Concept maps were first developed in our research program in 1972 as a way to represent changes in children’s understanding of science concepts over the 12-year span of schooling.  We were using modified Piagetian clinical interviews to assess changes in their knowledge over time, but we found the interview transcripts were too difficult to analyze for changes in specific aspects of the children’s knowledge.  Instead we prepared concept maps from the interviews.” 

From: http://wwwcsi.unian.it/educa/mappeconc/jdn_an2.html

38

First uses… to represent knowledge in a visual format

lungs

oxygenateblood

removeCO2

pulmonaryartery

pulmonaryvein

leftatrium

rightventricle

lungs

oxygenateblood

removeCO2

pulmonaryartery

pulmonaryvein

leftatrium

rightventricle

The primary parts of the system are the heart, blood cells, and vessels. The human heart, a pump, is made of cardiac muscle Cardiac muscles have a unique feature of forming connections between two adjacent cardiac cells. This allows the muscle cells to contract powerfully and quickly involuntarilyThe brain is unable to increase or decrease the heart's beating The heart is comprised of four chambers; two upper chambers called atriums, and two lower chambers called ventriclesThe blood flows through the right side to the lungs where it picks up oxygen. The blood then returns to the right. Next, it flows into the left where it I xxxx

tissue within the body by approximately 9 pints of blood through 100,000 miles of vesselsThe primary parts of the system are the heart, blood cells, and vessels. The human heart, a pump, is made of cardiac muscle Cardiac muscles have a unique feature of forming connections between two adjacent cardiac cells. This allows the muscle cells to contract powerfully and quickly involuntarilyThe brain is unable to increase or decrease the heart's beating The heart is comprised of four chambers; two upper chambers called atriums, and two lower chambers called ventriclesThe blood flows through the right side to the lungs where it picks up oxygen. The blood then returns to the right. Next, it flows into the left where it I xxxx

The human circulatory system is a transportation system. Nutrients and oxygen are carried to living tissue within the body by approximately 9 pints of blood through 100,000 miles of vesselsThe primary parts of the system are the heart, blood cells, and vessels. The human heart, a pump, is made of cardiac muscle Cardiac muscles have a unique feature of forming connections between two adjacent cardiac cells. This allows the muscle cells to contract powerfully and quickly involuntarilyThe brain is unable to increase or decrease the heart's beating The heart is comprised of four chambers; two upper chambers called atriums, and two lower chambers called ventriclesThe blood flows through the right side to the lungs where it picks up oxygen. The blood then returns to the right. Next, it flows into the left where it I xxxx

Novak interview dataWas science content knowledge

Mind Map

39

Finnish research withconcept maps… Mainly for knowledge representation for instructional use but also for

representing the structure of a curriculum and for group communication Pasi Eronen, Jussi Nuutinenn and Erkki Sutinen, (http://

www.cs.joensuu.fi/pages/avt/concept.htm), Joensuu (computer science) Mauri Ählberg, Helsinki (education) and Erkki Rautama (computer science) University of Art and Design, Helsinki (

http://www2.uiah.fi/~araike/papers/articles/CinemaSense_Collaborative_Cinemastudies_DeafWay2002.htm) (see also: Future Learning Environment 3)

Text graphs (Helsinki): http://www.cs.hut.fi/Research/TextGraph/ Kari Lehtonen, Helsinki Polytechnic, concept maps as a portfolio component

(http://cs.stadia.fi/~lehtonen/DPF/dpf-berlin-02-muotoiltu.doc) Also School astronomy and Vocational Training and Education  4th IEEE International Conference on Advanced Learning Technologies

Joensuu, Finland, August 30 - September 1, 2004

40

Concept map for assessment: score validity???

Concept maps contains propositions

These propositions scores are generally considered to be valid and reliable measures of science content knowledge organization (Ruiz-Primo, Schultz, Li, Shavelson, CREST in California. . .).

essaysinterviews

tests

lungsoxygenate

blood

CO2artery

pulmonary

atriumventricle

veinlungs

oxygenate

blood

CO2artery

pulmonary

atriumventricle

vein

lungs

oxygenateblood

removeCO2

pulmonaryvein

leftatrium

lungs

oxygenateblood

removeCO2

pulmonaryvein

leftatrium

observations

41

e.g.,…

Rye and Rubba (2002) reported that traditional concept map scores were related to California Achievement total test scores (r = 0.73). (Note that Crocker and Algina say that validation coefficients rarely exceed r=0.50.)

Concept maps (cognitive maps, concept maps) may be an appropriate approach for assessing structural knowledge (Jonassen, Beissner, & Yacci, 1993).

For example, concept maps have been used to visualize the change from novice to expert.

42

Scoring Concept Maps

Traditionally, concept maps are scored by teachers or trained raters using scoring rubrics (e.g., Lomask’s rubric)

Although this marking approach is time consuming and fairly subjective, map scores usually correlate well with more traditional measures of science content knowledge (multiple choice, fill-in-the blank, and essays)

Complex scoring rubrics decrease the concept map score reliability (so keep scoring simple)

43

Scoring Concept Maps

C3 describes our automatic system for scoringconcept maps:

collect –>convert –> compare

1. Collect raw map data2. Convert raw data into a mathematical network

representation3. Compare the mathematical network

representation of two maps (e.g., student to teacher, student to expert, student to student)

44

1. Collect raw data

What raw data can a computer “extract” from a concept map?

Term counts – in open-ended maps, count required terms included

Propositions – a link connecting two terms and a link label

Associations – geometric distance between pairs of terms. Small values indicate stronger relationship.

45

Link and distance data

Distance Array

Link Array

lungs

oxygenated deoxygenated

pulmonary artery

pulmonary vein

left atrium

right ventricle

a b c d e f ga left atrium - b lungs 0 - c oxygenate 0 1 - d pulmonary artery 0 1 0 - e pulmonary vein 1 1 0 0 - f deoxgenate 0 1 0 0 0 - g right ventricle 0 0 0 1 0 0 -

a b c d e f ga left atrium - b lungs 120 - c oxygenate 150 36 - d pulmonary artery 108 84 120 - e pulmonary vein 73 102 114 138 - f deoxgenate 156 42 54 84 144 - g right ventricle 66 102 138 42 114 120 -

moves through

to the

passes into

to the

Most approaches use only link label information, usually called “propositions”.

(n2-n)/2 pair-wise comparisons

46

Link and distance

Link data (propositions) – are the common way to compare/assess concept maps

Distance data – not common, based on James Deese’s (1965) ideas on the structure of association in language and thought, card-sorting task approaches (Vygotsky in Luria, 1979, Miller, 1969), Kintsch and Landauer’s ideas on representing text structure, and neural network methods (Elman, e.g., 1995)

47

Using our Finnish Mind Map example

Borrowed from Anni, Anna, Paula, Esa, ja Herkko

Found in the hallway on the second floor

See next slide

48

muistahuumori !

konkretisoi !

opettajanoma tarina

elävöittää

kytkeoppilaanarkeen !

liikuta oppilas ylös penkistä

kikkoja

istumajärjestys !

huiputa !

perustele !

haasta,kysele !

yllätä !

oppettajanvaikutus- mahdollisuudet

pelkkäkalvoshow

samatyötapa

liian pitkään

vältä ! työtavat

vaihto kyllin usein

demot,konkreettisetesimerkit

tekeminen vs. pelkkäkuunteleminen

nopeat

oppilaiden erot

lisätehtäviä

hitaat

tukiopetus. apu

huomaaerot

luokkakohtaiseterot

ikäluokkavaikuttaa

näennäinenkeskittyminen ?

hiljainenluokka

vilkas luokka

erityisenpaljonkikkoja

eipalautettaopettajalle

49

Collect Mind Map raw data

hilja

inen

luok

ka

huom

aa e

rot

kikk

oja

luok

kako

htai

set e

rot

oppe

ttaja

n va

ikut

us-m

ahdo

llisu

udet

oppi

laid

en e

rot

työt

avat

vilk

as lu

okka

vältä

Link arrayhiljainen luokka -- huomaa erot 0 -- kikkoja 0 0 -- luokkakohtaiset erot 1 1 0 -- oppettajan vaikutus-mahdollisuudet 0 1 1 0 -- oppilaiden erot 0 1 0 0 0 -- työtavat 0 0 0 0 1 0 -- vilkas luokka 0 0 0 1 0 0 0 -- vältä 0 0 0 0 1 0 0 0 --

Distance arrayhiljainen luokka -- huomaa erot 127 -- kikkoja 245 199 -- luokkakohtaiset erot 79 52 225 -- oppettajan vaikutus-mahdollisuudet 214 122 100 164 -- oppilaiden erot 161 91 290 93 205 -- työtavat 234 111 175 164 76 166 -- vilkas luokka 73 117 288 68 232 105 227 -- vältä 302 207 114 252 88 282 122 320 --

9 main terms selected here (ALA-Mapper max=30)

50

Selecting terms

Selecting important terms (and their synonyms) is a critical step (for example, singular value decomposition in LSA derives terms). We use an expert(s) to determine terms.

Goldsmith, Johnson, and Acton (1991)

51

predictive validity of PFNets directly relates to the number of terms used

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0 10 20 30

pred

ictiv

e va

lidity

Number of terms

Goldsmith, Johnson, and Acton (1991)

So, perhaps the predictive validity of Concept Maps (and essays) directly relates to the number of terms used

52

2. Convert raw data into scores

Currently, we use a data reduction and comparison approach called Pathfinder network representation (PFNet, Schanveldt, 1990). Our future research will consider additional approaches, such as MDS and data-mining. http://interlinkinc.net/Pathfinder.html

PFNets describe the least weighted path to connect the terms

Scores are established by comparing the participant’s PFNet to a referent (expert) PFNet, and calculating the number of common links (the intersection)

Visual example

53

Finnish example: PFNet for distance data

kikkoja

oppettajan vaikutusmahdollisuudet

työtavat

oppilaiden erot

huomaa erot

luokkakohtaiset erot

hiljainen luokka

vilkas luokka

vältä

PFNet for distance data

54

Compare student to expert referent

kikkoja

oppettajan vaikutusmahdollisuudet

työtavat

oppilaiden erot

huomaa erot

luokkakohtaiset erot

hiljainen luokka

vilkas luokka

vältä

Expert Referent PFNet

kikkoja

oppettajan vaikutusmahdollisuudet

työtavat

oppilaiden erot

huomaa erot

luokkakohtaiset erot

hiljainen luokka

vilkas luokka

vältä

Student PFNet

O

O6 of 8 common links

55

Poindexter and Clariana

Participants – 23 undergraduate students in intro EdPsyc course (Penn State Erie)

Food rewards for participation Setup – complete a demographic survey

and how to make a concept map lesson Text based lesson interventions –

instructional text on the “heart” with either proposition specific or relational lesson approach

Poindexter, M. T., & Clariana, R. B. (in press). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), in press. link to doc file

#1st

56

Treatments Relational condition, participants were required to

“unscramble” sentences (following Einstein, McDaniel, Bowers, & Stevens, 1984) in one paragraph in each of the five sections or about 20% of the total text content

Proposition-specific condition (following Hamilton, 1985), participants answered three or four adjunct constructed response questions (taken nearly verbatim from the text) provided at the end of each of the five sections, for a total of 17 questions covering about 20% of the total text content (no feedback was provided).

57

Posttests

Concept map (use 26 terms provided)• Link-based common scores

• Distance-based common scores

Multiple-choice tests (Dwyer, 1976)• Identification (20)

• Terminology (20)

• Comprehension (20)

58

Means and sd

Treatments Posttests ID TERM COMP Map-prop Map-assoc control 15.1 12.3 7.3 14.1 9.0

(4.4) (4.6) (5.4) (4.6) (3.6)

proposition- 16.3 14.6 13.8 16.5 11.5 specific

(5.6) (5.7) (3.7) (8.3) (3.4)

relational 17.0 12.7 12.4 13.9 10.7 (2.6) (3.5) (3.0) (9.4) (4.6)

Map-link Map-dist

59

Analysis

MANOVA (relational, proposition-specific, and control) and five dependent variables including ID, TERM, COMP, Map-prop, and Map-assoc.

COMP was significance, F = 5.25, MSe = 17.836, p = 0.015, none of the other dependent variables were significance.

Follow-up Scheffé tests revealed that the proposition-specific group’s COMP mean was significantly greater than the control group’s COMP mean (see previous Table).

60

Correlations

ID TERM COMP Prop ID -- TERM 0.71 -- COMP 0.50 0.74 -- Map-prop 0.56 0.77 0.53 -- Map-assoc 0.45 0.69 0.71 0.73 All sig. at p<.05

Compare to Taricani & Clariana

next

Map-link

Map-linkMap-distance

61

Taricani and Clariana – Replication of Poindexter and Clariana

Taricani, E. M. & Clariana, R. B. (in press). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 53 (4), in press.

TermComp

Link data 0.78 0.54

Distance data 0.48 0.61

62

Compare these two . . .

Poindexter & Clariana TermComp

Link data 0.77 0.53

Distance data 0.69 0.71

Taricani & Clariana TermComp

Link data 0.78 0.54

Distance data 0.48 0.61

63

Clariana, Koul, & Salehi

Participants – A group of 24 practicing teachers enrolled in CI 400

Lesson intervention – while researching online, completed concept maps in pairs (newsprint & yellow stickies) to describe the structure and function of the heart and then individually wrote essays on this topic from their maps.

Clariana, R. B., Koul, R., & Salehi, R. (in press). The criterion related validity of a computer-based approach for scoring concept maps. International Journal of Instructional Media, 33 (3), in press.

# 2nd

64

Posttests

Essays Multiple-raters using holistic rubric Computer-derived LSA Essay scores

(http://www.personal.psu.edu/rbc4/frame.htm)

Concept Maps Multiple-raters using Lomask’s rubric ALA-Mapper PFNet link and distance

agreement with an expert

65

Correlation matrix

Map Essay LSA LinkMap 1 Essay 0.49 1 LSA 0.31 0.73 1 Link data 0.36 0.76 0.83 1 Distance data 0.60 0.77 0.71 0.82 1

p < .05 shown in boldface type.

Human Computer

Many investigators have noted the close relationship between maps and essays.

66

Overview: Tools to score Essays

ETS – PEG (Project Essay Grade), e-rater, Criterion and other products… http://www.ets.org/research/erater.html

Walter Kintsch (and Landau) at CU-Boulder – Latent semantic analysis, many uses, i.e., score online training for the Army - http://lsa.colorado.edu/

Vantage Learning essay scoring products - http://www.vantagelearning.com/

ALA-Reader: http://www.personal.psu.edu/rbc4/score.htm

67

ALA-Reader

… an electrical signal starts the heartbeat, by causing the atrium to contract. The blood then flows through the pulmonary valve into the pulmonary artery and then into the lungs. Once inside the lungs, the blood gives up the carbon dioxide (cleansed) and receives oxygen. This oxygenated blood …

atrium

contract

lungs

cleansed oxygenated

P artery

P valve

Text PFNet

Link array

68

Clariana & Koul

Participants – Again, a group of 24 practicing teachers enrolled in CI 400

Lesson – while researching the topic “the structure and function of the heart” online, students completed concept maps using Inspiration software and later wrote an essay on this topic from their maps.

Clariana, R.B., & Koul, R. (2004). A computer-based approach for translating text into concept map-like representations. In A.J.Canas, J.D.Novak, and F.M.Gonzales, Eds., Concept maps: theory, methodology, technology, vol. 2, in the Proceedings of the First International Conference on Concept Mapping, Pamplona, Spain, Sep 14-17, pp.131-134. http://cmc.ihmc.us/papers/cmc2004-045.pdf

# 3rd

69

Posttests

Essays Multiple-raters using holistic rubric Computer-derived LSA Essay scores

(http://www.personal.psu.edu/rbc4/frame.htm)

Concept Maps Multiple-raters using Lomask’s rubric ALA-Mapper PFNet link and distance agreement

with an expert ALA-Reader PFNet link scores (from 1 to 5)

(so far, only looked at essay scores)

70

ALA-Rater PFNet scores

The scores for each text and rater-pair are shown ordered from best to worst.

ALA-Reader scores were moderately related to the combined text score, Pearson r = 0.69, and ranked 5th overall.

71

Comments and Questions

??

72

Demo ALA-Reader

Download ALA-Reader.exe Create terms file (can include 2 synonyms) Create 2 expert baseline reference texts called

expert1.txt and expert2.txt (i.e., Instructor, best student)

Use it (type in the students essay file name) Files created

• Summary file called report.txt

• Multiple *.prx files (PRX folder)

Available at: www.personal.psu.edu/rbc4

73

Other methods for eliciting and representing knowledge structure

Monday, April 11, 2005

74

agenda

Today is a hands-on demonstration day Brief overview of the ideas SPSS for representing Pathfinder KU-Mapper

My intent, you will know enough to begin to use these approaches

75

Eliciting structural knowledge

Every method for eliciting knowledge should be viewed as “sampling”

Caution, never forget the likely effects of contiguity (time, space, etc.) dominating over semantics (meaning)

essaysinterviews

tests

lungsoxygenate

blood

CO2artery

pulmonary

atriumventricle

veinlungs

oxygenate

blood

CO2artery

pulmonary

atriumventricle

vein

lungs

oxygenateblood

removeCO2

pulmonaryvein

leftatrium

lungs

oxygenateblood

removeCO2

pulmonaryvein

leftatrium

observations

76

Dave’s ideas

Knowledgerepresentation

Knowledgecomparison

Knowledgeelicitation

Jonassen, Beissner, & Yacci (1993), page 22

77

Dave’s ideas

graphbuilding

similarityratings

semanticproximity

wordassociations

cardsort

orderedrecall

freerecall

additivetrees

hierarchicalclustering

orderedtrees minimum

spanningtrees

linkweighted

Pathfindernets

NetworksDimensional

principalcomponents

MDS – multidimensional scaling

clusteranalysis

expert/novice

qualitativegraph

comparisons

quantitativegraph

comparisons

relatednesscoefficients

scalingsolutions

C of PFNets

Trees

Knowledgerepresentation

Knowledgecomparison

Knowledgeelicitation

Jonassen, Beissner, & Yacci (1993), page 22

78

Eliciting structural knowledge

Vygotsky (in Luria, 1979); Miller (1969) card-sorting approaches

Deese’s (1965) ideas on the structure of association in language and thought

Kintsch and Landauer’s ideas on representing text structure, and latent semantic analysis

Recent neural network representations (e.g., Elman, 1995)

79

Analyzing Deese free association data with MDS

Hands-on with MDS in SPSS• A good description of MDS:

http://www.statsoft.com/textbook/stmulsca.html

• (Aside: a good description of Factor analysis: http://www.statsoft.com/textbook/stfacan.html )

Hands-on with Pathfinder KNOT

80

Deese, free recall data (p.56)

mot

h

inse

ct

win

g

bird

fly yello

w

flow

er

bug

coco

on

colo

r

blue

bees

sum

mer

suns

hine

gard

en

sky

natu

re

sprin

g

butt

erfly

moth 100 12 12 12 11 1 0 4 11 0 0 2 2 5 1 1 1 1 15insect 12 100 9 9 17 1 1 33 10 1 1 3 0 0 0 0 1 0 12wing 12 9 100 44 19 0 0 3 2 0 0 10 0 0 0 0 3 0 13bird 12 9 44 100 21 1 0 3 2 1 1 10 0 1 0 1 5 0 12fly 11 17 19 21 100 1 1 8 6 1 2 6 0 3 0 2 4 0 11yellow 1 1 0 1 1 100 7 0 0 17 23 2 2 7 5 2 4 3 5flower 0 1 0 0 1 7 100 2 0 3 7 2 1 6 18 2 6 2 6bug 4 33 3 3 8 0 2 100 7 0 0 5 0 0 0 0 2 0 4cocoon 11 10 2 2 6 0 0 7 100 0 0 4 1 1 1 0 2 0 22color 0 1 0 1 1 17 3 0 0 100 32 0 0 2 0 8 0 0 0blue 0 1 0 1 2 23 7 0 0 32 100 1 2 4 4 46 3 2 2bees 2 3 10 10 6 2 2 5 4 0 1 100 1 2 3 0 4 2 7summer 2 0 0 0 0 2 1 0 1 0 2 1 100 5 2 0 1 10 0sunshine 5 0 0 1 3 7 6 0 1 2 4 2 5 100 2 3 2 15 4garden 1 0 0 0 0 5 18 0 1 0 4 3 2 2 100 0 4 4 2sky 1 0 0 1 2 2 2 0 0 8 46 0 0 3 0 100 0 1 0nature 1 1 3 5 4 4 6 2 2 0 3 4 1 2 4 0 100 2 3spring 1 0 0 0 0 3 2 0 0 0 2 2 10 15 4 1 2 100 2butterfly 15 12 13 12 11 5 6 4 22 0 2 7 0 4 2 0 3 2 100

Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

Full array (n * n): 19 x 19 = 361Half array ((n2 – n)/2): ((19 x 19) –19 )/2 = 171

100 participants are shown a list of related words, one at a time, and asked to free recall a related term

81

Deese, free recall data (p.56)m

oth

inse

ct

win

g

bird

fly yello

w

flow

er

bug

coco

on

colo

r

blue

bees

sum

mer

suns

hine

gard

en

sky

natu

re

sprin

g

butt

erfly

moth 100 12 12 12 11 1 0 4 11 0 0 2 2 5 1 1 1 1 15insect 12 100 9 9 17 1 1 33 10 1 1 3 0 0 0 0 1 0 12wing 12 9 100 44 19 0 0 3 2 0 0 10 0 0 0 0 3 0 13bird 12 9 44 100 21 1 0 3 2 1 1 10 0 1 0 1 5 0 12fly 11 17 19 21 100 1 1 8 6 1 2 6 0 3 0 2 4 0 11yellow 1 1 0 1 1 100 7 0 0 17 23 2 2 7 5 2 4 3 5flower 0 1 0 0 1 7 100 2 0 3 7 2 1 6 18 2 6 2 6bug 4 33 3 3 8 0 2 100 7 0 0 5 0 0 0 0 2 0 4cocoon 11 10 2 2 6 0 0 7 100 0 0 4 1 1 1 0 2 0 22color 0 1 0 1 1 17 3 0 0 100 32 0 0 2 0 8 0 0 0blue 0 1 0 1 2 23 7 0 0 32 100 1 2 4 4 46 3 2 2bees 2 3 10 10 6 2 2 5 4 0 1 100 1 2 3 0 4 2 7summer 2 0 0 0 0 2 1 0 1 0 2 1 100 5 2 0 1 10 0sunshine 5 0 0 1 3 7 6 0 1 2 4 2 5 100 2 3 2 15 4garden 1 0 0 0 0 5 18 0 1 0 4 3 2 2 100 0 4 4 2sky 1 0 0 1 2 2 2 0 0 8 46 0 0 3 0 100 0 1 0nature 1 1 3 5 4 4 6 2 2 0 3 4 1 2 4 0 100 2 3spring 1 0 0 0 0 3 2 0 0 0 2 2 10 15 4 1 2 100 2butterfly 15 12 13 12 11 5 6 4 22 0 2 7 0 4 2 0 3 2 100

Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

Full array (n * n): 19 x 19 = 361Half array ((n2 – n)/2): ((19 x 19) –19 )/2 = 171

82

Using MDS in SPSS

Start SPSS and open the deese.sav file Under Analyze, select Scale, then select

Multidimensional Scaling (ALSCAL)… Move Variable from left to right Create distances from data Model Options Next page

83

Select all of these

84

-2 -1 0 1

Dimension 1

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

Dim

ensi

on

2

moth

insect

wing

birdflyyellow

flower

bug

cocoon

colorblue

bees

summer

sunshinegarden

sky

nature

spring

butterfl

Euclidean distance model

Derived Stimulus Configuration

Multi-dimensional scaling (MDS) of Deese data

85

Both are “correct”.

Side issue, the MDS obtains alternate (e.g., enantiomorphic) visual representations

Oulu

PoriTampere

Helsinki

Jyväsklyä

Oulu

PoriTampere

Helsinki

Jyväsklyä

Is this map correct?

geographic data, for example, may be oriented in different ways

86

How good is the representation?

many dimensions (as many as 19) reduced to 2 dimensions

Check the “stress” value to estimate how strained the results are

-2 -1 0 1

Dimension 1

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

Dim

ensi

on

2

moth

insect

wing

birdflyyellow

flower

bug

cocoon

colorblue

bees

summer

sunshinegarden

sky

nature

spring

butterfl

Euclidean distance model

Derived Stimulus Configuration

An algorithmic, power, approach rather than based on a model so no assumptions about data structure are required…

87

Side trip Wordnet: http://wordnet.princeton.edu/

http://wordnet.princeton.edu/cgi-bin/webwn

What is the Visual Thesaurus? – The Visual Thesaurus offers stunning visual displays of the English language. Looking up a word creates an interactive visual map with your word in the center of the display, connected to related words and meanings.

Type “bird” in at: http://www.visualthesaurus.com/trialover.jsp

88

Pathfinder Network (PFNet) analysis Pathfinder is a mathematical approach for representing

and comparing networks, see: http://interlinkinc.net/index.html

Pathfinder data reduction is based on the least weighted path between nodes (terms), so for example, Deese’s 171 data points become 18 data points. Only the salient or important data is retained.

Pathfinder PFNet uses, for example:• Library reference analysis

• Measuring Team Knowledge (Nancy J. Cooke) next slide

• Use google to see many more

89

Pathfinder for cognitive task analysis

Shope, DeJoode, Cooke, and Pedersen (2004)

90

PFNet of same data

Now let’s try Pathfinder analysis of the

same Deese data set… Find the pfnet folder Double-click to run PCKNOT.bat (notice

the bat extension, see next slide below) We will do it together

91

Select the right PCKNOT file

92

PFNet of Deese datasummer

springsunshine

yellowcolor

blue

sky

flower

garden

nature

butterfly

cocoon moth

wing

beesbird

fly

bug

insect

93

MDS and PFNet of Deese data

summer

springsunshine

yellowcolor

blue

sky

flower

garden

nature

butterfly

cocoon moth

wing

beesbird

fly

bug

insect

summer

springsunshine

yellowcolor

blue

sky

flower

garden

nature

butterfly

cocoon moth

wing

beesbird

fly

bug

insect

-2 -1 0 1

Dimension 1

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

Dim

ensi

on

2

moth

insect

wing

birdflyyellow

flower

bug

cocoon

colorblue

bees

summer

sunshinegarden

sky

nature

spring

butterfl

Euclidean distance model

Derived Stimulus Configuration

Pathfinder KNOT PFNet SPSS MDS

94

MDS and PFNet data reduction

MDS uses all of the data points to reduce the dimensions in the representation, and so may be improperly driven by noise in the data or by unimportant data points

Pathfinder uses only the most important data

95

Transition to your real life example

Finally, you will collect *real* data (using my KU-Mapper software) and

analyze it with Pathfinder KNOT

96

KU-Mapper

Your data, determine 15 important terms in your research area (Finnish and English), create a “terms.txt” file with the 15 terms

Run KU-Mapper (do all 3 tasks: pair-wise, list-wise, and card sort)

Use KNOT to analyze and compare all three prx files

Download KU-Mapper from: http://www.personal.psu.edu/rbc4/KUmapper.htm

97

Debrief your data activity

What happened? What worked? What did not work? What would you do differently next time?

If you like as your final paper, describe how you might use this approach.

98

Final thoughts…

I enjoyed working with you If you want a credit,

• Email to let me know this

• Then be sure to send me you paper via email as soon as possible

99

Stop here

100

Possible research question on optimal scripts: Under- vs. over-scripting CSCL

Amount of scripting

Am

ount

of

colla

bora

tion

linear S-curve

with crash?

J-curve

Amount of scripting Amount of scripting

Some possibilities

101

generative learning strategies

Generative learning (Jonassen, 1988) recall - repetition, rehearsal, review, mnemonics integration - learner paraphrases, generates

questions, generates examples organization - learner analyzes key ideas by creating

headings, underlining keywords, outlining, categorizing (i.e., invent table categories, populate a table with existing ideas)

elaboration - generate mental images, create physical diagrams, sentence elaboration (i.e., invent stuff to fill cells in a table)+ +

+

102

I just "think" systemically and "n-dimensionally" on paper, with imagery…

My essential skill is simply--If you can explain it to me, I can draw a picture of it. It doesn't matter if it's something totally new to me, if you can make a coherent explanation, and let me understand it. I can "visualize it" and make a picture that shows you what you said.

This is why I work in aerospace. I'm able to sit down with SME's (Subject Matter Experts-in any discipline), let them do a "data-dump" and put a sketch in their hand at the end of the conversation that "say's it all". This skill is vital to helping disparate technical types talk to each other (communication across cultural barrier of the "dialect" of the various technical disciplines). It also provides a way for ideas to get from that rough-semi coherent stage and into a practical and "do-able" condition.

For example, One day I found myself working a Kelly Temp job for a bunch of Boeing System

Analysts doing a JAD (joint application development) project to design a computing architecture for a new tooling system for the 777. The first drawing came by accident, started a huge argument, and eventually (2 weeks later) resolved in a group wide "a-hah"... that put everyone on the same wavelength-allowing the new system to be built a lot more "right" than usual, quicker than usual.

From: http://visual.wiki.taoriver.net/moin.cgi/MichaelErickson