On

Lin

king

Lea

rnin

g,

Ass

essm

ent,

and

Inte

rpre

tatio

n

Min

Li

Min

Li

Stan

ford

Uni

vers

itySt

anfo

rd U

nive

rsity

Invi

ted

Talk

U

nive

rsity

of W

ashi

ngto

nA

pril

26th

, 200

1

Ove

rvie

w o

f Tal

k

•C

once

ptua

l fra

mew

ork:

The

ass

essm

ent t

riang

le

•D

isse

rtatio

n st

udy

–ev

alua

ting

the

links

bet

wee

n sc

ienc

e ac

hiev

emen

t to

asse

ssm

ents

•C

lass

room

ass

essm

ent s

tudy

–lo

okin

g in

to st

uden

ts’

lear

ning

in th

e cl

assr

oom

thro

ugh

Scie

nce

jour

nals

•D

irect

ions

and

impl

icat

ions

Con

cept

ual F

ram

ewor

k:T

he A

sses

smen

t Tri

angl

e

The

Ass

essm

ent T

rian

gle

Lear

ning

/Ach

ieve

men

t(c

ogni

tion)

Asse

ssm

ent

Inte

rpre

tatio

n

Pelle

grin

o, C

hudo

wsk

y, &

Gla

ser,

in p

ress

Dis

sert

atio

n St

udy:

Eva

luat

ing

the

Lin

k be

twee

n Sc

ienc

e A

chie

vem

ent t

o A

sses

smen

ts

The

Ass

essm

ent T

rian

gle:

D

isse

rtat

ion

Stud

ySc

ienc

e ac

hiev

emen

t as

four

type

s of k

nowl

edge

:•

Dec

lara

tive

•Pr

oced

ural

•Sc

hem

atic

•St

rate

gic

Asse

ssm

ent m

etho

ds:

•M

ultip

le-c

hoic

e•

Free

-res

pons

e•

Perf

orm

ance

-ass

essm

ent

Eval

uate

Inte

rpre

tatio

ns:

•Lo

gica

lly

•Em

piric

ally

Def

inin

g Sc

ienc

e A

chie

vem

ent

Scie

nce

Achi

evem

ent

Scie

nce

Achi

evem

ent

Def

initi

ons,

term

s & fa

cts

Step

s, ac

tions

, &

alg

orith

ms

Theo

ries &

m

enta

l mod

els

Stra

tegi

es &

co

nditi

onal

kn

owle

dge

Dec

lara

tive

Proc

edur

alSc

hem

atic

Stra

tegi

c

Lin

king

Kno

wle

dge

Typ

es to

A

sses

smen

t Met

hods

Ass

essm

ent M

etho

d

Mul

tiple

-cho

ice,

free

-res

pons

e

Perf

orm

ance

ass

essm

ent

Mul

tiple

-cho

ice,

free

-res

pons

e, c

once

pt-m

appi

ng

Perf

orm

ance

ass

essm

ent w

ith a

n op

en st

ruct

ure

Dec

lara

tive

Kno

win

g th

at

Proc

edur

alK

now

ing

how

Sche

mat

icK

now

ing

why

Stra

tegi

cK

now

ing

abou

t kno

win

g

Kno

wle

dge

Typ

e

Met

hod:

Sam

ple

•TI

MSS

scie

nce

item

s (A

mer

ican

Pop

2)

–B

ookl

et 8

•

22 m

ultip

le-c

hoic

e•

10 fr

ee-r

espo

nse

–Pe

rfor

man

ce a

sses

smen

t tas

ks

•Te

n ex

perts

sele

cted

for t

hink

-alo

ud st

udy

–5

phys

ics a

nd 5

bio

logy

gra

duat

e st

uden

ts

Met

hod:

Pro

cedu

re

The

link

was

eva

luat

ed lo

gica

lly a

nd e

mpi

rica

lly:

The

link

was

eva

luat

ed lo

gica

lly a

nd e

mpi

rica

lly:

•Lo

gica

l ana

lysi

s–

Cod

ing

the

char

acte

ristic

s of i

tem

s and

link

ing

item

s to

kno

wle

dge

type

s

•Em

piric

al a

naly

sis

–Ex

plor

ator

y an

d co

nfirm

ator

y fa

ctor

ana

lysi

s to

exam

ine

the

unde

rlyin

g co

varia

nce

patte

rns

–Th

ink-

alou

d st

udy

to in

fer s

tude

nts’

cog

nitio

n

Log

ical

Ana

lysi

s:C

lass

ifica

tion

of T

IMSS

Item

s by

Kno

wle

dge

Type

13 56 4

3

1

Dec

lara

tive

Sche

mat

ic

Proc

edur

al

Stra

tegi

c

Num

ber o

f Ite

ms

0

Exa

mpl

e: A

Dec

lara

tive-

know

ledg

e It

em

P4.

Wha

t hap

pens

whe

n an

ani

mal

hib

erna

tes?

A.

The

re is

no

life

in a

ny o

f its

par

ts.

B.

It st

ops b

reat

hing

.

C.

Its t

empe

ratu

re is

hig

her t

han

whe

n it

is a

ctiv

e.

D.

It is

abs

orbi

ng e

nerg

y fo

r use

whe

n it

is a

ctiv

e.

E.

It is

usi

ng le

ss e

nerg

y th

an w

hen

it is

act

ive.

Exa

mpl

e: A

Pro

cedu

ral-k

now

ledg

e It

em

P1.

The

grap

h sh

ows t

he p

rogr

ess m

ade

by a

n an

t mov

ing

alon

g a

stra

ight

line

.

If th

e an

t kee

ps m

ovin

g at

the

sam

e sp

eed,

how

far w

ill it

hav

etra

vele

d at

the

end

of 3

0 se

cond

s?

A.

5

cmB

.

6cm

C.

20

cmD

.

30cm

Exa

mpl

e: A

Sch

emat

ic-k

now

ledg

e It

em

P5.

The

wat

er in

a tu

be is

hea

ted,

as s

how

n in

the

diag

ram

. As t

he w

ater

is h

eate

d, th

e ba

lloon

incr

ease

sin

size

. Exp

lain

why

.

Fact

or A

naly

sis

A g

ood

stat

istic

al fi

t in

dica

ted

by fi

t m

easu

res:

-χ2

=357

.47,

df=3

33, P

=.17

-C

FI=.

999

0, .4

1

Dec

lara

tive

.72

bsm

sa7

0, .1

7

e11

.80

bsm

sa9

0, .1

2

e21

.60

bsm

sa11

0, .1

9

e31

0, .0

5

Sche

mat

ic.5

0bs

msq

110,

.22

e20

1

.41

bsss

p50,

.22

e19

1

.23

bsss

p20,

.12

e18

.15

bsm

sb3

0, .1

2

e17

1

.52

bsm

sa12

0, .2

0

e15

1

.76

bsm

sa8

0, .1

6

e14

1

0, .0

3

Proc

edur

al.6

6bs

msr

10,

.21

e28

1

.59

bsm

sp7

0, .2

1

e27

.84

bsm

sp1

0, .1

0

e26

1

.49

bsm

sb5

0, .2

4

e25

1

.53

bsm

sq13

0, .2

1

e21

.69

bsss

q18

0, .7

1

e23

1.4

7

bsm

sr2

0, .2

3

e24

1

.91

bsm

sb4

0, .0

7

e5.8

8bs

ssp3

0, .0

9

e6

1.23

bsss

p60,

.49

e8.5

1bs

msq

140,

.22

e9.5

8bs

ssq1

70,

.21

e10

.45

bsss

r4

0, .1

8e1

2.6

1bs

ssr5

0, .1

4

e13

11 111 11

.25

1

.12

1

.55

.73

.171.00

.54

bsm

sp4

0, .2

1

e71

-.03

.02

.31

.27

.32

.36

.18

.40

bsm

sq15

0, .2

2

e22

1

.84

bses

r30,

.36

e11

1

.58

.01

1

.02

1.00

.56

1.02.52

.03

.10

.73

.91

bsm

sb1

0, .0

8

e41

.10

-.23

.18

.32

.62

bsm

sb2

0, .2

3

e16

1

-.34

-.22

.48

.33

.47

.281.00

.27

.69

.94

.66

.25

1.70

.01

-.33

.11 .0

2.22

Fact

or A

naly

sis

•K

now

ledg

e-ty

pe it

ems c

lust

ered

to

geth

er, s

how

ing

sign

ifica

nt

load

ings

on

the

thre

e kn

owle

dge

fact

ors.

•D

ecla

rativ

e, p

roce

dura

l, an

d sc

hem

atic

kno

wle

dge

fact

ors

wer

e hi

ghly

cor

rela

ted.

•C

ompa

rison

with

alte

rnat

ive

mod

els (

e.g.

, one

gen

eral

fact

or)

favo

red

the

know

ledg

e-fa

ctor

m

odel

.

0, .4

1

Dec

lara

tive

.72

bsm

sa7

0, .1

7

e11

.80

bsm

sa9

0, .1

2

e21

.60

bsm

sa11

0, .1

9

e31

0, .0

5

Sche

mat

ic.5

0bs

msq

110,

.22

e20

1

.41

bsss

p50,

.22

e19

1

.23

bsss

p20,

.12

e18

.15

bsm

sb3

0, .1

2

e17

1

.52

bsm

sa12

0, .2

0

e15

1

.76

bsm

sa8

0, .1

6

e14

1

0, .0

3

Proc

edur

al.6

6bs

msr

10,

.21

e28

1

.59

bsm

sp7

0, .2

1

e27

.84

bsm

sp1

0, .1

0

e26

1

.49

bsm

sb5

0, .2

4

e25

1

.53

bsm

sq13

0, .2

1

e21

.69

bsss

q18

0, .7

1

e23

1.4

7

bsm

sr2

0, .2

3

e24

1

.91

bsm

sb4

0, .0

7

e5.8

8bs

ssp3

0, .0

9

e6

1.23

bsss

p60,

.49

e8.5

1bs

msq

140,

.22

e9.5

8bs

ssq1

70,

.21

e10

.45

bsss

r4

0, .1

8e1

2.6

1bs

ssr5

0, .1

4

e13

11 111 11

.25

1

.12

1

.55

.73

.171.00

.54

bsm

sp4

0, .2

1

e71

-.03

.02

.31

.27

.32

.36

.18

.40

bsm

sq15

0, .2

2

e22

1

.84

bses

r30,

.36

e11

1

.58

.01

1

.02

1.00

.56

1.02.52

.03

.10

.73

.91

bsm

sb1

0, .0

8

e41

.10

-.23

.18

.32

.62

bsm

sb2

0, .2

3

e16

1

-.34

-.22

.48

.33

.47

.281.00

.27

.69

.94

.66

.25

1.70

.01

-.33

.11 .0

2.22

Prot

ocol

Ana

lysi

sEx

ampl

es o

f Exp

erts

’ Ver

baliz

atio

ns

P4.

Wha

t hap

pens

whe

n an

ani

mal

hib

erna

tes?

A.

The

re is

no

life

in a

ny o

f its

par

ts.

B.

It st

ops b

reat

hing

.

C.

Its t

empe

ratu

re is

hig

her t

han

whe

n it

is a

ctiv

e.

D.

It is

abs

orbi

ng e

nerg

y fo

r use

whe

n it

is a

ctiv

e.

E.

It is

usi

ng le

ss e

nerg

y th

an w

hen

it is

act

ive.

The

decl

arat

ive-

know

ledg

e ite

m

Prot

ocol

Ana

lysi

sEx

ampl

es o

f Exp

erts

’ Ver

baliz

atio

ns

Expe

rt 1

:“W

hat h

appe

ns w

hen

an a

nim

al h

iber

nate

s?

Oka

y, w

hat I

kno

w, h

iber

natio

n m

eans

, sle

eps f

or a

lo

ng ti

me

and

hear

t rat

e sl

ows d

own

…”

Expe

rt 2

:“Th

e an

swer

is ju

st sa

id th

ere

(fro

m re

adin

g th

e st

atem

ent)…

all

the

proc

ess s

low

s dow

n. W

ell,

I kn

ow w

hen

anim

als h

iber

nate

, the

y la

y do

wn

and

they

do

not

use

too

muc

h en

ergy

.”

Prot

ocol

Ana

lysi

sEx

ampl

es o

f Exp

erts

’ Ver

baliz

atio

ns

P5.

The

wat

er in

a tu

be is

hea

ted,

as s

how

n in

the

diag

ram

. As t

he w

ater

is h

eate

d, th

e ba

lloon

incr

ease

sin

size

. Exp

lain

why

.

The

sche

mat

icTh

e sc

hem

atic

-- kno

wle

dge

item

know

ledg

e ite

m

Prot

ocol

Ana

lysi

sEx

ampl

es o

f Exp

erts

’ Ver

baliz

atio

ns

Expe

rt 1

:“…

whe

n it

is h

otte

r, th

e at

oms m

ove

fast

er.

So th

ey e

vapo

rate

, so

ther

e is

mor

e va

por.

That

vap

or,

that

is m

ore

pres

sure

on

the

ballo

on…

”

Expe

rt 2

:“Th

e ba

lloon

incr

ease

s bec

ause

of p

ress

ure,

th

at is

wha

t cau

ses t

he b

allo

on e

xpan

ds. A

nd a

lso,

whe

n yo

u he

at so

met

hing

, eve

n w

ithou

t wat

er, w

hen

you

heat

ga

s, pr

essu

re o

f the

vol

ume

tend

s to

incr

ease

.”

Con

clus

ions

of D

isse

rtat

ion

Stud

y

•Lo

gica

l and

fact

or a

naly

ses s

uppo

rted

the

dist

inct

ions

bet

wee

n kn

owle

dge

type

s.

•Pr

otoc

ol a

naly

sis r

evea

led

diff

eren

ces i

n us

e of

kn

owle

dge

type

s par

tially

due

to it

em

char

acte

ristic

s.

Cla

ssro

om A

sses

smen

t Stu

dy:

Cla

ssro

om A

sses

smen

t Stu

dy:

Loo

king

into

Stu

dent

s’ L

earn

ing

Loo

king

into

Stu

dent

s’ L

earn

ing

in th

e C

lass

room

in

the

Cla

ssro

om

thro

ugh

Scie

nce

Jour

nals

thro

ugh

Scie

nce

Jour

nals

Scie

nce

jour

nal

is a

com

pila

tion

of e

ntrie

s th

at p

rovi

des

a pa

rtial

rec

ord

of th

e in

stru

ctio

nal e

xper

ienc

es a

stu

dent

had

in

her

cla

ssro

om d

urin

g a

certa

in p

erio

d of

tim

e.

The

Ass

essm

ent T

rian

gle:

Sc

ienc

e Jo

urna

lsLe

arni

ng/A

chie

vem

ent

Lear

ning

/Ach

ieve

men

t

Jour

nals

as

Jour

nals

as

Asse

ssm

ent T

ools

:As

sess

men

t Too

ls:

•at

the

indi

vidu

al le

vel a

nd a

t the

ag

greg

ated

cla

ssro

om le

vel.

•an

imm

edia

te/u

nobt

rusi

ve a

sses

smen

t

•O

ppor

tuni

ties t

o le

arn

-Ins

truct

iona

l im

plem

enta

tion

-Qua

lity

of te

ache

r fee

dbac

k

•St

uden

t per

form

ance

-Sci

entif

ic c

omm

unic

atio

n-C

once

ptua

l und

erst

andi

ng-P

roce

dura

l und

erst

andi

ng

The

Ass

essm

ent T

rian

gle:

Scie

nce

Jour

nals

Lear

ning

/Ach

ieve

men

tLe

arni

ng/A

chie

vem

ent

Jour

nals

as

Jour

nals

as

Asse

ssm

ent T

ools

Inte

rpre

tatio

nIn

terp

reta

tion

Asse

ssm

ent T

ools

•Can

scie

nce

jour

nals

pro

vide

trus

twor

thy

and

valid

evid

ence

on

stud

ent p

erfo

rman

ce?

•Wha

t do

jour

nals

tell

us a

bout

stud

ent p

erfo

rman

ce?

•Wha

t do

jour

nals

tell

us a

bout

opp

ortu

nity

to le

arn?

Met

hod

•Sa

mpl

e

–10

fifth

gra

de c

lass

room

s

–Tw

o Fu

ll O

ptio

n Sc

ienc

e Sy

stem

(FO

SS) u

nits

, V

aria

bles

in th

e fa

ll an

d M

ixtu

res i

n th

e sp

ring,

w

ere

taug

ht.

–A

rand

om st

ratif

ied

sam

ple

from

eac

h cl

ass:

2 lo

w,

2 m

iddl

e, a

nd 2

hig

h

Met

hod

•C

odin

g

–Ea

ch e

ntry

was

cod

ed in

to d

iffer

ent s

core

s:•I

nstru

ctio

nal i

mpl

emen

tatio

n•T

ype

of e

ntry

•Stu

dent

per

form

ance

•Tea

cher

feed

back

–A

n an

alyt

ic c

odin

g cr

iteria

def

ined

the

valu

es.

Met

hod

•Pr

oced

ures

–Pr

e-po

stte

st d

esig

n us

ing

perf

orm

ance

ass

essm

ents

–28

Var

iabl

es a

nd 2

2 M

ixtu

res j

ourn

als w

ere

code

d by

two

code

rs.

Tec

hnic

al C

hara

cter

istic

s of

Jou

rnal

Sco

res

Relia

bilit

yRe

liabi

lity Ty

pe o

f ent

rySt

uden

t pe

rfor

man

ceTe

ache

r fe

edba

ck

Var

iabl

esM

ixtu

res

85 85.8

5.8

4.9

1.8

5

% o

f Agr

eem

ent

Inte

rcod

er R

elia

bilit

y

Valid

ityVa

lidity

Stud

ents

’ jou

rnal

scor

es w

ere

corr

elat

ed w

ith th

eir

perf

orm

ance

ass

essm

ent s

core

s (on

ave

rage

r =

.52)

.

Stud

ent P

erfo

rman

ce

0

0.51

1.52

2.53

Scie

ntifc

com

mun

icat

ion

Con

eptu

alun

ders

tand

ing

Proc

edur

alun

ders

tand

ing

Var

iabl

es in

the

fall

Mix

ture

s in

the

spri

ng

Score

Opp

ortu

nity

to L

earn

: L

earn

ing

Act

iviti

es

0510152025

Def

inin

gEx

empl

ifyin

gA

pply

ing

conc

epts

Var

iabl

esM

ixtu

res

Percentage

0510152025

Def

inin

gE

xem

plify

ing

App

lyin

gC

once

pts

Var

iabl

esM

ixtu

res

0510152025303540 Pred

ictin

gRe

sults In

terp

retin

g

Res&

inte

rpre

t Proc

edur

es

Expe

rimen

tsD

esig

ning

Opp

ortu

nity

to L

earn

: L

earn

ing

Act

iviti

esPercentage

Opp

ortu

nity

to L

earn

: T

each

er F

eedb

ack

•Te

ache

rs d

id n

ot p

rovi

de fe

edba

ck d

espi

te e

rror

s or

mis

conc

eptio

ns th

at w

ere

evid

ent i

n th

e st

uden

ts’j

ourn

als.

•O

nly

4 am

ong

the

10 te

ache

rs p

rovi

ded

feed

back

!

No

feed

back

Wro

ng

feed

back

Feed

back

on

how

to

impr

ove

Gra

des/

Che

ck m

ark

01020304050

Percentage

4 fe

edba

ck

clas

sroo

ms

Con

clus

ions

of C

lass

room

A

sses

smen

t Stu

dy

•Sc

ienc

e jo

urna

ls c

an b

e re

liabl

y sc

ored

and

be

used

as a

va

lid a

sses

smen

t too

l.

•St

uden

ts d

id p

oorly

in sc

ient

ific

com

mun

icat

ion

and

show

ed p

artia

l sci

ence

und

erst

andi

ng in

thei

r jou

rnal

s.

•M

ost t

each

ers d

id n

ot e

ffec

tivel

y us

e sc

ienc

e jo

urna

ls.

•Te

ache

rs h

ad v

ery

limite

d co

nten

t kno

wle

dge.

The

y di

d no

t kno

w h

ow to

pro

mot

e or

ass

ess s

tude

nt le

arni

ng.

Dir

ectio

ns a

nd Im

plic

atio

nsD

irec

tions

and

Impl

icat

ions

Dir

ectio

ns a

nd Im

plic

atio

ns

Diss

erta

tion

Stud

yD

isser

tatio

n St

udy

•W

hat o

ther

ass

essm

ent m

etho

ds c

an ta

p in

to th

ese

know

ledg

e ty

pes?

•H

ow c

an w

e m

easu

re ty

pes o

f kno

wle

dge

at th

e cl

assr

oom

leve

l?

•H

ow c

an w

e us

e th

e di

stin

ctio

ns b

etw

een

know

ledg

e ty

pes t

o im

prov

e te

ache

rs’ p

ract

ice?

•H

ow c

an w

e us

e th

ese

dist

inct

ions

to p

rom

ote

stud

ents

’ le

arni

ng?

Dir

ectio

ns a

nd Im

plic

atio

nsCl

assr

oom

Ass

essm

ent S

tudy

Clas

sroo

m A

sses

smen

t Stu

dy

•C

an jo

urna

ls b

e us

ed to

gat

her i

nfor

mat

ion

on

teac

hing

for t

he a

ccou

ntab

ility

pur

pose

s?

•H

ow c

an w

e m

ake

the

use

of jo

urna

ls re

leva

nt to

an

d pr

actic

al fo

r tea

cher

s in

thei

r pra

ctic

e?

•H

ow c

an w

e us

e th

ese

findi

ngs t

o im

prov

e te

ache

rs’

clas

sroo

m a

sses

smen

t pra

ctic

es?

Dir

ectio

ns a

nd Im

plic

atio

nsM

y Re

sear

ch o

n As

sess

men

tM

y Re

sear

ch o

n As

sess

men

t

Lear

ning

/Ach

ieve

men

t(c

ogni

tion)

Asse

ssm

ent

Inte

rpre

tatio

n

TH

AN

K Y

OU

!

I loo

k fo

rwar

d to

yo

ur q

uest

ions

and

com

men

ts..