Meaningful Assessment and the Impact of Technology

Meaningful Assessment and the Impact of Technology

Catherine Kelley, Ph.D.

Senior Faculty Consultant

Assistant Professor

Meaningful Assessment and the Impact of Technology

Assessment of Mobile Computing at Seton Hall

Catherine Kelley, Ph.D.

Assistant Professor, Psychology

Senior Faculty Consultant, TLTC

Meaningful Assessment and the Impact of Technology

Why assess?

• Track progress being made on your project

• Greater understanding of teaching & learning

• Ensure high-quality experience for your students

• Communicate successes / challenges to colleagues

Meaningful Assessment and the Impact of Technology

Traditional (Flawed) theoretical model:

• Teaching => Learning

• Teaching + Computers => Improved Learning

• What is wrong with this model?

Meaningful Assessment and the Impact of Technology

Traditional (Flawed) Research Design:

• Control group - no computer

• Experimental group - computer

• Measure exam scores

• What is wrong with this design?

Meaningful Assessment and the Impact of Technology

Some problems with traditional model

• Hawthorne effects• Multivariate nature of change means that an

experiment would be hopelessly confounded• Your outcome variables may (will probably!)

change with new teaching techniques– You may not know yet what the really important

outcome variables are!

Meaningful Assessment and the Impact of Technology

What about a multivariate model?

• So Many Variables!

• Each So Difficult to Measure!

• Huge n required to do any meaningful multivariate study

• Not a bad idea, but was outside of the intended scope of our project

Meaningful Assessment and the Impact of Technology

So what do you do?

• What about educational quality variables?

• Theoretical models of educational quality: e.g. Chickering & Gamson, “Powerful Partnerships,” etc.

• Common themes emerge

Meaningful Assessment and the Impact of Technology

What is “educational quality”?

• Student - professor contact

• Student - student contact

• Prompt feedback on performance

• Explore course materials in many ways

• Take responsibility for learning

• Etc.

Meaningful Assessment and the Impact of Technology

Seton Hall Instruments

• Two Versions: “Standard Form”, “New Form”

• Standard Form has been used for several years

• New Form has been piloted with considerable success

Meaningful Assessment and the Impact of Technology

Standard Form

• “Compare your mobile courses, or those incorporating information technology, to your non-mobile courses, or those not incorporating information technology. Indicate how the use of information technology impacts each of the following items.”

Meaningful Assessment and the Impact of Technology

Standard Form (examples)

• Have contact with my professors on course content

• Spend time with other students on course related materials

• Discuss, write and think about course material in depth

• Get prompt feedback on my performance from my professors ….

• 29 of these questions

Meaningful Assessment and the Impact of Technology

New Form

• Same questions, but students answer only about one course

• And then indicate how much IT is used in that class

• But also, how many traditional pedagogical techniques are used in that class

Meaningful Assessment and the Impact of Technology

Results:Old Form

• Mobile courses look better on educational quality - Sum across all measures

Meaningful Assessment and the Impact of Technology

Results: Old Form

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Very Negative Negative Neutral Positive Very Positive

Pe

rce

nt

Re

sp

on

din

g

Meaningful Assessment and the Impact of Technology

Results: New Form

• The more IT, the higher the educational quality score

Meaningful Assessment and the Impact of Technology

Results: New form

-80

-60

-40

-20

0

20

40

60

80

0 5 10 15 20 25

Number Computer Techniques

Ed

uc

ati

on

al Q

ua

lity

Meaningful Assessment and the Impact of Technology

Results: New Form

• The more traditional techniques, the higher the educational quality score – but effect is very weak

Meaningful Assessment and the Impact of Technology

Results: New Form

-80

-60

-40

-20

0

20

40

60

80

0 10 20 30 40

Number Traditional Techniques

Ed

uc

ati

on

al Q

ua

lity

Meaningful Assessment and the Impact of Technology

Summary of findings

• Students perceive that educational variables are higher in Mobile classes

• Student satisfaction with Mobile remains high

Meaningful Assessment and the Impact of Technology

The future

• Institutionalizing the assessment process– Risks: over-surveying students, poor

response rate– Asking some tough questions: Retention,

student skill, negative impact of technology

• Surveys of faculty, administrators

Meaningful Assessment and the Impact of Technology

The future

• Assessment reports should also make use of archival data, not just surveys– E.g. how many courses make use of

technology? And to what extent?

• Are we realistic in our expectations of faculty? Administrators?