Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)

7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)

http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 1/48

Seeking Evidence of Impact for Technology-Based Innovations in

Continually Dynamic Settings

Chuck Dziuban

Patsy Moskal

Research Initiative for Teaching Effectiveness

Center for Distributed Learning

University of Central Florida



Changing Baselines and

Landscape Amnesia



The Unnatural History of the Sea



A Jessica Hagy Inspired Model of

Changing Baselines

“You look like somebody that I used to know”

Ability toremember what itwas like back then

Your real knowledgeof how it is today

Data you can get your hands on



A Jessica Hagy Inspired Model of

Success & Withdrawal

Attention spansare short

A little A lot

How often you show success and withdrawal

P e o p l e

s ’ I n t e r e s

t

L e v e l

That isfascinating

Whatever



The Evolving Evidence of Impact

Model at UCF



UCF Distributed Learning Impact

Evaluation – Iteration (1)

Students Faculty

Success

SatisfactionDemographic

profiles

Retention

Strategies for

success

Reactive behavior

patternsWill

Scholarship of

Teaching and

Learning

Work?




Evaluation – Iteration (2) (Green is new)

Students Faculty

Success


profiles

Retention

Strategies for

success

Reactive behavior

patterns

Online programs

Writing project model

Web 2.0 teaching

Higher order

evaluation models

Student evaluation of

instruction

Theater

Informationfluency

Generational

comparisons





Students Faculty

Success


profiles

Retention

Strategies for

success

Reactive behavior

patterns

Generational

comparisons Informationfluency

Chemistry

Psychology

Writing Center

Education

Mathematics

Communications

Access





Students Faculty

Success


profiles

Retention

Strategies for

success

Reactive behavior

patterns

Generational

comparisons Informationfluency

Scholarship of

Teaching and

Learning

The faculty

research culturePedagogyAmbivalence

Engagement

Access



UCF Distributed Learning Impact Evaluation –

Iteration (5): A Second Generation Research Agenda

Students Faculty

HELPEnter your second generationresearch ideas in the text box



The Impact of Online Courses



UCF SCH Growth

600000

700000

800000

900000

1000000

1100000

1200000

1300000

1400000

1500000

1600000

02-03 03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 12-13

F2F OTHER VIDEO BLENDED (WEB) ONLINE (WEB)



34,999

59.8% 11,304

19.3% 4,435

7.6%

504

0.9%

1,363

2.3%1,485

2.5%1,825

3.1%

802

1.4%

744

1.3%

828

1.4%

209

0.4%

“Live” Main

Campus Students49,852 “Live” Rosen

Campus Students

2,604

Web

Students

18,565

“Live”

Regional

Students5,198

UCF Fall 2011 Headcount



Student Success



Student Success

88 88 87 8791

8791 91 90 90

949088 88 88 88 89 88

0

10

2030

40

50

60

70

80

90

100

Spring 10 Sum 10 Fall 10 Spring 11 Sum 11 Fall 11

F2F (n=669,518) Blended (n=60,309) Fully Online (n=157,922)



Student Withdrawal

3 2 3 3 2 43 1 3 3 2 45 4 5 4 4 5

0

10

2030

40

50

60

70

80

90

100

Spring 10 Sum 10 Fall 10 Spring 11 Sum 11 Fall 11

F2F (n=749,656) Blended (n=60,958) Fully Online (n=158,031)



Student Satisfaction



Student satisfaction with fully online

courses

46 45

5 3 1

4347

5 4 20

10

20

30

40

50

60

70

80

90

100

Very Satisfied Satisfied Neither Unsatisfied Very

Unsatisified

P e r c e n t

Fully Online (n=1368) Blended (n=877)



Students’ preference for taking future

online courses

69

21

5 4 1

63

21

7 5 30

10

20

30

40

50

60

70

80

90

100

Definitely Probably Not Sure Probably Not Definitely Not

P e r c e n t




Students’ preference for taking future

blended courses

29 2925

5 2

42 39

116

20

10

20

30

40

50

60

70

80

90

100

Definitely Probably Not Sure Probably Not Definitely Not

P e r c e n t




Students’ preference for class modalities

34

11 188

12 15

28

12

24

11 11 13

0

10

20

30

40

50

60

70

80

90

100

Fully

Online

Mostly

Online

Equal Mostly F2F Fully F2F No

preference

P e r c e n t




Faculty Satisfaction



Sections previously taught by faculty

23 24

77

14 19

3439

12

15

24

1117

5

11

19

8

10

7

8

2410 5

53

3016+ sections

11-15 sections

6-10 sections

1-5 sections

None

Online

(n=176)

Blended

(n=163)

V/RV

(n=142)

F2F, no Web

(n=160)

F2F, with Web

(n=168)



Number of students faculty feel they

can effectively teach

3627 28 34

25

3242 41

3742

1116

13

1918

68 8 5

147 11 8 10

100+ students

76-100 students

51-75 students

31-50 students

<30 students

Online

(n=176)

Blended

(n=162)

V/RV

(n=120)

F2F, no Web

(n=167)

F2F, with Web

(n=170)



Faculty satisfaction by modality

Online

(n=153)

Blended

(n=143)

V/RV

(n=72)

F2F, no Web

(n=150)

F2F, with Web

(n=152)

20

41

18

18

31

36

25

1018

57

7

39

45

2511

8

45

38

15

S a t i s f i e d

N e u t r a l o r u n s a t i s f i e d

Very satisfied

Satisfied

Neutral

Unsatisfied

Very

unsatisfied



Positive aspects of teaching an

online course (n=196)

48

19 1611 10 8

0

10

2030

40

50

6070

80

90

100

P e r c e n

t

Accessibility

Flexibility Resources Depth

Interaction Organization



Less positive aspects of teaching an

online course (n=196)

46

2312 11

3

0

10

2030

40

50

6070

80

90

100

P e r c e n

t

Disengaged

students

Disconnect

with students

Integrity

issues

Technology

issues

Time

consuming



What are the students saying

about me?

Evidence that held rock solid.

“This guy is so

boring that my pillowneeds a pillow.”

-Rate My Professors



Ambivalence and the Structure of

Political Opinion



Facilitation of learning

Communication of ideas

Excellent Very Good Good Fair Poor

Then...

The probability of an overall rating of Excellent = .97 &

The probability of an overall rating of Fair or Poor =.00

If...

A decision rule for the probability of faculty member

receiving an overall rating of Excellent (n=1,280,890)

Respect and concern for students

A comparison of excellent ratings by course



A comparison of excellent ratings by course

modality--unadjusted and adjusted for instructors

satisfying the rule (n=1,171,664)

Blended 52 97Fully Online 48 97

F2F 48 97

Blended LC 43 97Lecture Capture 42 97

Course Overall If Rule 1

Modality % Excellent % Excellent

A comparison of excellent ratings by top ten most



A comparison of excellent ratings by top ten most

heavily populated departments--unadjusted and adjusted

for instructors satisfying the rule (n= 1,286,890)

Overall If Rule 1Department % Excellent % Excellent

Political Science 56 97

History 56 97

Crim. Justice 55 97

Psychology 53 97

Finance 51 98

Microbiology 50 98English 48 97

Biology 44 97

Mathematics 38 97

A comparison of excellent ratings by college unadjusted



A comparison of excellent ratings by college unadjusted

and adjusted for instructors satisfying the rule

(n=1,280,890)

Overall If Rule 1College % Excellent % Excellent

Education 60 98

Nursing 54 98

Health & Public Affairs 53 98

Arts & Humanities 51 97

Optics & Photonics 49 94

Hospitality Management 48 97Medicine 46 98

Sciences 45 97

Business Administration 44 97

Engineering & Comp Sci 39 97

A comparison of excellent ratings by class level



A comparison of excellent ratings by class level

unadjusted and adjusted for instructors satisfying the

rule (n=1,171,664)

Freshman 46 97Sophomore 45 97

Junior 48 97

Senior 51 97

Class Overall If Rule 1

Level % Excellent % Excellent

A comparison of excellent ratings by class size decile



A comparison of excellent ratings by class size decile

unadjusted and adjusted for instructors satisfying the rule

(n=1,17,664)

Overall If Rule 1Decile % Excellent % Excellent

1 54 97

2 51 97

3 54 98

4 52 98

5 46 97

6 47 977 48 97

8 44 97

9 44 97

10 41 97



Principles that Guide Evidence

of Impact



There is good, fast, and cheap –

Pick any two



Don’t let technology drive the

evaluation or assessment



Statistics and measurement are

great but they are not everything



Push access and flexibility out as

far as possible



A great product makes a huge

difference



Just because you can doesn’t

mean you should



Progress made is better made in

small steps



Simple really is better



Decision makers don’t knowwhat you are talking about.

Make it relevant to their world



One more chance!

Students Faculty

HELPEnter your second generationresearch ideas in the text box



For more information contact:

Dr. Chuck Dziuban

(407) 823-5478

[email protected]

Dr. Patsy Moskal

(407) 823-0283

[email protected]

http://rite.ucf.edu

http://www.if.ucf.edu/

Documents

Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)