A Synthesis of OER Efficacy and Perceptions Research: 2015-2016

John Hilton III, PhDBrigham Young University

@johnhiltoniii

By John Hilton III. This work is licensed under the Creative Commons Attribution 4.0 International License, except where noted.

In 2015 and 2016 there were…• Four peer-reviewed studies published that

focused on comparing how students performed when using OER versus traditional resources

• Three peer-reviewed studies (plus two white papers) published that focused on comparing user perceptions of OER with traditional resources

Why Should We Care AboutOER Research?

The Babson 2014 survey found that college professors rate “proven efficacy” and “trusted quality” as the number 1 and number 2 most important criteria for selecting teaching resources.

-One key way to establish efficacy an quality is through research.

Image by Petr Kratochvil (Website)

"Sacramento City College has a large percentage of its student body who are below the poverty line. When I require students to spend $78 on a book, many of them simply cannot. Having a free textbook is a matter of equity, it is allowing low-income students to have an equal shot at earning a good grade! I had a young woman in my office today, crying that she had to choose between feeding her children and buying books for school! Because of [an OER], she doesn't have to pay for a book in my class!”

– Jessica Coppola (Sacramento City College)

Public Domain Image Courtesy of George

Hodan

If you didn't have to pay for textbooks, how would you use the

money you saved?

If you didn't have to pay for textbooks, how would you use the money you saved?

“Fresh fruits and vegetables. Money is extremely tight now so I am eating mostly rice and beans with some cheese. I miss eating fresh food.”

Additional Comments on Textbook PricesComment Category

Number of Responses

Sample Comment

Positive 12 (2%) “I appreciate keeping textbooks when it was a good, thought out, well written choice.”

Additional Comments on Textbook PricesComment Category

Number of Responses

Sample Comment

Positive 12 (2%) “I appreciate keeping textbooks when it was a good, thought out, well written choice.”

Neutral 49 (8%) “I…like it when teachers use free online articles instead.”

Additional Comments on Textbook PricesComment Category

Number of Responses

Sample Comment

Positive 12 (2%) “I appreciate keeping textbooks when it was a good, thought out, well written choice.”

Neutral 49 (8%) “I…like it when teachers use free online articles instead.”

Negative 339 (56%) “New editions don't usually change enough to be worth the cost.”

Additional Comments on Textbook PricesComment Category

Number of Responses

Sample Comment

Positive 12 (2%) “I appreciate keeping textbooks when it was a good, thought out, well written choice.”

Neutral 49 (8%) “I…like it when teachers use free online articles instead.”

Negative 339 (56%) “New editions don't usually change enough to be worth the cost.”

Extremely Negative

223 (37%) “Textbooks are the biggest scam targeting the poorest demographic. Requiring specific editions is a gross abuse of power for monetary gain.”

Let Me Tell You Some Stories About Efficacy

You can see http://openedgroup.org/review if you’d like more details on these research studies.

Assessing the Impact and Efficacy of the Open-Access ChemWiki Textbook Project.

Allen, G., Guzman-Alvarez, A., Molinaro, M., Larsen, D. (2015). Educause Learning Initiative Brief, January 2015.

• Same semester

• Same teacher• Same lecture • Same

homework• Same TAs• Same exams• Back to back

class periods

478 students 448 students

FREE290.05Total cost > 130,000

Total cost = 0.00

A Multi-Institutional Study of the Impact of Open Textbook Adoption on the Learning Outcomes of Post-secondary Students

Fischer, Hilton, Robinson, and Wiley

Journal of Computing in Higher Education (2015)

OER Users took 15% More Credits

Fall Winter02468

101214

OER Users Others

1 control class did better than treatment

Passing with a C- or Better

5 treatment classes did better than control

9 classes = no sig. difference

“The Tidewater Z-Degree and the INTRO Model for Sustaining OER Adoption.”

Wiley, Williams, DeMarte, HiltonEducation Policy Analysis Archives, 24(41), pp.1-12.

When a student drops, it..Slows down their graduation

Costs the institution tuition dollars(refunds)

If all business courses at TCC had the lower drop rate, TCC would retain $101,042 in additional income each year.

Improving Course Throughput Rates and Open Educational Resources: Results from the Z Degree Program at Tidewater Community College

Hilton, Fischer, Wiley, and Williams

Accepted in the International Review of Research in Open and Distance Learning

Course Throughput Rate

IRRODL (in press)

Drop Deadline

WithdrawDeadline

FinalGrade

Students

Commercial vs OER

2.3% | 1.8%

9.9% | 8.1%

68% | 74%

(Face to Face)

59.8% | 66.4%

Drop

Withdraw

C or Better

CTRIRRODL (in press)

Commercial vs OER

4.0% | 1.4%

13.7% | 13.1%

66% | 70%

(Online)

Drop

Withdraw

C or Better

IRRODL (in press)54.2% | 59.8%CTR

We’ve Just Heard Four Stories About…• A dozen institutions• Thousands of students

Image Credit: Tulane Public Relations

When using OER students–Dropped fewer courses–Were less likely to withdraw–We more likely to pass their classes–Took more classes– Saved millions of dollars

Let Me Tell You Some Stories About Perceptions of Quality

You can see http://openedgroup.org/review if you’d like more details on these research studies.

Exploring faculty use of open educational resources at British Columbia post-secondary institutions. Jhangiani, R. S., Pitt, R., Hendricks, C., Key, J., & Lalonde, C. (2016). BC Campus Research Report. Victoria, BC.

Educators who had adopted OER rated the quality of OER as significantly higher than those who had not adopted OER [1.88 vs. 3.28 on a 5-point Likert scale].

Mainstreaming Open Textbooks: Educator Perspectives on the Impact of OpenStax College open textbooks. Pitt, R. The International Review of Research in Open And Distributed Learning, 16(4).

79 respondents (~ 2/3 of the sample) strongly agreed or agreed that

use of OER such as OpenStax increased learners’ satisfaction

with the learning experience.

University students and faculty have positive perceptions of open/alternative resources and their utilization in a textbook replacement initiative.

Delimont, N., Turtle, E. C., Bennett, A., Adhikari, K., & Lindshield, B. L. (2016). Research in Learning Technology, 24.

Researchers surveyed 524 students in thirteen different courses at Kansas State University

regarding their use of OER. They found:

• Students used the OER more than commercial textbooks.

• Students rated the OER as “good quality.”• Students preferred using OER to buying

textbooks.

Thirteen instructors were interviewed; all but one said they “preferred teaching

their course with [OER] instead of a traditional textbook.”

Examining Student Perceptions of an Open Statistics Book

Illowsky, B. S., Hilton III, J., Whiting, J., & Ackerman, J. D., Open Praxis, 8(3): 265-277.

Across two years, 325 students at one college

were surveyed regarding their

use on an Open Statistics

Textbook

Better 24%

Same 64%

Worse12%

How would you rate the quality of the OER as compared to other

textbooks you have used?

OER Adoption Study: Using Open Educational Resources in the College Classroom

The California OER Council (2016). OER Adoption Study: Using Open Educational Resources in the College Classroom.

16 faculty shared their perceptions about their use of OER

• Seven faculty of sixteen felt that the OER textbook was superior to the traditional textbook for the course. Five faculty rated the OER as equivalent to the traditional textbook.

• Faculty were not as positive about the support materials (PowerPoints, Test banks) available with the OER textbooks. Half of the faculty felt that the support materials lacked quality. 25% of faculty felt that implementing the support materials took a significant amount of time. In their comments, the biggest comment made by faculty was about the need for support materials or the amount of time they spent in developing them for this adoption.

351 students

completed a survey

about their use of OER.

42% Better

39% Same

11% Worse

8% No Answer

How does OER Compare to Commercial Textbooks?

Of the 351 students, 100% wanted to use

OER textbooks in the future and

would recommend the use of OER to

friends.

100%

Across multiple recent perceptions studies, researchers found…

Students prefer OERFaculty prefer OEROER helps students

14 Peer Reviewed Studies of Perceptions of OER Quality

http://openedgroup.org/

6,050 Professors and Students

http://openedgroup.org/

50%Same35%

Better

15%Worse

http://openedgroup.org/

13 Peer Reviewed Studies of Efficacy

http://openedgroup.org/

119,720 Students

http://openedgroup.org/

95% Same or Better Outcomes

http://openedgroup.org/

Overall Takeaways• From the dawn of time until today there have

been 24 efficacy and/or perception studies that compare OER with commercial products. Nine of these have been done in the last two years…not bad!

Overall Takeaways• From the dawn of time until today there have been

24 efficacy and/or perception studies that compare OER with commercial products. Ten of these have been done in the last two years…not bad!

• But we need more – and increasingly rigorous – studies.

How can we do better research?http://openedgroup.org/toolkit

Control for teacher effect Interrupted Time SeriesSame teacher using both resources in

the same semester

How can we do better research?http://openedgroup.org/toolkit

Control for student differencesStatistically control for differencesPre-post test to determine initial differences

How can we do better research?http://openedgroup.org/toolkit

When doing perception studies, specifically ask for comparisons between open and commercial resources.

Consider asking questions about “use” in perceptions studies.

Take Away MessageWe need more research and also more stories of impact. Please gather those stories and distribute them however you can. Help us see the human side of the incredible benefits that come from replacing commercial resources with open ones.

A Synthesis of OER Efficacy and Perceptions Research: 2015-2016

John Hilton III, PhDBrigham Young University

@johnhiltoniii

By John Hilton III. This work is licensed under the Creative Commons Attribution 4.0 International License, except where noted.