An iLearn Math Program Evaluation Plan

Claudia LopezOLIT 508

Barbra PortzlineFall 2012

Background

The Highway to Success Learning Communities is at the UNM-Valencia Campus.

We help students transition into College level English & Math courses.

We help all students that tested low in English and/or Math in their Compass or ACT tests.

Rationale

New self-paced math program (iLearn) implemented Summer 2011.

Most of our Math classes now are self-paced iLearn classes.

Some students in these classes are not getting through the material in a timely manor therefore leaving them frustrated and not able to pass the final.

Instructors in these iLearn classes do not lecture, but are there along with a tutor for support or questions.

iLearn program has many glitches, and errors.

Purpose of This Evaluation

• To find out how effective the iLearn program is.

• To find out if this program should be continued or discontinued.

•Results will be used to improve our Math Department.

Primary Stakeholders

• iLearn Program Designers:• Directly impacted by results

whether to keep iLearn program or the need to improve it.

•Highway to Success Program Manager:• Makes final decision as to

continue or discontinue iLearn.

Secondary Stakeholders

• Instructors:• Have a vested interest to

make sure that students pass the final.

• Highway to Success Tutors:• Vested interest in making

sure that students understand the concepts in order to pass.

• Students:• Have a vested interest since

they have to pass the class in order to keep their scholarships, etc.

Tertiary Stakeholders

• Other learning institutions:• May be interested in

implementing iLearn program.

• Community members:• May be considering

taking a self-paced Math course.

Key Questions

Key questions in this evaluation will focus on finding out if there is a need to continue using the iLearn program. Questions will help us understand the effect that the program has on the students as far as retention and being able to continue on to the next level math class. These questions may also help the instructors in planning supplemental worksheets, and/or study guides to help the students understand concepts.

The Key Questions

• In what way(s) is the iLearn math program making a difference at the Highway to Success?

• How do the instructors make sure that the students are learning the concepts that are needed to pass the final?

• To what extent of computer knowledge should the students have in order to use the iLearn program?

• Have the pass rates improved or declined since iLearn was implemented in Summer 2011? If pass rates declined, what can be done to improve them? If pass rates improved, how can we keep the rates high?

The Key Questions Continued

• Students will often minimize the iLearn program by getting on Facebook or other social media. How can we make sure that the time being tracked on iLearn is actually being spent on that program?

• What learning styles is the iLearn program targeting, and how is this being done?

• How effective are the tutors in helping to understand the instruction being presented in the iLearn program?

• Would a special training for math instructors and tutors be effective and in what ways?

Evaluation Design

• This evaluation is meant to be formative.• Conducted for the purpose of either improving the iLearn program

or discontinuing it.• The evaluation will be conducted by one of the Math instructors

along with a Master tutor.• Results will shared with all Highway to Success Math instructors

as well as the program manager.• Existing data will be used from previous semesters’ iLearn

classes, and instructor taught classes.• Students will fill out surveys using mostly open-ended questions.• Evaluations will be given to students in the 10th-12th week of the

semester.• Interviews will be conducted with the instructors & tutors with

questions similar to the students.

Evaluation Design Continued

• Two models that will be used include 2 levels.• Level 1 will be used to measure the reactions of the students

in regards to the information that is being retained with the iLearn program.

• A Level 4 evaluation will be used to measure the effectiveness of the iLearn program in relation to how well it works for the department as well as the instructors.

• This design will include the case study design using Kirkpatrick’s approach.

• We will be using both qualitative and quantitative data in order to answer the key questions.

• The main key question that needs to be answered is if iLearn is helping students pass, or are they failing because they are not understanding the curriculum the iLearn program provides?

• Instrument validity will be important in this evaluation.

Data Analysis – Qualitive

• Interview Questions:• Department Manager will analyze questions carefully & decide

how to categorize the results.• The results will then be put into different categories according

to the key questions.• Each category will be coded, and the frequency of the codes

will be counted.• Data will be displayed using tables, matrices, charts and

diagrams.

• Survey Questions:• Instructors and Department Manager will analyze survey

questions.• Data will be carefully categorized and coded.• This data will be compared to the interview questions as well as

the archival data.• The frequency of these codes will be counted as well.• This data will be displayed in tables, charts and diagrams.

Data Analysis – Quantative

• Archival Data:• Pass rates from previous semesters will be compared to the

survey and interview questions.

• Pass rates will help the math department decide if the iLearn program is improving pass rates compared to the traditional instructor led math class

• Data will be analyzed carefully by the Department Manager, the Administrative assistant, and the math instructors.

• Data will be recalculated to make sure that there is an accurate pass rate percentage being revealed.

Evaluation Issues & Constraints

• Monetary:• Highway to Success has lost their Title V grant & is now

running under UNM-Valencia’s budget.• Since Instructor & tutor are doing the evaluation there will not

be an extra charge, regular pay will be given.• The only charges will be paper, envelopes, and stamps as

needed.• Time:

• Evaluation will be performed a few weeks before finals, instructors might not be willing to give students time for filling out surveys.

• Students might not be willing to fill out surveys due to being behind.

• Instructor might not have time to do interview. For this reason interviews may be done over the phone or on Skype.

Evaluation Issues & Constraints Continued

• Attitudes:• Some instructors do not

want to be bothered with interviews, or evaluations even when it comes to the iLearn program.

• Students attitudes might come from the frustration with the iLearn program, and might not want to fill out the lengthy survey.

Communication & Reporting of Findings

• Primary Stakeholders:• Will get ongoing reports, either by phone or mail, or ongoing

meetings as needed as well as the final report.

• iLearn Program Designers:• Will receive ongoing emails as needed, so they may make

improvements as needed, they will also receive a final report.

• Secondary and Tertiary Stakeholders:• Will be able to see a final report presented as a PowerPoint

presentation with graphs on the Highway to Success website.• Instructors will see the final report in a meeting with a

PowerPoint presentation.

Evaluation Budget

Bibliography

Russ-Eft, D., & Preskill, H. (2009). Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. New York, NY, USA: Basic Books.

Images:

http://www.langevin.com

http://www.transformleaders.tv

http://www.tutor2u.net

http://programsuccess.wordpress.com

http://baneofyourresistance.com

http://harveymackay.com/