Looking for the gaps: Using a limiting factors

approach when evaluating online

learning programsDr. Susan Lowes

Teachers College/Columbia UniversityVSS/2009

“Limiting factors” approach

Term comes from the field of ecology, where often cannot see program results for years

Goal is to identify the current unmanaged factors that are likely to prevent the project from achieving its objectives

Shift focus from looking at success (or only at success) to looking at what is preventing greater success—the limiting factors

Evaluation questions

Are there unmanaged “limiting factors” that raise doubts about whether a program can achieve its goals?

Are there unmanaged “limiting factors” that are preventing a successful program from achieving even greater success?

Limiting factors in online programs

For evaluators, online programs have two key characteristics: Distributed players Hidden activities

The challenge is to see how all the players, distributed over space and over time, work together

DESIGNERS

TEACHERS

STUDENTSPRINCIPALS

SITE COORDINATORS

PARENTS

VIRTUAL SCHOOL

Distributed players

One curriculum designer with distributed teachers

One administrator with distributed teachers, students, and parents

One teacher with distributed students and parents

One student with distributed teachers One site coordinator with distributed

teachers One parent with distributed teachers

Distributed over space

Distributed over time

Responsibilities are distributed over time

Interactions are distributed over time

Hidden activities

Student learning

Teacher

Administrator

Other students

Site coordinator

Curriculum designer

Examples of hidden activities

Looking at teaching and learning online is challenging and very time consuming Curriculum designers cannot see

teachers teaching, only the residual artifacts

Administrators cannot see teachers teaching or students learning, only the residual artifacts

Site coordinators cannot see teachers teaching or students learning, and sometimes not even the residual artifacts

Students cannot always see each other learning, since increasingly use email, Facebook pages, Skype, etc.

In online programs, limiting factors are therefore found in the gaps

DESIGNERS

TEACHERS

STUDENTSPRINCIPALS

SITE COORDINATORS

PARENTS

VIRTUAL SCHOOL

Examples of gaps between players

Between an instructional designer works without input from a teacher, designing courses that are not teachable

Between a teacher who cannot get his/her students to interact online and those students

Between that teacher and the “manager” of the students, generally the site coordinator

Between the student who cannot complete an assignment, has a computer crash, is taken off for a week by his/her parents and the teacher

Between an administrator who puts students into a course and the teacher who feels he/she should not be there

IBOCOURSE

DEVELOPER COURSE DELIVERER TEACHERSITE

COORDINATOR

                     

Develop requirements document        

Do course needs assessment        

Make decision to offer course        

Hire teacher-designerHire instructional

designer      

Design course Design course      

Approve course content        

Evaluate course structure in terms of

requirements document        

  Hire course teacher      

    Train course teacherTake teacher

training  

    Hire and train site coordinator  Take site coordinator

course

Recruit students Recruit students Recruit students    

    Register students    

   Administer readiness pre-

assessment to students    

    Supervise course teacher Teach course  

    Supervise site coordinators   Supervise students

Evaluate course interaction in terms of

requirements document  

Evaluate course interaction in terms of requirements

document    

Evaluate site coordination in terms of

requirements document  

Evaluate site coordination in terms of requirements

document    

    Assess student satisfactionAdminister internal

assessments  

    Assess teacher satisfaction    

Administer external assessments        

Grade external assessments        

And limiting factors arise because the activities are distributed over time

TIM

E